[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[
https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383100#comment-17383100
]
ASF GitHub Bot commented on HUDI-648:
-
liujinhui1994 commented on a change in pull request #2710:
URL: https://github.com/apache/hudi/pull/2710#discussion_r672063131
##
File path:
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/error/FlinkHoodieBackedErrorTableWriter.java
##
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.error;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hudi.client.HoodieFlinkWriteClient;
+import org.apache.hudi.client.WriteStatus;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.model.HoodieKey;
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.model.HoodieRecordPayload;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.table.HoodieTable;
+
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class FlinkHoodieBackedErrorTableWriter
extends
+HoodieBackedErrorTableWriter>, List,
List> {
+
+ private static final Logger LOG =
LogManager.getLogger(FlinkHoodieBackedErrorTableWriter.class);
+
+ public static HoodieBackedErrorTableWriter create(Configuration conf,
HoodieWriteConfig writeConfig, HoodieEngineContext context) {
+return new FlinkHoodieBackedErrorTableWriter(conf, writeConfig, context);
+ }
+
+ FlinkHoodieBackedErrorTableWriter(Configuration hadoopConf,
HoodieWriteConfig writeConfig, HoodieEngineContext engineContext) {
+super(hadoopConf, writeConfig, engineContext);
+ }
+
+ @Override
+ protected void initialize(HoodieEngineContext engineContext,
HoodieTableMetaClient datasetMetaClient) {
+
+try {
+ bootstrapErrorTable(datasetMetaClient);
+} catch (IOException e) {
+ LOG.error("init error table fail", e);
+}
Review comment:
ok，i will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction
> writes
>
>
> Key: HUDI-648
> URL: https://issues.apache.org/jira/browse/HUDI-648
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer, Spark Integration, Writer Core
>Reporter: Vinoth Chandar
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:normal, user-support-issues
> Attachments: image-2021-03-03-11-40-21-083.png
>
>
> We would like a way to hand the erroring records from writing or compaction
> back to the users, in a separate table or log. This needs to work generically
> across all the different writer paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383113#comment-17383113 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot commented on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-882324273 ## CI report: * 63481f5de5e64f47f708efa22b627083bb007424 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383116#comment-17383116 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-882324273 ## CI report: * 63481f5de5e64f47f708efa22b627083bb007424 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1009) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383841#comment-17383841 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-882324273 ## CI report: * 63481f5de5e64f47f708efa22b627083bb007424 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1009) * bdfa3848db3f1c4a20a4be61af0788f28fcc69de UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383849#comment-17383849 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-882324273 ## CI report: * bdfa3848db3f1c4a20a4be61af0788f28fcc69de Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1037) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383871#comment-17383871 ] ASF GitHub Bot commented on HUDI-648: - lw309637554 commented on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-883195678 @liujinhui1994 hello , does it ready to review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383880#comment-17383880 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 opened a new pull request #3308: URL: https://github.com/apache/hudi/pull/3308 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request When the schema and data message formats do not match, the writing will fail. We will record the error message to facilitate troubleshooting ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request org.apache.hudi.error.TestHoodieBackedErrorTable This unit test verifies this scenario *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383885#comment-17383885 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #2710: URL: https://github.com/apache/hudi/pull/2710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383882#comment-17383882 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #3308: URL: https://github.com/apache/hudi/pull/3308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383889#comment-17383889 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #3309: URL: https://github.com/apache/hudi/pull/3309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383888#comment-17383888 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 opened a new pull request #3309: URL: https://github.com/apache/hudi/pull/3309 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383935#comment-17383935 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 opened a new pull request #3308: URL: https://github.com/apache/hudi/pull/3308 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request When the schema and data message formats do not match, the writing will fail. We will record the error message to facilitate troubleshooting ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request org.apache.hudi.error.TestHoodieBackedErrorTable This unit test verifies this scenario *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383960#comment-17383960 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #2710: URL: https://github.com/apache/hudi/pull/2710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383967#comment-17383967 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #3308: URL: https://github.com/apache/hudi/pull/3308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383972#comment-17383972 ] ASF GitHub Bot commented on HUDI-648: - lw309637554 commented on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-883195678 @liujinhui1994 hello , does it ready to review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17383981#comment-17383981 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 opened a new pull request #3309: URL: https://github.com/apache/hudi/pull/3309 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384030#comment-17384030 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-882324273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384032#comment-17384032 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #3309: URL: https://github.com/apache/hudi/pull/3309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384088#comment-17384088 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 opened a new pull request #3312: URL: https://github.com/apache/hudi/pull/3312 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request When the schema and data message formats do not match, the writing will fail. We will record the error message to facilitate troubleshooting ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request org.apache.hudi.error.TestHoodieBackedErrorTable This unit test verifies this scenario ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384091#comment-17384091 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384101#comment-17384101 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 opened a new pull request #3308: URL: https://github.com/apache/hudi/pull/3308 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request When the schema and data message formats do not match, the writing will fail. We will record the error message to facilitate troubleshooting ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request org.apache.hudi.error.TestHoodieBackedErrorTable This unit test verifies this scenario *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384119#comment-17384119 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #2710: URL: https://github.com/apache/hudi/pull/2710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384128#comment-17384128 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #3308: URL: https://github.com/apache/hudi/pull/3308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384134#comment-17384134 ] ASF GitHub Bot commented on HUDI-648: - lw309637554 commented on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-883195678 @liujinhui1994 hello , does it ready to review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384139#comment-17384139 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 opened a new pull request #3309: URL: https://github.com/apache/hudi/pull/3309 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request *(For example: This pull request adds quick-start document.)* ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request *(Please pick either of the following options)* This pull request is a trivial rework / code cleanup without any test coverage. *(or)* This pull request is already covered by existing tests, such as *(please describe tests)*. (or) This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end.* - *Added HoodieClientWriteTest to verify the change.* - *Manually verified the change by running a job locally.* ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384193#comment-17384193 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-882324273 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384196#comment-17384196 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 closed pull request #3309: URL: https://github.com/apache/hudi/pull/3309 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384198#comment-17384198 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384192#comment-17384192 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 opened a new pull request #3312: URL: https://github.com/apache/hudi/pull/3312 ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request When the schema and data message formats do not match, the writing will fail. We will record the error message to facilitate troubleshooting ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request org.apache.hudi.error.TestHoodieBackedErrorTable This unit test verifies this scenario ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384217#comment-17384217 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384219#comment-17384219 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-883359159 move to [https://github.com/apache/hudi/pull/3312/files](url) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384220#comment-17384220 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 82153aa36c1187d2419833d35f347875d9a52082 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384221#comment-17384221 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 removed a comment on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-883359159 move to [https://github.com/apache/hudi/pull/3312/files](url) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384224#comment-17384224 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #2710: URL: https://github.com/apache/hudi/pull/2710#issuecomment-883360492 > @liujinhui1994 hello , does it ready to review? I restarted a PR https://github.com/apache/hudi/pull/3312 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384257#comment-17384257 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 82153aa36c1187d2419833d35f347875d9a52082 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1054) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384306#comment-17384306 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 82153aa36c1187d2419833d35f347875d9a52082 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1054) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384308#comment-17384308 ] ASF GitHub Bot commented on HUDI-648: - codecov-commenter commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883444320 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3312](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (82153aa) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `31.79%`. > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3312 +/- ## = - Coverage 47.74% 15.95% -31.80% + Complexity 5591 504 -5087 = Files 938 285 -653 Lines 4182311980-29843 Branches 4213 994 -3219 = - Hits 19968 1911-18057 + Misses20070 9904-10166 + Partials 1785 165 -1620 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `0.00% <0.00%> (-34.56%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.84% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <ø> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...pache/hudi/error/HoodieBackedErrorTableWriter.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2Vycm9yL0hvb2RpZUJhY2tlZEVycm9yVGFibGVXcml0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/metrics/HoodieMetrics.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllTWV0cmljcy5qYXZh) | `0.00% <0.00%> (-72.50%)` | :arrow_down: | | [...main/java/org/apache/hudi/metrics/HoodieGauge.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=gi
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384316#comment-17384316 ] ASF GitHub Bot commented on HUDI-648: - codecov-commenter edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883444320 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3312](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (82153aa) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `20.16%`. > The diff coverage is `18.18%`. [](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3312 +/- ## = - Coverage 47.74% 27.57% -20.17% + Complexity 5591 1330 -4261 = Files 938 392 -546 Lines 4182315719-26104 Branches 4213 1395 -2818 = - Hits 19968 4335-15633 + Misses2007011054 -9016 + Partials 1785 330 -1455 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.16% <18.18%> (-13.40%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `?` | | | hudisync | `4.88% <ø> (-51.10%)` | :arrow_down: | | huditimelineservice | `?` | | | hudiutilities | `59.84% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <ø> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...pache/hudi/error/HoodieBackedErrorTableWriter.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2Vycm9yL0hvb2RpZUJhY2tlZEVycm9yVGFibGVXcml0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/metrics/HoodieMetrics.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllTWV0cmljcy5qYXZh) | `0.00% <0.00%> (-72.50%)` | :arrow_down: | | [...va/org/apache/hudi/client/SparkRDDWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384328#comment-17384328 ] ASF GitHub Bot commented on HUDI-648: - codecov-commenter edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883444320 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3312](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (82153aa) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `8.79%`. > The diff coverage is `18.18%`. [](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3312 +/- ## - Coverage 47.74% 38.94% -8.80% + Complexity 5591 2025-3566 Files 938 470 -468 Lines 4182320151 -21672 Branches 4213 2066-2147 - Hits 19968 7848 -12120 + Misses2007011465-8605 + Partials 1785 838 -947 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `?` | | | hudiclient | `21.16% <18.18%> (-13.40%)` | :arrow_down: | | hudicommon | `?` | | | hudiflink | `?` | | | hudihadoopmr | `?` | | | hudisparkdatasource | `67.04% <ø> (-0.06%)` | :arrow_down: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.84% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <ø> (ø)` | | | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `0.00% <0.00%> (-43.38%)` | :arrow_down: | | [...pache/hudi/error/HoodieBackedErrorTableWriter.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2Vycm9yL0hvb2RpZUJhY2tlZEVycm9yVGFibGVXcml0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...in/java/org/apache/hudi/metrics/HoodieMetrics.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvSG9vZGllTWV0cmljcy5qYXZh) | `0.00% <0.00%> (-72.50%)` | :arrow_down: | | [...va/org/apache/hudi/client/SparkRDDWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384329#comment-17384329 ] ASF GitHub Bot commented on HUDI-648: - codecov-commenter edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883444320 # [Codecov](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#3312](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (82153aa) into [master](https://codecov.io/gh/apache/hudi/commit/a086d255c89d12eb42cad8c5ae0e000f3b83bbe6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a086d25) will **decrease** coverage by `0.22%`. > The diff coverage is `12.15%`. [](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) ```diff @@ Coverage Diff @@ ## master#3312 +/- ## - Coverage 47.74% 47.52% -0.23% - Complexity 5591 5598 +7 Files 938 944 +6 Lines 4182342078 +255 Branches 4213 4238 +25 + Hits 1996819997 +29 - Misses2007020286 +216 - Partials 1785 1795 +10 ``` | Flag | Coverage Δ | | |---|---|---| | hudicli | `39.97% <ø> (ø)` | | | hudiclient | `34.28% <14.73%> (-0.28%)` | :arrow_down: | | hudicommon | `48.38% <1.61%> (-0.26%)` | :arrow_down: | | hudiflink | `59.48% <ø> (+0.04%)` | :arrow_up: | | hudihadoopmr | `51.99% <66.66%> (-0.03%)` | :arrow_down: | | hudisparkdatasource | `67.04% <ø> (-0.06%)` | :arrow_down: | | hudisync | `55.97% <ø> (ø)` | | | huditimelineservice | `64.07% <ø> (ø)` | | | hudiutilities | `59.84% <ø> (-0.04%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/hudi/pull/3312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [.../apache/hudi/client/AbstractHoodieWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9BYnN0cmFjdEhvb2RpZVdyaXRlQ2xpZW50LmphdmE=) | `0.00% <ø> (ø)` | | | [...pache/hudi/error/HoodieBackedErrorTableWriter.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2Vycm9yL0hvb2RpZUJhY2tlZEVycm9yVGFibGVXcml0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...org/apache/hudi/client/HoodieFlinkWriteClient.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L0hvb2RpZUZsaW5rV3JpdGVDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [.../hudi/error/FlinkHoodieBackedErrorTableWriter.java](https://codecov.io/gh/apache/hudi/pull/3312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aHVkaS1jbGllbnQvaHVkaS1mbGluay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvZXJyb3IvRmxpbmtIb29kaWVCYWNrZWRFcnJvclRhYmxlV3JpdGVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../org/apache/hudi/clie
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385853#comment-17385853 ] ASF GitHub Bot commented on HUDI-648: - lw309637554 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885351154 @liujinhui1994 hello, does it ready to review? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385874#comment-17385874 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885357803 OK, let's get started. Your guidance please @lw309637554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385878#comment-17385878 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885360664 Is there a partner who can help me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17385879#comment-17385879 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 removed a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-885360664 Is there a partner who can help me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293513#comment-17293513 ] liujinhui commented on HUDI-648: Can you let me implement it? I have implemented the function initially, and the error record has been written into it. basepath/.hoodie/error [~vinoth] [~rxu] > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Priority: Major > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17293945#comment-17293945 ] Vinoth Chandar commented on HUDI-648: - absolutely. you have it now. > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294219#comment-17294219 ] liujinhui commented on HUDI-648: Thank you very much, the core code has been completed and I have tested it myself, and there is currently a lack of unit tests. I will perform PR after supplementing the unit tests. > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294249#comment-17294249 ] Raymond Xu commented on HUDI-648: - [~liujinhui] thanks for taking this up. Just a note on the implementation: we were discussing it over the linked RFC 20 and thinking having the hudi table along side the main table, which is different from what we were thinking, as in having `.hoodie/errors/`. Would that differ a lot from what you have implemented? cc [~vinoth] > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294251#comment-17294251 ] liujinhui commented on HUDI-648: [~xushiyan] I mainly refer to the design of RFC-20, my implementation is not too different from RFC-20 > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294253#comment-17294253 ] Raymond Xu commented on HUDI-648: - [~liujinhui] ok great! > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294254#comment-17294254 ] liujinhui commented on HUDI-648: !image-2021-03-03-11-40-21-083.png! > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294255#comment-17294255 ] Raymond Xu commented on HUDI-648: - [~liujinhui] ok cool! we can discuss further over the pr. thanks! > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294318#comment-17294318 ] liujinhui commented on HUDI-648: Thank you very much. Although I have performed functional verification, I have not yet written a unit test. I will complete the unit test as soon as possible.[~xushiyan] > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300188#comment-17300188 ] liujinhui commented on HUDI-648: [~xushiyan] [~vinoth] When querying the error record, we introduce a configuration, similar to hhoodie.datasource.query.table=error basepath = "hdfs://xxx/datalake/data/tmp/"; or Directly specify basepath = "hdfs://xxx/datalake/data/tmp/.hoodie/errors"; Which one is better? Please help with comments > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320498#comment-17320498 ] Vinoth Chandar commented on HUDI-648: - [~liujinhui] this slipped past me. are you still working on this? Is there a PR we can look at? directly using basepath is preferrable IMO. Thats what we did for the metadata table. > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324600#comment-17324600 ] Raymond Xu commented on HUDI-648: - [~vinoth] The PR is [https://github.com/apache/hudi/pull/2710] somehow not linked by itself. We previously discussed over the RFC about putting error table alongside the main table or having a global error table. I'm also ok with putting it alongside metadata table, having everything in one place. [~liujinhui] would you kindly update the RFC to reflect the latest design please? given you've implemented in a slightly different way. > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325963#comment-17325963 ] Vinoth Chandar commented on HUDI-648: - I see it linked now. I queued the PR up for review. > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17395644#comment-17395644 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 82153aa36c1187d2419833d35f347875d9a52082 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1054) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17395809#comment-17395809 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 82153aa36c1187d2419833d35f347875d9a52082 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1054) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397837#comment-17397837 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 82153aa36c1187d2419833d35f347875d9a52082 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1054) * 33b0889dec4ae73ef391655eb9032da09bc3650f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397882#comment-17397882 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 82153aa36c1187d2419833d35f347875d9a52082 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1054) * 33b0889dec4ae73ef391655eb9032da09bc3650f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1681) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17397889#comment-17397889 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 33b0889dec4ae73ef391655eb9032da09bc3650f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1681) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17400244#comment-17400244 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-900099299 @lw309637554 @xushiyan Hope to get your suggestions, please help review. thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17401301#comment-17401301 ] ASF GitHub Bot commented on HUDI-648: - pratyakshsharma commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-901344677 @liujinhui1994 Thank you for raising this. Quite useful for debugging. Can you please resolve the conflicts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402961#comment-17402961 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-903460163 > @liujinhui1994 Thank you for raising this. Quite useful for debugging. Can you please resolve the conflicts? This is actually not used for debugging. There may be some wrong records in the production environment. Users of these records expect a way to trace back to facilitate the location of specific problems. Ok i will handle the conflict -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17402984#comment-17402984 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 33b0889dec4ae73ef391655eb9032da09bc3650f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1681) * 1e4e0ccce4d5fe641bb760bf6b16aca41dfe7bcc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403010#comment-17403010 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 1e4e0ccce4d5fe641bb760bf6b16aca41dfe7bcc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1860) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403029#comment-17403029 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 1e4e0ccce4d5fe641bb760bf6b16aca41dfe7bcc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1860) * a60f2df2b73e6c09ced00e19c341d0a74c70dbe6 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403044#comment-17403044 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 1e4e0ccce4d5fe641bb760bf6b16aca41dfe7bcc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1860) * a60f2df2b73e6c09ced00e19c341d0a74c70dbe6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1862) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403058#comment-17403058 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * a60f2df2b73e6c09ced00e19c341d0a74c70dbe6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1862) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403059#comment-17403059 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * a60f2df2b73e6c09ced00e19c341d0a74c70dbe6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1862) * b0a203a766d22bd828505812029de9da3f129b3e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403087#comment-17403087 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * a60f2df2b73e6c09ced00e19c341d0a74c70dbe6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1862) * b0a203a766d22bd828505812029de9da3f129b3e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1865) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403096#comment-17403096 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * b0a203a766d22bd828505812029de9da3f129b3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1865) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403126#comment-17403126 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * b0a203a766d22bd828505812029de9da3f129b3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1865) * 912b00f0b4aea4e884a096d1a4c4811912bd428d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403138#comment-17403138 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * b0a203a766d22bd828505812029de9da3f129b3e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1865) * 912b00f0b4aea4e884a096d1a4c4811912bd428d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1867) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403139#comment-17403139 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 912b00f0b4aea4e884a096d1a4c4811912bd428d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1867) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403466#comment-17403466 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 912b00f0b4aea4e884a096d1a4c4811912bd428d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1867) * e74aabacfbf507524476bc081e882ac98419c5cb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403474#comment-17403474 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 912b00f0b4aea4e884a096d1a4c4811912bd428d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1867) * e74aabacfbf507524476bc081e882ac98419c5cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1869) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17403502#comment-17403502 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * e74aabacfbf507524476bc081e882ac98419c5cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1869) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17406679#comment-17406679 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * e74aabacfbf507524476bc081e882ac98419c5cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1869) * 96c2766c17379ebb3743f7d4d28ed93a5aee67f8 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17406690#comment-17406690 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * e74aabacfbf507524476bc081e882ac98419c5cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1869) * 96c2766c17379ebb3743f7d4d28ed93a5aee67f8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1946) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17406704#comment-17406704 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 96c2766c17379ebb3743f7d4d28ed93a5aee67f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1946) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407033#comment-17407033 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 96c2766c17379ebb3743f7d4d28ed93a5aee67f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1946) * 15fd5d70f8c0ce86883a7a59b4ac90d1ba6b3df9 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407038#comment-17407038 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 96c2766c17379ebb3743f7d4d28ed93a5aee67f8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1946) * 15fd5d70f8c0ce86883a7a59b4ac90d1ba6b3df9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1953) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407056#comment-17407056 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 15fd5d70f8c0ce86883a7a59b4ac90d1ba6b3df9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1953) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17407397#comment-17407397 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[
https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1740#comment-1740
]
ASF GitHub Bot commented on HUDI-648:
-
pratyakshsharma commented on a change in pull request #3312:
URL: https://github.com/apache/hudi/pull/3312#discussion_r702302452
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##
@@ -315,6 +315,17 @@ public void rollbackFailedBootstrap() {
*/
public abstract O insert(I records, final String instantTime);
+ /**
+ * Inserts the given Error HoodieRecords, into the table. This API is
intended to be used for normal writes.
+ *
+ * This API is intended to be used to write wrong records.
+ *
+ * @param records Error HoodieRecords to insert
+ * @param instantTime Instant time of the commit
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public abstract O insertError(I records, final String instantTime);
Review comment:
So this api inserts error records into a separate error table or the
regular Hudi table? The java doc seems to be misleading. Can we mention into
the error table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction
> writes
>
>
> Key: HUDI-648
> URL: https://issues.apache.org/jira/browse/HUDI-648
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer, Spark Integration, Writer Core
>Reporter: Vinoth Chandar
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:normal, user-support-issues
> Attachments: image-2021-03-03-11-40-21-083.png
>
>
> We would like a way to hand the erroring records from writing or compaction
> back to the users, in a separate table or log. This needs to work generically
> across all the different writer paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[
https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1741#comment-1741
]
ASF GitHub Bot commented on HUDI-648:
-
pratyakshsharma commented on a change in pull request #3312:
URL: https://github.com/apache/hudi/pull/3312#discussion_r702303108
##
File path:
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##
@@ -724,6 +725,21 @@ object DataSourceWriteOptions {
val KAFKA_AVRO_VALUE_DESERIALIZER = KAFKA_AVRO_VALUE_DESERIALIZER_CLASS.key()
@Deprecated
val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"
+
+ val ERROR_TABLE_ENABLE_OPT_KEY: ConfigProperty[String] = ConfigProperty
+.key("hoodie.write.error.table.enabled")
+.defaultValue("false")
+.withDocumentation("Enable the internal Error Table which saves error
record");
+
+ val HOODIE_WRITE_STATUS_CLASS_OPT_KEY: ConfigProperty[String] =
ConfigProperty
+.key("hoodie.writestatus.class")
+.defaultValue(classOf[WriteStatus].getName)
+.withDocumentation("");
+
+ val WRITESTATUS_FAILURE_FRACTION_OPT_KEY: ConfigProperty[String] =
ConfigProperty
+.key("hoodie.memory.writestatus.failure.fraction")
+.defaultValue("")
+.withDocumentation("");
Review comment:
Can we add documentation here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction
> writes
>
>
> Key: HUDI-648
> URL: https://issues.apache.org/jira/browse/HUDI-648
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer, Spark Integration, Writer Core
>Reporter: Vinoth Chandar
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:normal, user-support-issues
> Attachments: image-2021-03-03-11-40-21-083.png
>
>
> We would like a way to hand the erroring records from writing or compaction
> back to the users, in a separate table or log. This needs to work generically
> across all the different writer paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410287#comment-17410287 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 15fd5d70f8c0ce86883a7a59b4ac90d1ba6b3df9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1953) * 2e29817bf2b771333f14b276e55c34e873915f09 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[
https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410289#comment-17410289
]
ASF GitHub Bot commented on HUDI-648:
-
liujinhui1994 commented on a change in pull request #3312:
URL: https://github.com/apache/hudi/pull/3312#discussion_r702519243
##
File path:
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##
@@ -724,6 +725,21 @@ object DataSourceWriteOptions {
val KAFKA_AVRO_VALUE_DESERIALIZER = KAFKA_AVRO_VALUE_DESERIALIZER_CLASS.key()
@Deprecated
val SCHEMA_PROVIDER_CLASS_PROP = "hoodie.deltastreamer.schemaprovider.class"
+
+ val ERROR_TABLE_ENABLE_OPT_KEY: ConfigProperty[String] = ConfigProperty
+.key("hoodie.write.error.table.enabled")
+.defaultValue("false")
+.withDocumentation("Enable the internal Error Table which saves error
record");
+
+ val HOODIE_WRITE_STATUS_CLASS_OPT_KEY: ConfigProperty[String] =
ConfigProperty
+.key("hoodie.writestatus.class")
+.defaultValue(classOf[WriteStatus].getName)
+.withDocumentation("");
+
+ val WRITESTATUS_FAILURE_FRACTION_OPT_KEY: ConfigProperty[String] =
ConfigProperty
+.key("hoodie.memory.writestatus.failure.fraction")
+.defaultValue("")
+.withDocumentation("");
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction
> writes
>
>
> Key: HUDI-648
> URL: https://issues.apache.org/jira/browse/HUDI-648
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer, Spark Integration, Writer Core
>Reporter: Vinoth Chandar
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:normal, user-support-issues
> Attachments: image-2021-03-03-11-40-21-083.png
>
>
> We would like a way to hand the erroring records from writing or compaction
> back to the users, in a separate table or log. This needs to work generically
> across all the different writer paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[
https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410290#comment-17410290
]
ASF GitHub Bot commented on HUDI-648:
-
liujinhui1994 commented on a change in pull request #3312:
URL: https://github.com/apache/hudi/pull/3312#discussion_r702519527
##
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##
@@ -315,6 +315,17 @@ public void rollbackFailedBootstrap() {
*/
public abstract O insert(I records, final String instantTime);
+ /**
+ * Inserts the given Error HoodieRecords, into the table. This API is
intended to be used for normal writes.
+ *
+ * This API is intended to be used to write wrong records.
+ *
+ * @param records Error HoodieRecords to insert
+ * @param instantTime Instant time of the commit
+ * @return Collection of WriteStatus to inspect errors and counts
+ */
+ public abstract O insertError(I records, final String instantTime);
Review comment:
sure，done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction
> writes
>
>
> Key: HUDI-648
> URL: https://issues.apache.org/jira/browse/HUDI-648
> Project: Apache Hudi
> Issue Type: New Feature
> Components: DeltaStreamer, Spark Integration, Writer Core
>Reporter: Vinoth Chandar
>Assignee: liujinhui
>Priority: Major
> Labels: pull-request-available, sev:normal, user-support-issues
> Attachments: image-2021-03-03-11-40-21-083.png
>
>
> We would like a way to hand the erroring records from writing or compaction
> back to the users, in a separate table or log. This needs to work generically
> across all the different writer paths.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410295#comment-17410295 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 15fd5d70f8c0ce86883a7a59b4ac90d1ba6b3df9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1953) * 2e29817bf2b771333f14b276e55c34e873915f09 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2044) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17410318#comment-17410318 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 2e29817bf2b771333f14b276e55c34e873915f09 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2044) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411051#comment-17411051 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-914125738 @hudi-bot travies -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411052#comment-17411052 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-914126972 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411053#comment-17411053 ] ASF GitHub Bot commented on HUDI-648: - liujinhui1994 removed a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-914125738 @hudi-bot travies -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411062#comment-17411062 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 2e29817bf2b771333f14b276e55c34e873915f09 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2044) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2067) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411091#comment-17411091 ] ASF GitHub Bot commented on HUDI-648: - hudi-bot edited a comment on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-883316285 ## CI report: * 58a46294a5ae693281d543a86af4785e09decd8f UNKNOWN * 3b88d82271c25fa3db151b6560e97879d27c7119 UNKNOWN * 2e29817bf2b771333f14b276e55c34e873915f09 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2044) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2067) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411515#comment-17411515 ] ASF GitHub Bot commented on HUDI-648: - vinothchandar commented on pull request #3312: URL: https://github.com/apache/hudi/pull/3312#issuecomment-914642326 @liujinhui1994 taking this up now. lets work together to land this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Assignee: liujinhui >Priority: Major > Labels: pull-request-available, sev:normal, user-support-issues > Attachments: image-2021-03-03-11-40-21-083.png > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-648) Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction writes
[ https://issues.apache.org/jira/browse/HUDI-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062242#comment-17062242 ] liujinhui commented on HUDI-648: Hello, I also encountered this problem recently. Occasionally the kafka data received by hudi is wrong, the consumption is reported wrong, and it needs to be skipped. It seems that it cannot be skipped at the moment. Do you have a solution for this? [~vinoth] > Implement error log/table for Datasource/DeltaStreamer/WriteClient/Compaction > writes > > > Key: HUDI-648 > URL: https://issues.apache.org/jira/browse/HUDI-648 > Project: Apache Hudi (incubating) > Issue Type: New Feature > Components: DeltaStreamer, Spark Integration, Writer Core >Reporter: Vinoth Chandar >Priority: Major > Fix For: 0.6.0 > > > We would like a way to hand the erroring records from writing or compaction > back to the users, in a separate table or log. This needs to work generically > across all the different writer paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
