[
https://issues.apache.org/jira/browse/GOBBLIN-1584?focusedWorklogId=692128&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692128
]
ASF GitHub Bot logged work on GOBBLIN-1584:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 07/Dec/21 22:43
Start Date: 07/Dec/21 22:43
Worklog Time Spent: 10m
Work Description: umustafi commented on a change in pull request #3438:
URL: https://github.com/apache/gobblin/pull/3438#discussion_r764417093
##########
File path:
gobblin-modules/gobblin-sql/src/main/java/org/apache/gobblin/writer/commands/BaseJdbcBufferedInserter.java
##########
@@ -65,6 +67,9 @@
protected PreparedStatement insertPstmtForFixedBatch;
private final Retryer<Boolean> retryer;
+ // If this config is true, the inserter can insert duplicate primary records
according to the specific language
+ protected final boolean replaceExistingValues;
Review comment:
fair argument, I moved this field to the base class of
`MySqlBufferedInserter` but not the other child classes because it is the only
one that implements this logic so far. As per your comment below I have added
it as a field to all the `*WriterCommands` that will fail fast if the config is
turned on before creating the `*Inserter`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 692128)
Time Spent: 3h 20m (was: 3h 10m)
> Add Replace Logic To Mysql Writer
> ---------------------------------
>
> Key: GOBBLIN-1584
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1584
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> We only support {{insertion}} of new values into tables, not
> {{{}update/upsert{}}}. If you run an ingestion job with a record containing a
> record with the same primary key the whole job will fail because of the
> duplicate entry. We don't expect ingestion jobs to always contain _only new_
> records, so we should handle duplicate entries, ingestion of data already in
> the table or updates to old values. The insert vs. replace logic should be
> configurable to the user as well.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
