[
https://issues.apache.org/jira/browse/GOBBLIN-1904?focusedWorklogId=880369&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-880369
]
ASF GitHub Bot logged work on GOBBLIN-1904:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 14/Sep/23 12:03
Start Date: 14/Sep/23 12:03
Worklog Time Spent: 10m
Work Description: gautamguptabasant commented on code in PR #3768:
URL: https://github.com/apache/gobblin/pull/3768#discussion_r1325845330
##########
gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java:
##########
@@ -277,11 +261,13 @@ private List<WorkUnit> createWorkUnits(
}
/**
- *
+ * Generates {@link WorkUnit}s based on a bunch of config values like max
number of partitions, early stop,
+ * dynamic partitioning, dynamic probing, etc.
*/
- private List<WorkUnit> generateWorkUnitsStrategy(SourceEntity sourceEntity,
SourceState state, long previousWatermark) {
- Boolean disableSoft =
state.getPropAsBoolean(SOURCE_QUERYBASED_SALESFORCE_IS_SOFT_DELETES_PULL_DISABLED,
false);
- log.info("disable soft delete pull: " + disableSoft);
+ @VisibleForTesting
+ List<WorkUnit> generateWorkUnitsHelper(SourceEntity sourceEntity,
SourceState state, long previousWatermark) {
+ boolean isSoftDeletePullDisabled =
state.getPropAsBoolean(SOURCE_QUERYBASED_SALESFORCE_IS_SOFT_DELETES_PULL_DISABLED,
false);
+ log.info("disable soft delete pull: " + isSoftDeletePullDisabled);
Review Comment:
Yes, the current state is a not great - we have some of these configs in
SfConfig, some keys defined in SalesforceConfigurationKeys and some as class
constants.
Will take sorting this up as part part of a separate PR.
Issue Time Tracking
-------------------
Worklog Id: (was: 880369)
Time Spent: 2h (was: 1h 50m)
> Refactor Salesforce Source to promote testability
> -------------------------------------------------
>
> Key: GOBBLIN-1904
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1904
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-core
> Reporter: William Lo
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Salesforce source is difficult to test due to many internal classes that
> cannot mock inputs and outputs. We should clean this class and use
> composition to promote testability.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
