[jira] [Created] (GOBBLIN-865) Add feature that enables PK-chunking in partition
Alex Li created GOBBLIN-865: --- Summary: Add feature that enables PK-chunking in partition Key: GOBBLIN-865 URL: https://issues.apache.org/jira/browse/GOBBLIN-865 Project: Apache Gobblin Issue Type: Task Reporter: Alex Li In SFDC(salesforce) connector, we have partitioning mechanisms to split a giant query to multiple sub queries. There are 3 mechanisms: * simple partition (equally split by time) * dynamic pre-partition (generate histogram and split by row numbers) * user specified partition (set up time range in job file) However there are tables like Task and Contract are failing time to time to fetch full data. We may want to utilize PK-chunking to partition the query. The pk-chunking doc from SFDC - [https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (GOBBLIN-995) Add function to instantiate the BulkConnection in SFDC connector
Alex Li created GOBBLIN-995: --- Summary: Add function to instantiate the BulkConnection in SFDC connector Key: GOBBLIN-995 URL: https://issues.apache.org/jira/browse/GOBBLIN-995 Project: Apache Gobblin Issue Type: New Feature Reporter: Alex Li In SalesforceExtractor class, we instantiated BulkConnection directly. {code:java} this.bulkConnection = new BulkConnection(config); {code} This code makes it is impossible to inject a customized BulkConnection. In contrast, httpClient was instantiated in a function. We could extend the class and override the function to return a customized httpClient (GaapHttPClient in our case) We should add a function to instantiate the BulkConnection. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1025) Add retry for PK-Chunking iterator
Alex Li created GOBBLIN-1025: Summary: Add retry for PK-Chunking iterator Key: GOBBLIN-1025 URL: https://issues.apache.org/jira/browse/GOBBLIN-1025 Project: Apache Gobblin Issue Type: Improvement Reporter: Alex Li In SFDC connector, there is a class called `ResultIterator` (I will change the name to SalesforceRecordIterator). It was using by only PK-Chunking currently. It encapsulated fetching a list of result files to a record iterator. However, the csvReader.nextRecord() may throw out network IO exception. We should do retry in this case. When a result file is fetched partly and one network IO exception happens, we are in a special situation - first half of the file is already fetched to our local, but another half of the file is still on datasource. We need to 1. reopen the file stream 2. skip all the records that we already fetched, seek the cursor to the record which we haven't fetched yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1070) gobblin json converter not able to take in union with more than 2 types
Alex Li created GOBBLIN-1070: Summary: gobblin json converter not able to take in union with more than 2 types Key: GOBBLIN-1070 URL: https://issues.apache.org/jira/browse/GOBBLIN-1070 Project: Apache Gobblin Issue Type: Improvement Reporter: Alex Li gobblin-core has a converter that is hard-coded to take only 2 types if the field type is union. The ideal behavior is to support an arbitrary number of types. This is blocking Zendesk's tickets dataset from being ingested as there's a column that has 4+types. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1097) ResultChainingIterator.add should check if the argument iterator is null
Alex Li created GOBBLIN-1097: Summary: ResultChainingIterator.add should check if the argument iterator is null Key: GOBBLIN-1097 URL: https://issues.apache.org/jira/browse/GOBBLIN-1097 Project: Apache Gobblin Issue Type: Bug Reporter: Alex Li ResultChainingIterator.add should check if the argument iterator is null. It fails, if the argument is null. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1101) Enhance bulk api retry for ExceedQuota
Alex Li created GOBBLIN-1101: Summary: Enhance bulk api retry for ExceedQuota Key: GOBBLIN-1101 URL: https://issues.apache.org/jira/browse/GOBBLIN-1101 Project: Apache Gobblin Issue Type: Bug Reporter: Alex Li 1. ExceedQuota exception Below is SFDC doc about ExceedQuota {code:java} One of the limits customers frequently reach is the concurrent request limit. Once a synchronous Apex request runs longer than 5 seconds, it begins counting against this limit. Each organization is allowed 10 concurrent long-running requests. If the limit is reached, any new synchronous Apex request results in a runtime exception. This behavior occurs until the organization’s requests are below the limit. {code} If the ExceedQuota exception happens, we should let the thread sleep 5 minutes and try again. There should not be a retryLimit for this exception. 2. Except stack in log file For example we set up retryLimit to 10, we retried 10 times, and failed; we need to print out exception stack in log file, there are 10 of them in the exception stack. SSL Exception(root cause) retry and get > ExceedQuota retry and get > ExceedQuota a lot > We'd better skip all the retry exception, only keep the root cause exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1179) Add a typed config to replace properties
Alex Li created GOBBLIN-1179: Summary: Add a typed config to replace properties Key: GOBBLIN-1179 URL: https://issues.apache.org/jira/browse/GOBBLIN-1179 Project: Apache Gobblin Issue Type: Task Reporter: Alex Li Add a typed config to replace *_properties.get(“ini.file.userName”)_* with *config.userName* The gobblin config file is an ini file. Java loads the ini file to a properties instance. The way to use the config information is to get from the properties. |workUnitState.getPropAsBoolean(BULK_API_USE_QUERY_ALL)| |workUnitState.getPropAsInt(FETCH_RETRY_LIMIT_KEY, DEFAULT_FETCH_RETRY_LIMIT)| |Math.max(MIN_SIZE,Math.min(MAX_SIZE, workUnitState.getPropAsInt(PARTITION_SIZE, DEFAULT_SIZE))); // partition size must be >= min and <= max, otherwise use default| Problems # No consistent key naming model * A long dot-separated key string is used, easy to run into typos. The config code is pretty verbose: We use *properties.getProp(key, default)* * Key collision if the same type is used in multiple places, e.g kafka.brokers # No ownership management * in gobblin connector package: We have multiple constant static classes. GobblinKeys, QueryBaseKeys, GaapKeys, and SalesforceConnectorKeys.We can even directly read config values by state.getProp(*“my.key”*) without creating any constant key. # No static validation * Required & default value * Type check * Date range * Enum # No dependency check * If users set to *useGaap=true*, there must be *gaap.url* and *gaap.credential*. And this needs to be verified at both runtime and compile time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1186) Fix SFDC source.querybased.salesforce.is.soft.deletes.pull.disabled not available for simple mode
Alex Li created GOBBLIN-1186: Summary: Fix SFDC source.querybased.salesforce.is.soft.deletes.pull.disabled not available for simple mode Key: GOBBLIN-1186 URL: https://issues.apache.org/jira/browse/GOBBLIN-1186 Project: Apache Gobblin Issue Type: Bug Reporter: Alex Li *Problem statement* source.querybased.salesforce.is.soft.deletes.pull.disabled doesn’t work for simple mode, it works only for dynamic mode. the reason is - we explicitly set up the key-value for the dynamic mode [https://github.com/hanghangliu/gobblin/blob/9029a89b85ef373f78d603b14d6aaa75998f3356/gobblin-salesforce/src/main/java/org/apache/gobblin/salesforce/SalesforceSource.java#L327] *Root cause* The extract state is blank(please see code) What we set up in job file is not able to see in extractor state. [https://github.com/hashdoop/hashdoop-incubator-gobblin/blob/a871e5c5d6f539bcfbcc4e2850685c58dd72dd1a/gobblin-core/src/main/java/org/apache/gobblin/source/extractor/extract/QueryBasedSource.java#L234] *Solution:* explicitly set up the {{soft.deletes.pull.disabled}} for simple mode, as we did for dynamic mode -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1188) Fix log message for SFDC iterators
Alex Li created GOBBLIN-1188: Summary: Fix log message for SFDC iterators Key: GOBBLIN-1188 URL: https://issues.apache.org/jira/browse/GOBBLIN-1188 Project: Apache Gobblin Issue Type: Bug Reporter: Alex Li We printed out same message twice. remove it from hasNext(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (GOBBLIN-1188) Fix log message for SFDC iterators
[ https://issues.apache.org/jira/browse/GOBBLIN-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Li updated GOBBLIN-1188: - Priority: Minor (was: Major) > Fix log message for SFDC iterators > -- > > Key: GOBBLIN-1188 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1188 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Alex Li >Priority: Minor > > We printed out same message twice. > remove it from hasNext(). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1202) Add retry for REST API call
Alex Li created GOBBLIN-1202: Summary: Add retry for REST API call Key: GOBBLIN-1202 URL: https://issues.apache.org/jira/browse/GOBBLIN-1202 Project: Apache Gobblin Issue Type: Improvement Reporter: Alex Li SFDC objects have index on their column - *SystemModstamp* This index could be in disk. When we execute {code:java} Select count(systemmodstamp) from table_name group by day_only(systemmodstamp) {code} If the index is in disk, it needs to load. It would be timeout. Retry would result it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (GOBBLIN-1208) Fix - restApiRetryLimit cannot be set to 0
[ https://issues.apache.org/jira/browse/GOBBLIN-1208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Li updated GOBBLIN-1208: - Description: Fix - restApiRetryLimit cannot be set to 0 if we set restApiRetryLimit=0, the code should be execute 1 time. change code: {code:java} private JsonArray getRecordsForQuery(SalesforceConnector connector, String query) { RestApiProcessingException exception = null; for (int i = 0; i < workUnitConf.restApiRetryLimit; i++) { {code} to {code:java} private JsonArray getRecordsForQuery(SalesforceConnector connector, String query) { RestApiProcessingException exception = null; for (int i = 0; i < workUnitConf.restApiRetryLimit+1; i++) { {code} was: Fix - restApiRetryLimit cannot be set to 0 if we set restApiRetryLimit=0, the code should be execute 1 time. change code: {code:java} private JsonArray getRecordsForQuery(SalesforceConnector connector, String query) { RestApiProcessingException exception = null; for (int i = 0; i < workUnitConf.restApiRetryLimit; i++) { {code} to {code:java} private JsonArray getRecordsForQuery(SalesforceConnector connector, String query) { RestApiProcessingException exception = null; for (int i = 0; i < workUnitConf.restApiRetryLimit+1; i++) { {code} > Fix - restApiRetryLimit cannot be set to 0 > -- > > Key: GOBBLIN-1208 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1208 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Alex Li >Priority: Major > > Fix - restApiRetryLimit cannot be set to 0 > if we set restApiRetryLimit=0, the code should be execute 1 time. > change code: > {code:java} > private JsonArray getRecordsForQuery(SalesforceConnector connector, String > query) { > RestApiProcessingException exception = null; > for (int i = 0; i < workUnitConf.restApiRetryLimit; i++) { > {code} > to > {code:java} > private JsonArray getRecordsForQuery(SalesforceConnector connector, String > query) { > RestApiProcessingException exception = null; > for (int i = 0; i < workUnitConf.restApiRetryLimit+1; i++) { > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1208) Fix - restApiRetryLimit cannot be set to 0
Alex Li created GOBBLIN-1208: Summary: Fix - restApiRetryLimit cannot be set to 0 Key: GOBBLIN-1208 URL: https://issues.apache.org/jira/browse/GOBBLIN-1208 Project: Apache Gobblin Issue Type: Improvement Reporter: Alex Li Fix - restApiRetryLimit cannot be set to 0 if we set restApiRetryLimit=0, the code should be execute 1 time. change code: {code:java} private JsonArray getRecordsForQuery(SalesforceConnector connector, String query) { RestApiProcessingException exception = null; for (int i = 0; i < workUnitConf.restApiRetryLimit; i++) { {code} to {code:java} private JsonArray getRecordsForQuery(SalesforceConnector connector, String query) { RestApiProcessingException exception = null; for (int i = 0; i < workUnitConf.restApiRetryLimit+1; i++) { {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)