Wenqin Wang created GOBBLIN-1323:
------------------------------------
Summary: make columnListInQuery in RestApiExtractor overridable in
child class for Dynamics Extractor
Key: GOBBLIN-1323
URL: https://issues.apache.org/jira/browse/GOBBLIN-1323
Project: Apache Gobblin
Issue Type: Bug
Components: gobblin-core
Reporter: Wenqin Wang
Assignee: Abhishek Tiwari
Recently we've discovered that in our Dynamics ingestion pipeline, we would not
remove invalid fields in our query before pulling data for dynamics entities,
and therefore gobblin ingestion would fail due to field not found error. This
feature is implemented for Salesforce in RestApiExtrator.java, and the logic is
specifically tailored for Salesforce. Therefore i'm proposing some changes and
hope to get Gobblin dev's guidance and reviews.
For Dynamics, to enable this feature, here is the solution that I think works
the best:
since the query looks different from Salesforce query, we need to change the
way we extract fields from client-specified-query. I will make a protected
method in RestApiExtractor.java to wrap this Utils.getColumnListFromQuery, and
override it in DynamicsExtractor.java (the child class) to write specific logic
for dynamics use case. After that, we can override the buildDataQuery method in
DynamicsExtractor.java ourselves to replace user-specified-query with validated
fields. This way it would not affect other extractors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
