[
https://issues.apache.org/jira/browse/GOBBLIN-1080?focusedWorklogId=404795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404795
]
ASF GitHub Bot logged work on GOBBLIN-1080:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 17/Mar/20 15:30
Start Date: 17/Mar/20 15:30
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2925:
[GOBBLIN-1080] Add configuration to add schema creation time in converter
URL: https://github.com/apache/incubator-gobblin/pull/2925#discussion_r393765960
##########
File path:
gobblin-core-base/src/main/java/org/apache/gobblin/converter/filter/AvroProjectionConverter.java
##########
@@ -74,10 +74,15 @@ public AvroProjectionConverter init(WorkUnitState
workUnit) {
*/
@Override
public Schema convertSchema(Schema inputSchema, WorkUnitState workUnit)
throws SchemaConversionException {
+ Schema outputSchema = inputSchema;
if (this.fieldRemover.isPresent()) {
- return this.fieldRemover.get().removeFields(inputSchema);
+ outputSchema = this.fieldRemover.get().removeFields(inputSchema);
}
- return inputSchema;
+
if(workUnit.getPropAsBoolean(ConfigurationKeys.CONVERTER_AVRO_INCLUDE_SCHEMA_CREATION_TIME,
+
ConfigurationKeys.DEFAULT_CONVERTER_AVRO_INCLUDE_SCHEMA_CREATION_TIME)) {
Review comment:
Given the timestamp is set in the properties of an Avro schema object, does
that matter for other writers that are not aware of the existence of this
property ? We can try with a simple unit test to see if setting properties will
impact writer.
I mostly concern on the operability problem since there's no guarantee that
this converter is and will used everywhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 404795)
Time Spent: 2.5h (was: 2h 20m)
> Add configuration to preserve schema creation time in converter
> ---------------------------------------------------------------
>
> Key: GOBBLIN-1080
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1080
> Project: Apache Gobblin
> Issue Type: Task
> Reporter: Zihan Li
> Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
