[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337669&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337669
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 02/Nov/19 06:20
Start Date: 02/Nov/19 06:20
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341800826
 
 

 ##
 File path: 
gobblin-core/src/test/java/org/apache/gobblin/converter/avro/JsonRecordAvroSchemaToAvroConverterTest.java
 ##
 @@ -85,5 +85,17 @@ public void testConverter()
 Assert.assertEquals(recordInArray.get("field1").toString(), "test1");
 
 Assert.assertEquals((record.get("enumField")).toString(), "ENUM2");
+
+Assert.assertTrue(record.get("arrayFieldWithUnion") instanceof 
GenericArray);
 
 Review comment:
   Please add more cases for this converter which is pretty unit-testable.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337669)
Time Spent: 1h 40m  (was: 1.5h)

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:160)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

autumnust commented on a change in pull request #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341800826
 
 

 ##
 File path: 
gobblin-core/src/test/java/org/apache/gobblin/converter/avro/JsonRecordAvroSchemaToAvroConverterTest.java
 ##
 @@ -85,5 +85,17 @@ public void testConverter()
 Assert.assertEquals(recordInArray.get("field1").toString(), "test1");
 
 Assert.assertEquals((record.get("enumField")).toString(), "ENUM2");
+
+Assert.assertTrue(record.get("arrayFieldWithUnion") instanceof 
GenericArray);
 
 Review comment:
   Please add more cases for this converter which is pretty unit-testable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

autumnust commented on a change in pull request #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341800803
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   +1 
   
   BTW we should try to stay aligned with Avro schema guide to always place 
null in the first to indicate nullable schema. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337670&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337670
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 02/Nov/19 06:20
Start Date: 02/Nov/19 06:20
Worklog Time Spent: 10m 
  Work Description: autumnust commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341800803
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   +1 
   
   BTW we should try to stay aligned with Avro schema guide to always place 
null in the first to indicate nullable schema. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337670)
Time Spent: 1h 40m  (was: 1.5h)

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:160)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337652&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337652
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 02/Nov/19 03:16
Start Date: 02/Nov/19 03:16
Worklog Time Spent: 10m 
  Work Description: ahmedahamid commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341779189
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   Just following up: I tried adding a `null` value to this array but the 
`JsonRecordAvroSchemaToAvroConverterTest.testConverter` failed:
   ```java
   Could not convert field arrayFieldWithUnion
   org.apache.gobblin.converter.DataConversionException: Could not convert 
field arrayFieldWithUnion
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:127)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertRecord(JsonRecordAvroSchemaToAvroConverter.java:71)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverterTest.testConverter(JsonRecordAvroSchemaToAvroConverterTest.java:69)
   ...
   Caused by: java.lang.RuntimeException: Field: arrayFieldWithUnion is not 
nullable and contains a null value
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:278)
at 
org.apache.gobblin.converter.avro.JsonElementConversionWithAvroSchemaFactory$ArrayConverter.convertField(JsonElementConversionWithAvroSchemaFactory.java:98)
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:280)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:125)
... 51 more
   ```
   
   I think the key issue here is that `UnionConverter` does not determine its 
nullability correctly; it uses the nullability it receives from 
`JsonElementConversionWithAvroSchemaFactory.getConvertor()` to decide its own 
nullability. This is incorrect because even though `arrayFieldWithUnion` is a 
non-nullable array, the union within its `items` _is_ nullable, i.e. 
`UnionConverter` would return `false` if you invoke `isNullable()` on it even 
though `["string", "null"]` is nullable because it is construing 
`arrayFieldWithUnion`'s non-nullability as  its own (see the values 
`UnionConverter` provides to `super`).
   
   One possible fix could be to have `UnionConverter` override `isNullable()` 
so it can decide its own nullability based on its schemas, e.g.
   
   ```java
   @Override
   public boolean isNullable() {
 return this.firstSchema.getType() == Schema.Type.NULL ||
 this.secondSchema.getType() == Schema.Type.NULL;
   }
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337652)
Time Spent: 1.5h  (was: 1h 20m)

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 

[GitHub] [incubator-gobblin] ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341779189
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   Just following up: I tried adding a `null` value to this array but the 
`JsonRecordAvroSchemaToAvroConverterTest.testConverter` failed:
   ```java
   Could not convert field arrayFieldWithUnion
   org.apache.gobblin.converter.DataConversionException: Could not convert 
field arrayFieldWithUnion
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:127)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertRecord(JsonRecordAvroSchemaToAvroConverter.java:71)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverterTest.testConverter(JsonRecordAvroSchemaToAvroConverterTest.java:69)
   ...
   Caused by: java.lang.RuntimeException: Field: arrayFieldWithUnion is not 
nullable and contains a null value
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:278)
at 
org.apache.gobblin.converter.avro.JsonElementConversionWithAvroSchemaFactory$ArrayConverter.convertField(JsonElementConversionWithAvroSchemaFactory.java:98)
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:280)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:125)
... 51 more
   ```
   
   I think the key issue here is that `UnionConverter` does not determine its 
nullability correctly; it uses the nullability it receives from 
`JsonElementConversionWithAvroSchemaFactory.getConvertor()` to decide its own 
nullability. This is incorrect because even though `arrayFieldWithUnion` is a 
non-nullable array, the union within its `items` _is_ nullable, i.e. 
`UnionConverter` would return `false` if you invoke `isNullable()` on it even 
though `["string", "null"]` is nullable because it is construing 
`arrayFieldWithUnion`'s non-nullability as  its own (see the values 
`UnionConverter` provides to `super`).
   
   One possible fix could be to have `UnionConverter` override `isNullable()` 
so it can decide its own nullability based on its schemas, e.g.
   
   ```java
   @Override
   public boolean isNullable() {
 return this.firstSchema.getType() == Schema.Type.NULL ||
 this.secondSchema.getType() == Schema.Type.NULL;
   }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337650&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337650
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 02/Nov/19 03:13
Start Date: 02/Nov/19 03:13
Worklog Time Spent: 10m 
  Work Description: ahmedahamid commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341779189
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   Just following up. I tried adding a `null` value to this array but the 
`JsonRecordAvroSchemaToAvroConverterTest.testConverter` failed:
   ```java
   Could not convert field arrayFieldWithUnion
   org.apache.gobblin.converter.DataConversionException: Could not convert 
field arrayFieldWithUnion
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:127)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertRecord(JsonRecordAvroSchemaToAvroConverter.java:71)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverterTest.testConverter(JsonRecordAvroSchemaToAvroConverterTest.java:69)
   ...
   Caused by: java.lang.RuntimeException: Field: arrayFieldWithUnion is not 
nullable and contains a null value
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:278)
at 
org.apache.gobblin.converter.avro.JsonElementConversionWithAvroSchemaFactory$ArrayConverter.convertField(JsonElementConversionWithAvroSchemaFactory.java:98)
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:280)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:125)
... 51 more
   ```
   
   I think the key issue here is that `UnionConverter` does not determine its 
nullability correctly; it uses the nullability it receives from 
`JsonElementConversionWithAvroSchemaFactory.getConvertor()` to decide its own 
nullability. This is incorrect because `arrayFieldWithUnion` is a non-nullable 
array. However, the union itself (i.e. `UnionConverter`) would return `false` 
if you invoke `isNullable()` on it even though `["string", "null"]` _is_ 
nullable (because it is construing `arrayFieldWithUnion`'s non-nullability as  
its own (see the values `UnionConverter` provides to `super`).
   
   One possible fix could be to have `UnionConverter` override `isNullable()` 
so it can decide its own nullability based on its schemas, e.g.
   
   ```java
   @Override
   public boolean isNullable() {
 return this.firstSchema.getType() == Schema.Type.NULL ||
 this.secondSchema.getType() == Schema.Type.NULL;
   }
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337650)
Time Spent: 1h 10m  (was: 1h)

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.conv

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337651&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337651
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 02/Nov/19 03:13
Start Date: 02/Nov/19 03:13
Worklog Time Spent: 10m 
  Work Description: ahmedahamid commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341779189
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   Just following up: I tried adding a `null` value to this array but the 
`JsonRecordAvroSchemaToAvroConverterTest.testConverter` failed:
   ```java
   Could not convert field arrayFieldWithUnion
   org.apache.gobblin.converter.DataConversionException: Could not convert 
field arrayFieldWithUnion
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:127)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertRecord(JsonRecordAvroSchemaToAvroConverter.java:71)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverterTest.testConverter(JsonRecordAvroSchemaToAvroConverterTest.java:69)
   ...
   Caused by: java.lang.RuntimeException: Field: arrayFieldWithUnion is not 
nullable and contains a null value
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:278)
at 
org.apache.gobblin.converter.avro.JsonElementConversionWithAvroSchemaFactory$ArrayConverter.convertField(JsonElementConversionWithAvroSchemaFactory.java:98)
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:280)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:125)
... 51 more
   ```
   
   I think the key issue here is that `UnionConverter` does not determine its 
nullability correctly; it uses the nullability it receives from 
`JsonElementConversionWithAvroSchemaFactory.getConvertor()` to decide its own 
nullability. This is incorrect because `arrayFieldWithUnion` is a non-nullable 
array. However, the union itself (i.e. `UnionConverter`) would return `false` 
if you invoke `isNullable()` on it even though `["string", "null"]` _is_ 
nullable (because it is construing `arrayFieldWithUnion`'s non-nullability as  
its own (see the values `UnionConverter` provides to `super`).
   
   One possible fix could be to have `UnionConverter` override `isNullable()` 
so it can decide its own nullability based on its schemas, e.g.
   
   ```java
   @Override
   public boolean isNullable() {
 return this.firstSchema.getType() == Schema.Type.NULL ||
 this.secondSchema.getType() == Schema.Type.NULL;
   }
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337651)
Time Spent: 1h 20m  (was: 1h 10m)

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.

[GitHub] [incubator-gobblin] ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341779189
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   Just following up: I tried adding a `null` value to this array but the 
`JsonRecordAvroSchemaToAvroConverterTest.testConverter` failed:
   ```java
   Could not convert field arrayFieldWithUnion
   org.apache.gobblin.converter.DataConversionException: Could not convert 
field arrayFieldWithUnion
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:127)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertRecord(JsonRecordAvroSchemaToAvroConverter.java:71)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverterTest.testConverter(JsonRecordAvroSchemaToAvroConverterTest.java:69)
   ...
   Caused by: java.lang.RuntimeException: Field: arrayFieldWithUnion is not 
nullable and contains a null value
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:278)
at 
org.apache.gobblin.converter.avro.JsonElementConversionWithAvroSchemaFactory$ArrayConverter.convertField(JsonElementConversionWithAvroSchemaFactory.java:98)
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:280)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:125)
... 51 more
   ```
   
   I think the key issue here is that `UnionConverter` does not determine its 
nullability correctly; it uses the nullability it receives from 
`JsonElementConversionWithAvroSchemaFactory.getConvertor()` to decide its own 
nullability. This is incorrect because `arrayFieldWithUnion` is a non-nullable 
array. However, the union itself (i.e. `UnionConverter`) would return `false` 
if you invoke `isNullable()` on it even though `["string", "null"]` _is_ 
nullable (because it is construing `arrayFieldWithUnion`'s non-nullability as  
its own (see the values `UnionConverter` provides to `super`).
   
   One possible fix could be to have `UnionConverter` override `isNullable()` 
so it can decide its own nullability based on its schemas, e.g.
   
   ```java
   @Override
   public boolean isNullable() {
 return this.firstSchema.getType() == Schema.Type.NULL ||
 this.secondSchema.getType() == Schema.Type.NULL;
   }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341779189
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   Just following up. I tried adding a `null` value to this array but the 
`JsonRecordAvroSchemaToAvroConverterTest.testConverter` failed:
   ```java
   Could not convert field arrayFieldWithUnion
   org.apache.gobblin.converter.DataConversionException: Could not convert 
field arrayFieldWithUnion
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:127)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertRecord(JsonRecordAvroSchemaToAvroConverter.java:71)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverterTest.testConverter(JsonRecordAvroSchemaToAvroConverterTest.java:69)
   ...
   Caused by: java.lang.RuntimeException: Field: arrayFieldWithUnion is not 
nullable and contains a null value
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:278)
at 
org.apache.gobblin.converter.avro.JsonElementConversionWithAvroSchemaFactory$ArrayConverter.convertField(JsonElementConversionWithAvroSchemaFactory.java:98)
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:280)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:125)
... 51 more
   ```
   
   I think the key issue here is that `UnionConverter` does not determine its 
nullability correctly; it uses the nullability it receives from 
`JsonElementConversionWithAvroSchemaFactory.getConvertor()` to decide its own 
nullability. This is incorrect because `arrayFieldWithUnion` is a non-nullable 
array. However, the union itself (i.e. `UnionConverter`) would return `false` 
if you invoke `isNullable()` on it even though `["string", "null"]` _is_ 
nullable (because it is construing `arrayFieldWithUnion`'s non-nullability as  
its own (see the values `UnionConverter` provides to `super`).
   
   One possible fix could be to have `UnionConverter` override `isNullable()` 
so it can decide its own nullability based on its schemas, e.g.
   
   ```java
   @Override
   public boolean isNullable() {
 return this.firstSchema.getType() == Schema.Type.NULL ||
 this.secondSchema.getType() == Schema.Type.NULL;
   }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337646&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337646
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 02/Nov/19 02:09
Start Date: 02/Nov/19 02:09
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: 
https://github.com/apache/incubator-gobblin/pull/2792#issuecomment-548978202
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=h1)
 Report
   > Merging 
[#2792](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/94a508b38ec8bd879614f2d9bf0eeb96513ca7cf?src=pr&el=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `62.5%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2792  +/-   ##
   
   + Coverage 45.32%   45.34%   +0.01% 
   - Complexity 8862 8865   +3 
   
 Files  1894 1894  
 Lines 7091070926  +16 
 Branches   7799 7801   +2 
   
   + Hits  3214132160  +19 
   + Misses3580335800   -3 
 Partials   2966 2966
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...ro/JsonElementConversionWithAvroSchemaFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9hdnJvL0pzb25FbGVtZW50Q29udmVyc2lvbldpdGhBdnJvU2NoZW1hRmFjdG9yeS5qYXZh)
 | `72.36% <62.5%> (-2.64%)` | `6 <0> (+1)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `64.35% <0%> (+0.92%)` | `27% <0%> (ø)` | :arrow_down: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `39.25% <0%> (+3.73%)` | `13% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `100% <0%> (+14.28%)` | `4% <0%> (+1%)` | :arrow_up: |
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `60% <0%> (+20%)` | `3% <0%> (+1%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=footer).
 Last update 
[94a508b...6e107ea](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   
 

This is an automated message fro

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

codecov-io edited a comment on issue #2792: [GOBBLIN-933] add support for array 
of unions in json schema
URL: 
https://github.com/apache/incubator-gobblin/pull/2792#issuecomment-548978202
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=h1)
 Report
   > Merging 
[#2792](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/94a508b38ec8bd879614f2d9bf0eeb96513ca7cf?src=pr&el=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `62.5%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2792  +/-   ##
   
   + Coverage 45.32%   45.34%   +0.01% 
   - Complexity 8862 8865   +3 
   
 Files  1894 1894  
 Lines 7091070926  +16 
 Branches   7799 7801   +2 
   
   + Hits  3214132160  +19 
   + Misses3580335800   -3 
 Partials   2966 2966
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...ro/JsonElementConversionWithAvroSchemaFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9hdnJvL0pzb25FbGVtZW50Q29udmVyc2lvbldpdGhBdnJvU2NoZW1hRmFjdG9yeS5qYXZh)
 | `72.36% <62.5%> (-2.64%)` | `6 <0> (+1)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `64.35% <0%> (+0.92%)` | `27% <0%> (ø)` | :arrow_down: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `39.25% <0%> (+3.73%)` | `13% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `100% <0%> (+14.28%)` | `4% <0%> (+1%)` | :arrow_up: |
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `60% <0%> (+20%)` | `3% <0%> (+1%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=footer).
 Last update 
[94a508b...6e107ea](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-943) remove pid file only when it exists

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-943?focusedWorklogId=337622&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337622
 ]

ASF GitHub Bot logged work on GOBBLIN-943:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 23:53
Start Date: 01/Nov/19 23:53
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on pull request #2793: 
[GOBBLIN-943] improve on stop function
URL: https://github.com/apache/incubator-gobblin/pull/2793
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-943
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   improve on stop function
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason: N/A
   
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337622)
Remaining Estimate: 0h
Time Spent: 10m

> remove pid file only when it exists
> ---
>
> Key: GOBBLIN-943
> URL: https://issues.apache.org/jira/browse/GOBBLIN-943
> Project: Apache Gobblin
>  Issue Type: Sub-task
>Reporter: Jay Sen
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] jhsenjaliya opened a new pull request #2793: [GOBBLIN-943] improve on stop function

[GitBox]

jhsenjaliya opened a new pull request #2793: [GOBBLIN-943] improve on stop 
function
URL: https://github.com/apache/incubator-gobblin/pull/2793
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-943
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   improve on stop function
   
   ### Tests
   - [ ] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason: N/A
   
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (GOBBLIN-943) remove pid file only when it exists

[Jay Sen (Jira)]

Jay Sen created GOBBLIN-943:
---

 Summary: remove pid file only when it exists
 Key: GOBBLIN-943
 URL: https://issues.apache.org/jira/browse/GOBBLIN-943
 Project: Apache Gobblin
  Issue Type: Sub-task
Reporter: Jay Sen






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337605
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 23:07
Start Date: 01/Nov/19 23:07
Worklog Time Spent: 10m 
  Work Description: ahmedahamid commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341779189
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   Just following up. I tried adding a `null` value to this array but the 
`JsonRecordAvroSchemaToAvroConverterTest.testConverter` failed:
   ```java
   Could not convert field arrayFieldWithUnion
   org.apache.gobblin.converter.DataConversionException: Could not convert 
field arrayFieldWithUnion
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:127)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertRecord(JsonRecordAvroSchemaToAvroConverter.java:71)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverterTest.testConverter(JsonRecordAvroSchemaToAvroConverterTest.java:69)
   ...
   Caused by: java.lang.RuntimeException: Field: arrayFieldWithUnion is not 
nullable and contains a null value
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:278)
at 
org.apache.gobblin.converter.avro.JsonElementConversionWithAvroSchemaFactory$ArrayConverter.convertField(JsonElementConversionWithAvroSchemaFactory.java:98)
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:280)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:125)
... 51 more
   ```
   
   I think the key issue here is that `UnionConverter`'s does not determine its 
nullability correctly; it uses the nullability it receives from 
`JsonElementConversionWithAvroSchemaFactory.getConvertor()` to decide its own 
nullability. This is incorrect because `arrayFieldWithUnion` is a non-nullable 
array. However, the union itself (i.e. `UnionConverter`) would return `false` 
if you invoke `isNullable()` on it because it is construing 
`arrayFieldWithUnion`'s non-nullability as  its own (see the values 
`UnionConverter` provides to `super`).
   
   One possible fix could be to have `UnionConverter` override `isNullable()` 
so it can decide its own nullability based on its schemas, e.g.
   
   ```java
   @Override
   public boolean isNullable() {
 return this.firstSchema.getType() == Schema.Type.NULL ||
 this.secondSchema.getType() == Schema.Type.NULL;
   }
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337605)
Time Spent: 50m  (was: 40m)

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConver

[GitHub] [incubator-gobblin] ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341779189
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   Just following up. I tried adding a `null` value to this array but the 
`JsonRecordAvroSchemaToAvroConverterTest.testConverter` failed:
   ```java
   Could not convert field arrayFieldWithUnion
   org.apache.gobblin.converter.DataConversionException: Could not convert 
field arrayFieldWithUnion
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:127)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertRecord(JsonRecordAvroSchemaToAvroConverter.java:71)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverterTest.testConverter(JsonRecordAvroSchemaToAvroConverterTest.java:69)
   ...
   Caused by: java.lang.RuntimeException: Field: arrayFieldWithUnion is not 
nullable and contains a null value
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:278)
at 
org.apache.gobblin.converter.avro.JsonElementConversionWithAvroSchemaFactory$ArrayConverter.convertField(JsonElementConversionWithAvroSchemaFactory.java:98)
at 
org.apache.gobblin.converter.avro.JsonElementConversionFactory$JsonElementConverter.convert(JsonElementConversionFactory.java:280)
at 
org.apache.gobblin.converter.avro.JsonRecordAvroSchemaToAvroConverter.convertNestedRecord(JsonRecordAvroSchemaToAvroConverter.java:125)
... 51 more
   ```
   
   I think the key issue here is that `UnionConverter`'s does not determine its 
nullability correctly; it uses the nullability it receives from 
`JsonElementConversionWithAvroSchemaFactory.getConvertor()` to decide its own 
nullability. This is incorrect because `arrayFieldWithUnion` is a non-nullable 
array. However, the union itself (i.e. `UnionConverter`) would return `false` 
if you invoke `isNullable()` on it because it is construing 
`arrayFieldWithUnion`'s non-nullability as  its own (see the values 
`UnionConverter` provides to `super`).
   
   One possible fix could be to have `UnionConverter` override `isNullable()` 
so it can decide its own nullability based on its schemas, e.g.
   
   ```java
   @Override
   public boolean isNullable() {
 return this.firstSchema.getType() == Schema.Type.NULL ||
 this.secondSchema.getType() == Schema.Type.NULL;
   }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337601
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 23:00
Start Date: 01/Nov/19 23:00
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: 
https://github.com/apache/incubator-gobblin/pull/2792#issuecomment-548978202
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=h1)
 Report
   > Merging 
[#2792](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/94a508b38ec8bd879614f2d9bf0eeb96513ca7cf?src=pr&el=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2792  +/-   ##
   
   + Coverage 45.32%   45.32%   +<.01% 
   - Complexity 8862 8864   +2 
   
 Files  1894 1894  
 Lines 7091070925  +15 
 Branches   7799 7800   +1 
   
   + Hits  3214132150   +9 
   - Misses3580335811   +8 
   + Partials   2966 2964   -2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...ro/JsonElementConversionWithAvroSchemaFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9hdnJvL0pzb25FbGVtZW50Q29udmVyc2lvbldpdGhBdnJvU2NoZW1hRmFjdG9yeS5qYXZh)
 | `73.33% <66.66%> (-1.67%)` | `6 <0> (+1)` | |
   | 
[...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh)
 | `73.58% <0%> (-7.55%)` | `9% <0%> (ø)` | |
   | 
[...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh)
 | `76.08% <0%> (-4.35%)` | `5% <0%> (ø)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `32.71% <0%> (-2.81%)` | `11% <0%> (-1%)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | |
   | 
[.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=)
 | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `65.27% <0%> (+1.85%)` | `28% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   | 
[...lin/util/filesystem/FileSy

[GitHub] [incubator-gobblin] codecov-io commented on issue #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

codecov-io commented on issue #2792: [GOBBLIN-933] add support for array of 
unions in json schema
URL: 
https://github.com/apache/incubator-gobblin/pull/2792#issuecomment-548978202
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=h1)
 Report
   > Merging 
[#2792](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/94a508b38ec8bd879614f2d9bf0eeb96513ca7cf?src=pr&el=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `66.66%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2792  +/-   ##
   
   + Coverage 45.32%   45.32%   +<.01% 
   - Complexity 8862 8864   +2 
   
 Files  1894 1894  
 Lines 7091070925  +15 
 Branches   7799 7800   +1 
   
   + Hits  3214132150   +9 
   - Misses3580335811   +8 
   + Partials   2966 2964   -2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2792?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...ro/JsonElementConversionWithAvroSchemaFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9hdnJvL0pzb25FbGVtZW50Q29udmVyc2lvbldpdGhBdnJvU2NoZW1hRmFjdG9yeS5qYXZh)
 | `73.33% <66.66%> (-1.67%)` | `6 <0> (+1)` | |
   | 
[...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh)
 | `73.58% <0%> (-7.55%)` | `9% <0%> (ø)` | |
   | 
[...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh)
 | `76.08% <0%> (-4.35%)` | `5% <0%> (ø)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `32.71% <0%> (-2.81%)` | `11% <0%> (-1%)` | |
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | |
   | 
[.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=)
 | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `65.27% <0%> (+1.85%)` | `28% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `100% <0%> (+14.28%)` | `4% <0%> (+1%)` | :arrow_up: |
   | ... and [1 
more](https://codecov.io/gh/apache/incubator-gobblin/pull/2792/diff?src=pr&el=tree-more)
 | |
   
   --
   
   [Con

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337594
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 22:41
Start Date: 01/Nov/19 22:41
Worklog Time Spent: 10m 
  Work Description: ahmedahamid commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341774289
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   This would probably need a `null` value to make sure they're handled 
properly.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337594)
Time Spent: 20m  (was: 10m)

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:160)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337595&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337595
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 22:41
Start Date: 01/Nov/19 22:41
Worklog Time Spent: 10m 
  Work Description: ahmedahamid commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341774289
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   This would probably need a `null` value to make sure it would handle/allow 
nulls.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337595)
Time Spent: 0.5h  (was: 20m)

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:160)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341774289
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   This would probably need a `null` value to make sure it would handle/allow 
nulls.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

ahmedahamid commented on a change in pull request #2792: [GOBBLIN-933] add 
support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792#discussion_r341774289
 
 

 ##
 File path: gobblin-core/src/test/resources/converter/jsonToAvroRecord.json
 ##
 @@ -1,4 +1,6 @@
-{"nullableField": null,
+{
+  "arrayFieldWithUnion": ["arrU1", "arrU2", "arrU3"],
 
 Review comment:
   This would probably need a `null` value to make sure they're handled 
properly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

***UNCHECKED*** [jira] [Work logged] (GOBBLIN-913) Initialize MySQL on Gobblin K8s Cluster

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-913?focusedWorklogId=337589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337589
 ]

ASF GitHub Bot logged work on GOBBLIN-913:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 22:31
Start Date: 01/Nov/19 22:31
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2781: [GOBBLIN-913] Add 
MySQL and configurations to cluster
URL: 
https://github.com/apache/incubator-gobblin/pull/2781#issuecomment-545699470
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=h1)
 Report
   > Merging 
[#2781](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr&el=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2781  +/-   ##
   
   - Coverage 45.34%   45.33%   -0.02% 
   + Complexity 8864 8863   -1 
   
 Files  1894 1894  
 Lines 7087970910  +31 
 Branches   7795 7799   +4 
   
   + Hits  3214132145   +4 
   - Misses3577635800  +24 
   - Partials   2962 2965   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh)
 | `89.61% <0%> (-6.17%)` | `6% <0%> (ø)` | |
   | 
[...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh)
 | `51.59% <0%> (-4.66%)` | `23% <0%> (ø)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `35.51% <0%> (-3.74%)` | `12% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `63.42% <0%> (-1.86%)` | `27% <0%> (-1%)` | |
   | 
[...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=)
 | `67.13% <0%> (+0.12%)` | `84% <0%> (+1%)` | :arrow_up: |
   | 
[...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==)
 | `58.36% <0%> (+0.17%)` | `15% <0%> (ø)` | :arrow_down: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=footer).
 Last update 
[60e3ab5...91f74a7](https://codecov.io/gh/apach

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2781: [GOBBLIN-913] Add MySQL and configurations to cluster

[GitBox]

codecov-io edited a comment on issue #2781: [GOBBLIN-913] Add MySQL and 
configurations to cluster
URL: 
https://github.com/apache/incubator-gobblin/pull/2781#issuecomment-545699470
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=h1)
 Report
   > Merging 
[#2781](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr&el=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2781  +/-   ##
   
   - Coverage 45.34%   45.33%   -0.02% 
   + Complexity 8864 8863   -1 
   
 Files  1894 1894  
 Lines 7087970910  +31 
 Branches   7795 7799   +4 
   
   + Hits  3214132145   +4 
   - Misses3577635800  +24 
   - Partials   2962 2965   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh)
 | `89.61% <0%> (-6.17%)` | `6% <0%> (ø)` | |
   | 
[...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh)
 | `51.59% <0%> (-4.66%)` | `23% <0%> (ø)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `35.51% <0%> (-3.74%)` | `12% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `63.42% <0%> (-1.86%)` | `27% <0%> (-1%)` | |
   | 
[...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=)
 | `67.13% <0%> (+0.12%)` | `84% <0%> (+1%)` | :arrow_up: |
   | 
[...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==)
 | `58.36% <0%> (+0.17%)` | `15% <0%> (ø)` | :arrow_down: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=footer).
 Last update 
[60e3ab5...91f74a7](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructu

[jira] [Work logged] (GOBBLIN-933) JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-933?focusedWorklogId=337581&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337581
 ]

ASF GitHub Bot logged work on GOBBLIN-933:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 22:03
Start Date: 01/Nov/19 22:03
Worklog Time Spent: 10m 
  Work Description: arjun4084346 commented on pull request #2792: 
[GOBBLIN-933] add support for array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below! @zxcware  please review
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-933
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   It handles unions inside an array.
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   enhanced a test case
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337581)
Remaining Estimate: 0h
Time Spent: 10m

> JsonRecordAvroSchemaToAvroConverter does not handle arrays of unions
> 
>
> Key: GOBBLIN-933
> URL: https://issues.apache.org/jira/browse/GOBBLIN-933
> Project: Apache Gobblin
>  Issue Type: Bug
>Reporter: Ahmed Abdul Hamid
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Using {{JsonRecordAvroSchemaToAvroConverter}} to convert an array of a union 
> type fails. For instance, using it with the following Avro schema:
> {code:java}
> {
>   "name": "arrayField",
>   "type": {
> "type": "array",
> "items": ["string", "null"]
>   }
> } {code}
> yields the following error:
> {code:java}
> java.lang.StackOverflowError
>   at org.apache.gobblin.configuration.State.getProp(State.java)
>   at 
> org.apache.gobblin.configuration.WorkUnitState.getProp(WorkUnitState.java:333)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:106)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory.getConvertor(JsonElementConversionFactory.java:160)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.getConverter(JsonElementConversionFactory.java:737)
>   at 
> org.apache.gobblin.converter.avro.JsonElementConversionFactory$UnionConverter.(JsonElementConversionFactory.java:729)
>  {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] arjun4084346 opened a new pull request #2792: [GOBBLIN-933] add support for array of unions in json schema

[GitBox]

arjun4084346 opened a new pull request #2792: [GOBBLIN-933] add support for 
array of unions in json schema
URL: https://github.com/apache/incubator-gobblin/pull/2792
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below! @zxcware  please review
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-933
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   It handles unions inside an array.
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason:
   enhanced a test case
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337575&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337575
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 21:49
Start Date: 01/Nov/19 21:49
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr&el=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2788  +/-   ##
   
   - Coverage 45.34%   45.32%   -0.03% 
   + Complexity 8864 8859   -5 
   
 Files  1894 1894  
 Lines 7087970879  
 Branches   7795 7795  
   
   - Hits  3214132126  -15 
   - Misses3577635788  +12 
   - Partials   2962 2965   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/async/AsyncDataDispatcher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2FzeW5jL0FzeW5jRGF0YURpc3BhdGNoZXIuamF2YQ==)
 | `79.66% <0%> (-8.48%)` | `13% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `35.51% <0%> (-3.74%)` | `12% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `63.42% <0%> (-1.86%)` | `27% <0%> (-1%)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=footer).
 Last update 
[60e3ab5...8c7f1be](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337575)
Time Spent: 1h 50m  (was: 1h 40m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and standardize configs

[GitBox]

codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of 
env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr&el=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2788  +/-   ##
   
   - Coverage 45.34%   45.32%   -0.03% 
   + Complexity 8864 8859   -5 
   
 Files  1894 1894  
 Lines 7087970879  
 Branches   7795 7795  
   
   - Hits  3214132126  -15 
   - Misses3577635788  +12 
   - Partials   2962 2965   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/async/AsyncDataDispatcher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2FzeW5jL0FzeW5jRGF0YURpc3BhdGNoZXIuamF2YQ==)
 | `79.66% <0%> (-8.48%)` | `13% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `35.51% <0%> (-3.74%)` | `12% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `63.42% <0%> (-1.86%)` | `27% <0%> (-1%)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=footer).
 Last update 
[60e3ab5...8c7f1be](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

***UNCHECKED*** [jira] [Work logged] (GOBBLIN-913) Initialize MySQL on Gobblin K8s Cluster

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-913?focusedWorklogId=337565&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337565
 ]

ASF GitHub Bot logged work on GOBBLIN-913:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 21:30
Start Date: 01/Nov/19 21:30
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on pull request #2781: [GOBBLIN-913] 
Add MySQL and configurations to cluster
URL: https://github.com/apache/incubator-gobblin/pull/2781#discussion_r341758582
 
 

 ##
 File path: gobblin-kubernetes/gobblin-service/dev-cluster/gaas-application.conf
 ##
 @@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Sample configuration properties for the Gobblin Service
+
+# Topology Catalog and Store
+gobblin.service.work.dir=/tmp/gobblin-as-service
+
+# TopologySpec Factory
+topologySpec.store.dir=${gobblin.service.work.dir}/topologySpecStore
+topologySpecFactory.topologyNames=localGobblinCluster
+topologySpecFactory.localGobblinCluster.description="StandaloneClusterTopology"
+topologySpecFactory.localGobblinCluster.version="1"
+topologySpecFactory.localGobblinCluster.uri="gobblinCluster"
+topologySpecFactory.localGobblinCluster.specExecutorInstance.class="org.apache.gobblin.runtime.spec_executorInstance.LocalFsSpecExecutor"
+topologySpecFactory.localGobblinCluster.specExecInstance.capabilities="source:dest"
+topologySpecFactory.localGobblinCluster.gobblin.cluster.localSpecProducer.dir=${gobblin.service.work.dir}/jobs
+
+# Flow Catalog and Store
+flowSpec.store.dir=${gobblin.service.work.dir}/flowSpecStore
+
+# Template Catalog
+gobblin.service.templateCatalogs.fullyQualifiedPath="file://"
+
+# JobStatusMonitor
+gobblin.service.jobStatusMonitor.enabled=false
+
+# FsJobStatusRetriever
+fsJobStatusRetriever.state.store.dir=${gobblin.service.work.dir}/state-store
+
+# DagManager
+gobblin.service.dagManager.enabled=true
+gobblin.service.dagManager.jobStatusRetriever.class="org.apache.gobblin.service.monitoring.FsJobStatusRetriever"
+gobblin.service.dagManager.dagStateStoreClass="org.apache.gobblin.service.modules.orchestration.FSDagStateStore"
+gobblin.service.dagManager.dagStateStoreDir=${gobblin.service.work.dir}/dagStateStoreDir
+
+# RestLI
+gobblin.service.port=6956
+
+# MySQL State Store
 
 Review comment:
   Ah I should rename it to be better then, `dev` was originally intended for 
`mysql` plus whatever job communication mechanism we are relying on, which is 
`fs`. The only difference between `dev` and `base` would be the mysql store, so 
I guess it would be better to rename it to mysql-cluster
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337565)
Time Spent: 4h 40m  (was: 4.5h)

> Initialize MySQL on Gobblin K8s Cluster
> ---
>
> Key: GOBBLIN-913
> URL: https://issues.apache.org/jira/browse/GOBBLIN-913
> Project: Apache Gobblin
>  Issue Type: New Feature
>Affects Versions: 0.15.0
>Reporter: William Lo
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] Will-Lo commented on a change in pull request #2781: [GOBBLIN-913] Add MySQL and configurations to cluster

[GitBox]

Will-Lo commented on a change in pull request #2781: [GOBBLIN-913] Add MySQL 
and configurations to cluster
URL: https://github.com/apache/incubator-gobblin/pull/2781#discussion_r341758582
 
 

 ##
 File path: gobblin-kubernetes/gobblin-service/dev-cluster/gaas-application.conf
 ##
 @@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Sample configuration properties for the Gobblin Service
+
+# Topology Catalog and Store
+gobblin.service.work.dir=/tmp/gobblin-as-service
+
+# TopologySpec Factory
+topologySpec.store.dir=${gobblin.service.work.dir}/topologySpecStore
+topologySpecFactory.topologyNames=localGobblinCluster
+topologySpecFactory.localGobblinCluster.description="StandaloneClusterTopology"
+topologySpecFactory.localGobblinCluster.version="1"
+topologySpecFactory.localGobblinCluster.uri="gobblinCluster"
+topologySpecFactory.localGobblinCluster.specExecutorInstance.class="org.apache.gobblin.runtime.spec_executorInstance.LocalFsSpecExecutor"
+topologySpecFactory.localGobblinCluster.specExecInstance.capabilities="source:dest"
+topologySpecFactory.localGobblinCluster.gobblin.cluster.localSpecProducer.dir=${gobblin.service.work.dir}/jobs
+
+# Flow Catalog and Store
+flowSpec.store.dir=${gobblin.service.work.dir}/flowSpecStore
+
+# Template Catalog
+gobblin.service.templateCatalogs.fullyQualifiedPath="file://"
+
+# JobStatusMonitor
+gobblin.service.jobStatusMonitor.enabled=false
+
+# FsJobStatusRetriever
+fsJobStatusRetriever.state.store.dir=${gobblin.service.work.dir}/state-store
+
+# DagManager
+gobblin.service.dagManager.enabled=true
+gobblin.service.dagManager.jobStatusRetriever.class="org.apache.gobblin.service.monitoring.FsJobStatusRetriever"
+gobblin.service.dagManager.dagStateStoreClass="org.apache.gobblin.service.modules.orchestration.FSDagStateStore"
+gobblin.service.dagManager.dagStateStoreDir=${gobblin.service.work.dir}/dagStateStoreDir
+
+# RestLI
+gobblin.service.port=6956
+
+# MySQL State Store
 
 Review comment:
   Ah I should rename it to be better then, `dev` was originally intended for 
`mysql` plus whatever job communication mechanism we are relying on, which is 
`fs`. The only difference between `dev` and `base` would be the mysql store, so 
I guess it would be better to rename it to mysql-cluster


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

***UNCHECKED*** [jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337563&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337563
 ]

ASF GitHub Bot logged work on GOBBLIN-941:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 21:26
Start Date: 01/Nov/19 21:26
Worklog Time Spent: 10m 
  Work Description: asfgit commented on pull request #2791: [GOBBLIN-941] 
Enhance DDL to add column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2791
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337563)
Time Spent: 3h 20m  (was: 3h 10m)

> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
>  Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which 
> would enforce avro2orc output preserving correct casing



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] asfgit closed pull request #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema

[GitBox]

asfgit closed pull request #2791: [GOBBLIN-941] Enhance DDL to add column and 
column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2791
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337490&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337490
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 19:08
Start Date: 01/Nov/19 19:08
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr&el=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2788  +/-   ##
   
   - Coverage 45.34%   45.32%   -0.02% 
   + Complexity 8864 8861   -3 
   
 Files  1894 1894  
 Lines 7087970879  
 Branches   7795 7795  
   
   - Hits  3214132129  -12 
   - Misses3577635787  +11 
   - Partials   2962 2963   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `85.71% <0%> (-14.29%)` | `3% <0%> (-1%)` | |
   | 
[...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh)
 | `73.58% <0%> (-7.55%)` | `9% <0%> (ø)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `32.71% <0%> (-6.55%)` | `11% <0%> (-2%)` | |
   | 
[...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh)
 | `76.08% <0%> (-4.35%)` | `5% <0%> (ø)` | |
   | 
[.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=)
 | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | |
   | 
[...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==)
 | `30.87% <0%> (+0.67%)` | `25% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Cod

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and standardize configs

[GitBox]

codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of 
env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr&el=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2788  +/-   ##
   
   - Coverage 45.34%   45.32%   -0.02% 
   + Complexity 8864 8861   -3 
   
 Files  1894 1894  
 Lines 7087970879  
 Branches   7795 7795  
   
   - Hits  3214132129  -12 
   - Misses3577635787  +11 
   - Partials   2962 2963   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `85.71% <0%> (-14.29%)` | `3% <0%> (-1%)` | |
   | 
[...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh)
 | `73.58% <0%> (-7.55%)` | `9% <0%> (ø)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `32.71% <0%> (-6.55%)` | `11% <0%> (-2%)` | |
   | 
[...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh)
 | `76.08% <0%> (-4.35%)` | `5% <0%> (ø)` | |
   | 
[.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=)
 | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | |
   | 
[...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==)
 | `30.87% <0%> (+0.67%)` | `25% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr&el=footer).
 Last update 
[60e3ab5...a7dcea7](https://codecov.io/gh/apache/incubator-gobbl

[jira] [Work logged] (GOBBLIN-913) Initialize MySQL on Gobblin K8s Cluster

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-913?focusedWorklogId=337461&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337461
 ]

ASF GitHub Bot logged work on GOBBLIN-913:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 18:25
Start Date: 01/Nov/19 18:25
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2781: [GOBBLIN-913] Add 
MySQL and configurations to cluster
URL: 
https://github.com/apache/incubator-gobblin/pull/2781#issuecomment-545699470
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=h1)
 Report
   > Merging 
[#2781](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr&el=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2781  +/-   ##
   
   + Coverage 45.34%   45.34%   +<.01% 
   + Complexity 8864 8861   -3 
   
 Files  1894 1894  
 Lines 7087970879  
 Branches   7795 7795  
   
   + Hits  3214132143   +2 
   + Misses3577635770   -6 
   - Partials   2962 2966   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh)
 | `78.12% <0%> (-1.57%)` | `15% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `64.81% <0%> (-0.47%)` | `28% <0%> (ø)` | |
   | 
[...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==)
 | `15.68% <0%> (+0.84%)` | `4% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=footer).
 Last update 
[60e3ab5...348aedb](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   
 

[jira] [Work logged] (GOBBLIN-940) Synchronization between workunit persistency and Helix job launching

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-940?focusedWorklogId=337462&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337462
 ]

ASF GitHub Bot logged work on GOBBLIN-940:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 18:32
Start Date: 01/Nov/19 18:32
Worklog Time Spent: 10m 
  Work Description: htran1 commented on pull request #2789: 
[GOBBLIN-940]Add synchronization on workunit persistency before Helix job 
launching
URL: https://github.com/apache/incubator-gobblin/pull/2789#discussion_r341686549
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
 ##
 @@ -295,23 +295,34 @@ protected void executeCancellation() {
 SerializationUtils.serializeState(this.fs, jobStateFilePath, 
this.jobContext.getJobState());
   }
 
-  LOGGER.debug("GobblinHelixJobLauncher.createJob: jobStateFilePath {}, 
jobState {} jobProperties {}",
+  // Block on all persistency of workunits finished
+  // It is necessary when underlying storage being and Helix activate 
task-execution before the workunit being persisted.
 
 Review comment:
   `storage being and`? Seems like something is missing here in this comment.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337462)
Time Spent: 40m  (was: 0.5h)

> Synchronization between workunit persistency and Helix job launching
> 
>
> Key: GOBBLIN-940
> URL: https://issues.apache.org/jira/browse/GOBBLIN-940
> Project: Apache Gobblin
>  Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] htran1 commented on a change in pull request #2789: [GOBBLIN-940]Add synchronization on workunit persistency before Helix job launching

[GitBox]

htran1 commented on a change in pull request #2789: [GOBBLIN-940]Add 
synchronization on workunit persistency before Helix job launching
URL: https://github.com/apache/incubator-gobblin/pull/2789#discussion_r341686549
 
 

 ##
 File path: 
gobblin-cluster/src/main/java/org/apache/gobblin/cluster/GobblinHelixJobLauncher.java
 ##
 @@ -295,23 +295,34 @@ protected void executeCancellation() {
 SerializationUtils.serializeState(this.fs, jobStateFilePath, 
this.jobContext.getJobState());
   }
 
-  LOGGER.debug("GobblinHelixJobLauncher.createJob: jobStateFilePath {}, 
jobState {} jobProperties {}",
+  // Block on all persistency of workunits finished
+  // It is necessary when underlying storage being and Helix activate 
task-execution before the workunit being persisted.
 
 Review comment:
   `storage being and`? Seems like something is missing here in this comment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2781: [GOBBLIN-913] Add MySQL and configurations to cluster

[GitBox]

codecov-io edited a comment on issue #2781: [GOBBLIN-913] Add MySQL and 
configurations to cluster
URL: 
https://github.com/apache/incubator-gobblin/pull/2781#issuecomment-545699470
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=h1)
 Report
   > Merging 
[#2781](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr&el=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/graphs/tree.svg?width=650&token=4MgURJ0bGc&height=150&src=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2781  +/-   ##
   
   + Coverage 45.34%   45.34%   +<.01% 
   + Complexity 8864 8861   -3 
   
 Files  1894 1894  
 Lines 7087970879  
 Branches   7795 7795  
   
   + Hits  3214132143   +2 
   + Misses3577635770   -6 
   - Partials   2962 2966   +4
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh)
 | `78.12% <0%> (-1.57%)` | `15% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `64.81% <0%> (-0.47%)` | `28% <0%> (ø)` | |
   | 
[...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==)
 | `15.68% <0%> (+0.84%)` | `4% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2781/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=footer).
 Last update 
[60e3ab5...348aedb](https://codecov.io/gh/apache/incubator-gobblin/pull/2781?src=pr&el=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

regarding GOBBLIN-926

[Jay Sen]

Hi Dev Team,

I face many issues because of inconsistencies between how Gobblin treats
the configurations.

so I have created this issue to use HOCON consistently, and I want to know
if this make sense before i go ahead and make all changes.

please take a look. (GOBBLIN-926
) and let me know your
thoughts on this.

Thanks
Jay

[jira] [Work logged] (GOBBLIN-913) Initialize MySQL on Gobblin K8s Cluster

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-913?focusedWorklogId=337444&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337444
 ]

ASF GitHub Bot logged work on GOBBLIN-913:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 17:46
Start Date: 01/Nov/19 17:46
Worklog Time Spent: 10m 
  Work Description: zxcware commented on pull request #2781: [GOBBLIN-913] 
Add MySQL and configurations to cluster
URL: https://github.com/apache/incubator-gobblin/pull/2781#discussion_r341678716
 
 

 ##
 File path: gobblin-kubernetes/gobblin-service/dev-cluster/gaas-application.conf
 ##
 @@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Sample configuration properties for the Gobblin Service
+
+# Topology Catalog and Store
+gobblin.service.work.dir=/tmp/gobblin-as-service
+
+# TopologySpec Factory
+topologySpec.store.dir=${gobblin.service.work.dir}/topologySpecStore
+topologySpecFactory.topologyNames=localGobblinCluster
+topologySpecFactory.localGobblinCluster.description="StandaloneClusterTopology"
+topologySpecFactory.localGobblinCluster.version="1"
+topologySpecFactory.localGobblinCluster.uri="gobblinCluster"
+topologySpecFactory.localGobblinCluster.specExecutorInstance.class="org.apache.gobblin.runtime.spec_executorInstance.LocalFsSpecExecutor"
+topologySpecFactory.localGobblinCluster.specExecInstance.capabilities="source:dest"
+topologySpecFactory.localGobblinCluster.gobblin.cluster.localSpecProducer.dir=${gobblin.service.work.dir}/jobs
+
+# Flow Catalog and Store
+flowSpec.store.dir=${gobblin.service.work.dir}/flowSpecStore
+
+# Template Catalog
+gobblin.service.templateCatalogs.fullyQualifiedPath="file://"
+
+# JobStatusMonitor
+gobblin.service.jobStatusMonitor.enabled=false
+
+# FsJobStatusRetriever
+fsJobStatusRetriever.state.store.dir=${gobblin.service.work.dir}/state-store
+
+# DagManager
+gobblin.service.dagManager.enabled=true
+gobblin.service.dagManager.jobStatusRetriever.class="org.apache.gobblin.service.monitoring.FsJobStatusRetriever"
+gobblin.service.dagManager.dagStateStoreClass="org.apache.gobblin.service.modules.orchestration.FSDagStateStore"
+gobblin.service.dagManager.dagStateStoreDir=${gobblin.service.work.dir}/dagStateStoreDir
+
+# RestLI
+gobblin.service.port=6956
+
+# MySQL State Store
 
 Review comment:
   if `dev` is built around file systems, do we need `mysql` environment here? 
A user can pick either `dev` suite or `mysql` suite.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337444)
Time Spent: 4h 20m  (was: 4h 10m)

> Initialize MySQL on Gobblin K8s Cluster
> ---
>
> Key: GOBBLIN-913
> URL: https://issues.apache.org/jira/browse/GOBBLIN-913
> Project: Apache Gobblin
>  Issue Type: New Feature
>Affects Versions: 0.15.0
>Reporter: William Lo
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2781: [GOBBLIN-913] Add MySQL and configurations to cluster

[GitBox]

zxcware commented on a change in pull request #2781: [GOBBLIN-913] Add MySQL 
and configurations to cluster
URL: https://github.com/apache/incubator-gobblin/pull/2781#discussion_r341681213
 
 

 ##
 File path: gobblin-kubernetes/gobblin-service/dev-cluster/mysql-deployment.yaml
 ##
 @@ -0,0 +1,55 @@
+apiVersion: v1
 
 Review comment:
   How do we organize the content between `fs` based and `mysql` based? Using 
the same parent folder `dev-cluster` is confusing. 
   
   Currently, we're using `kustomization` file to share and organize k8s 
application configs. I guess we'll have a mysql folder with its own 
`kustomization` file?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Work logged] (GOBBLIN-913) Initialize MySQL on Gobblin K8s Cluster

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/GOBBLIN-913?focusedWorklogId=337445&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337445
 ]

ASF GitHub Bot logged work on GOBBLIN-913:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 17:46
Start Date: 01/Nov/19 17:46
Worklog Time Spent: 10m 
  Work Description: zxcware commented on pull request #2781: [GOBBLIN-913] 
Add MySQL and configurations to cluster
URL: https://github.com/apache/incubator-gobblin/pull/2781#discussion_r341681213
 
 

 ##
 File path: gobblin-kubernetes/gobblin-service/dev-cluster/mysql-deployment.yaml
 ##
 @@ -0,0 +1,55 @@
+apiVersion: v1
 
 Review comment:
   How do we organize the content between `fs` based and `mysql` based? Using 
the same parent folder `dev-cluster` is confusing. 
   
   Currently, we're using `kustomization` file to share and organize k8s 
application configs. I guess we'll have a mysql folder with its own 
`kustomization` file?
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337445)
Time Spent: 4h 20m  (was: 4h 10m)

> Initialize MySQL on Gobblin K8s Cluster
> ---
>
> Key: GOBBLIN-913
> URL: https://issues.apache.org/jira/browse/GOBBLIN-913
> Project: Apache Gobblin
>  Issue Type: New Feature
>Affects Versions: 0.15.0
>Reporter: William Lo
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [incubator-gobblin] zxcware commented on a change in pull request #2781: [GOBBLIN-913] Add MySQL and configurations to cluster

[GitBox]

zxcware commented on a change in pull request #2781: [GOBBLIN-913] Add MySQL 
and configurations to cluster
URL: https://github.com/apache/incubator-gobblin/pull/2781#discussion_r341678716
 
 

 ##
 File path: gobblin-kubernetes/gobblin-service/dev-cluster/gaas-application.conf
 ##
 @@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Sample configuration properties for the Gobblin Service
+
+# Topology Catalog and Store
+gobblin.service.work.dir=/tmp/gobblin-as-service
+
+# TopologySpec Factory
+topologySpec.store.dir=${gobblin.service.work.dir}/topologySpecStore
+topologySpecFactory.topologyNames=localGobblinCluster
+topologySpecFactory.localGobblinCluster.description="StandaloneClusterTopology"
+topologySpecFactory.localGobblinCluster.version="1"
+topologySpecFactory.localGobblinCluster.uri="gobblinCluster"
+topologySpecFactory.localGobblinCluster.specExecutorInstance.class="org.apache.gobblin.runtime.spec_executorInstance.LocalFsSpecExecutor"
+topologySpecFactory.localGobblinCluster.specExecInstance.capabilities="source:dest"
+topologySpecFactory.localGobblinCluster.gobblin.cluster.localSpecProducer.dir=${gobblin.service.work.dir}/jobs
+
+# Flow Catalog and Store
+flowSpec.store.dir=${gobblin.service.work.dir}/flowSpecStore
+
+# Template Catalog
+gobblin.service.templateCatalogs.fullyQualifiedPath="file://"
+
+# JobStatusMonitor
+gobblin.service.jobStatusMonitor.enabled=false
+
+# FsJobStatusRetriever
+fsJobStatusRetriever.state.store.dir=${gobblin.service.work.dir}/state-store
+
+# DagManager
+gobblin.service.dagManager.enabled=true
+gobblin.service.dagManager.jobStatusRetriever.class="org.apache.gobblin.service.monitoring.FsJobStatusRetriever"
+gobblin.service.dagManager.dagStateStoreClass="org.apache.gobblin.service.modules.orchestration.FSDagStateStore"
+gobblin.service.dagManager.dagStateStoreDir=${gobblin.service.work.dir}/dagStateStoreDir
+
+# RestLI
+gobblin.service.port=6956
+
+# MySQL State Store
 
 Review comment:
   if `dev` is built around file systems, do we need `mysql` environment here? 
A user can pick either `dev` suite or `mysql` suite.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services