[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337123
]
ASF GitHub Bot logged work on GOBBLIN-941:
--
Author: ASF GitHub Bot
Created on: 01/Nov/19 00:56
Start Date: 01/Nov/19 00:56
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on pull request #2790:
[GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving
schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341419272
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
ddl.append(generateAvroToHiveColumnMapping(schema,
Optional.of(hiveColumns), true, dbName + "." + tblName));
+ try {
Review comment:
Yes, at least it's enabled in scoreevent. BTW, can you look at the new pr
for this commit? I have sent you the email. Thank you!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337123)
Time Spent: 3h 10m (was: 3h)
> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which
> would enforce avro2orc output preserving correct casing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance
DDL to add column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341419272
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
ddl.append(generateAvroToHiveColumnMapping(schema,
Optional.of(hiveColumns), true, dbName + "." + tblName));
+ try {
Review comment:
Yes, at least it's enabled in scoreevent. BTW, can you look at the new pr
for this commit? I have sent you the email. Thank you!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337115
]
ASF GitHub Bot logged work on GOBBLIN-941:
--
Author: ASF GitHub Bot
Created on: 01/Nov/19 00:26
Start Date: 01/Nov/19 00:26
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2790:
[GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving
schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341415129
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
ddl.append(generateAvroToHiveColumnMapping(schema,
Optional.of(hiveColumns), true, dbName + "." + tblName));
+ try {
Review comment:
Fair enough. Worth to check if the production have schema evolution enabled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337115)
Time Spent: 2h 50m (was: 2h 40m)
> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which
> would enforce avro2orc output preserving correct casing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337119 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 01/Nov/19 00:30 Start Date: 01/Nov/19 00:30 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341415701 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/task/HiveConverterUtils.java ## @@ -151,6 +154,30 @@ public static String generateCreateDuplicateTableDDL( dbName, tblName, inputDbName, inputTblName, tblLocation); } + public static String generateAlterSchemaDML( + String tableName, + Optional optionalDbName, Review comment: +1 for consistency. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337119) Time Spent: 3h (was: 2h 50m) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341415701 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/task/HiveConverterUtils.java ## @@ -151,6 +154,30 @@ public static String generateCreateDuplicateTableDDL( dbName, tblName, inputDbName, inputTblName, tblLocation); } + public static String generateAlterSchemaDML( + String tableName, + Optional optionalDbName, Review comment: +1 for consistency. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance
DDL to add column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341415129
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
ddl.append(generateAvroToHiveColumnMapping(schema,
Optional.of(hiveColumns), true, dbName + "." + tblName));
+ try {
Review comment:
Fair enough. Worth to check if the production have schema evolution enabled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337112=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337112 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 01/Nov/19 00:23 Start Date: 01/Nov/19 00:23 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341414676 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/materializer/HiveMaterializerFromEntityQueryGenerator.java ## @@ -118,11 +119,19 @@ public QueryBasedHivePublishEntity generatePublishQueries() throws DataConversio Map publishDirectories = publishEntity.getPublishDirectories(); List cleanupQueries = publishEntity.getCleanupQueries(); List cleanupDirectories = publishEntity.getCleanupDirectories(); +Optional avroSchema = Optional.absent(); Review comment: Your call. I am OK with both. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337112) Time Spent: 2h 40m (was: 2.5h) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341414676 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/materializer/HiveMaterializerFromEntityQueryGenerator.java ## @@ -118,11 +119,19 @@ public QueryBasedHivePublishEntity generatePublishQueries() throws DataConversio Map publishDirectories = publishEntity.getPublishDirectories(); List cleanupQueries = publishEntity.getCleanupQueries(); List cleanupDirectories = publishEntity.getCleanupDirectories(); +Optional avroSchema = Optional.absent(); Review comment: Your call. I am OK with both. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337111 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 01/Nov/19 00:23 Start Date: 01/Nov/19 00:23 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341414676 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/materializer/HiveMaterializerFromEntityQueryGenerator.java ## @@ -118,11 +119,19 @@ public QueryBasedHivePublishEntity generatePublishQueries() throws DataConversio Map publishDirectories = publishEntity.getPublishDirectories(); List cleanupQueries = publishEntity.getCleanupQueries(); List cleanupDirectories = publishEntity.getCleanupDirectories(); +Optional avroSchema = Optional.absent(); Review comment: You call. I am OK with both. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337111) Time Spent: 2.5h (was: 2h 20m) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341414676 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/materializer/HiveMaterializerFromEntityQueryGenerator.java ## @@ -118,11 +119,19 @@ public QueryBasedHivePublishEntity generatePublishQueries() throws DataConversio Map publishDirectories = publishEntity.getPublishDirectories(); List cleanupQueries = publishEntity.getCleanupQueries(); List cleanupDirectories = publishEntity.getCleanupDirectories(); +Optional avroSchema = Optional.absent(); Review comment: You call. I am OK with both. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (GOBBLIN-942) use HOCON for job configurations instead of Java Properties
[
https://issues.apache.org/jira/browse/GOBBLIN-942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jay Sen updated GOBBLIN-942:
Summary: use HOCON for job configurations instead of Java Properties (was:
HOCON vs Java Prop for job configurations)
> use HOCON for job configurations instead of Java Properties
> ---
>
> Key: GOBBLIN-942
> URL: https://issues.apache.org/jira/browse/GOBBLIN-942
> Project: Apache Gobblin
> Issue Type: Sub-task
>Reporter: Jay Sen
>Priority: Major
>
> currently {{PullFileLoader}} uses java property loader for *.pull & *.job
> files and uses hocon loader for *.json & *.conf files.
> This introduces lot of inconsistencies among how platform config gets treated
> vs job config. I guess previously we used the \{env:***} format for using the
> env variables but with GOBBLIN-939 it makes it HOCON compatible.
> the current issue with treating job config file as java property file is that
> it addes quote around the already quoted string which HOCON handles well.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (GOBBLIN-942) HOCON vs Java Prop for job configurations
Jay Sen created GOBBLIN-942:
---
Summary: HOCON vs Java Prop for job configurations
Key: GOBBLIN-942
URL: https://issues.apache.org/jira/browse/GOBBLIN-942
Project: Apache Gobblin
Issue Type: Sub-task
Reporter: Jay Sen
currently {{PullFileLoader}} uses java property loader for *.pull & *.job files
and uses hocon loader for *.json & *.conf files.
This introduces lot of inconsistencies among how platform config gets treated
vs job config. I guess previously we used the \{env:***} format for using the
env variables but with GOBBLIN-939 it makes it HOCON compatible.
the current issue with treating job config file as java property file is that
it addes quote around the already quoted string which HOCON handles well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-937) fix help text and align it with variable names
[ https://issues.apache.org/jira/browse/GOBBLIN-937?focusedWorklogId=337098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337098 ] ASF GitHub Bot logged work on GOBBLIN-937: -- Author: ASF GitHub Bot Created on: 31/Oct/19 23:33 Start Date: 31/Oct/19 23:33 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2786: [GOBBLIN-937] fix help text and align it with variable names URL: https://github.com/apache/incubator-gobblin/pull/2786 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337098) Time Spent: 0.5h (was: 20m) > fix help text and align it with variable names > -- > > Key: GOBBLIN-937 > URL: https://issues.apache.org/jira/browse/GOBBLIN-937 > Project: Apache Gobblin > Issue Type: Sub-task >Affects Versions: 0.14.0 >Reporter: Jay Sen >Priority: Trivial > Fix For: 0.15.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2786: [GOBBLIN-937] fix help text and align it with variable names
asfgit closed pull request #2786: [GOBBLIN-937] fix help text and align it with variable names URL: https://github.com/apache/incubator-gobblin/pull/2786 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] ZihanLi58 commented on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 commented on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2791#issuecomment-548599519 @autumnust Here is the new PR, Thx. ^^ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337086 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 22:49 Start Date: 31/Oct/19 22:49 Worklog Time Spent: 10m Work Description: ZihanLi58 commented on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2791#issuecomment-548599519 @autumnust Here is the new PR, Thx. ^^ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337086) Time Spent: 2h 20m (was: 2h 10m) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337081 ] ASF GitHub Bot logged work on GOBBLIN-939: -- Author: ASF GitHub Bot Created on: 31/Oct/19 22:37 Start Date: 31/Oct/19 22:37 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1) Report > Merging [#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.18%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2788 +/- ## = + Coverage 4.15% 45.34% +41.18% - Complexity 746 8863 +8117 = Files 1894 1894 Lines 7087770879+2 Branches 7793 7795+2 = + Hits 294632139+29193 + Misses6761735778-31839 - Partials314 2962 +2648 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | | [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: | |
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1) Report > Merging [#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.18%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2788 +/- ## = + Coverage 4.15% 45.34% +41.18% - Complexity 746 8863 +8117 = Files 1894 1894 Lines 7087770879+2 Branches 7793 7795+2 = + Hits 294632139+29193 + Misses6761735778-31839 - Partials314 2962 +2648 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | | [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: | | [...apache/gobblin/source/jdbc/SqlServerExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9qZGJjL1NxbFNlcnZlckV4dHJhY3Rvci5qYXZh) | `4.41% <0%> (+4.41%)` | `3% <0%> (+3%)` | :arrow_up: | | ... and [1088 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree-more) | | -- [Continue to review full report at
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337040 ] ASF GitHub Bot logged work on GOBBLIN-939: -- Author: ASF GitHub Bot Created on: 31/Oct/19 20:59 Start Date: 31/Oct/19 20:59 Worklog Time Spent: 10m Work Description: jhsenjaliya commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548566219 > I was wondering, would it be better if we were to utilise javaopts, so `-D` flags for these environment variables, rather than continuously modifying the scripts/configs? u mean like javaopts in bash ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337040) Time Spent: 1h 10m (was: 1h) > Integrate usage of env variables in gobblin scripts and configs > --- > > Key: GOBBLIN-939 > URL: https://issues.apache.org/jira/browse/GOBBLIN-939 > Project: Apache Gobblin > Issue Type: Improvement >Affects Versions: 0.14.0 >Reporter: Jay Sen >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > 1. standardize config with ENV variables > 2. define default env variables in gobblin-env.sh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337073 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 22:02 Start Date: 31/Oct/19 22:02 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2791#issuecomment-548577059 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=h1) Report > Merging [#2791](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/f60409ef0b6768bf46ddd137333d8d56981798fc?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `34.37%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2791 +/- ## - Coverage 45.34% 45.33% -0.01% + Complexity 8863 8862 -1 Files 1894 1894 Lines 7087970910 +31 Branches 7795 7799 +4 + Hits 3213732146 +9 - Misses3577835798 +20 - Partials 2964 2966 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh) | `51.59% <0%> (-4.66%)` | `23 <0> (ø)` | | | [...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==) | `58.36% <100%> (+0.17%)` | `15 <0> (ø)` | :arrow_down: | | [...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh) | `89.61% <16.66%> (-6.17%)` | `6 <0> (ø)` | | | [...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=) | `67.13% <75%> (+0.12%)` | `84 <1> (+1)` | :arrow_up: | | [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40% <0%> (-20%)` | `2% <0%> (-1%)` | | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0%> (+0.93%)` | `24% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=footer). Last update [f60409e...42a3dbe](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=lastupdated). Read the [comment
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
codecov-io edited a comment on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2791#issuecomment-548577059 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=h1) Report > Merging [#2791](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/f60409ef0b6768bf46ddd137333d8d56981798fc?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `34.37%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2791 +/- ## - Coverage 45.34% 45.33% -0.01% + Complexity 8863 8862 -1 Files 1894 1894 Lines 7087970910 +31 Branches 7795 7799 +4 + Hits 3213732146 +9 - Misses3577835798 +20 - Partials 2964 2966 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh) | `51.59% <0%> (-4.66%)` | `23 <0> (ø)` | | | [...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==) | `58.36% <100%> (+0.17%)` | `15 <0> (ø)` | :arrow_down: | | [...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh) | `89.61% <16.66%> (-6.17%)` | `6 <0> (ø)` | | | [...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=) | `67.13% <75%> (+0.12%)` | `84 <1> (+1)` | :arrow_up: | | [...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh) | `40% <0%> (-20%)` | `2% <0%> (-1%)` | | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0%> (+0.93%)` | `24% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=footer). Last update [f60409e...42a3dbe](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337068 ] ASF GitHub Bot logged work on GOBBLIN-939: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:42 Start Date: 31/Oct/19 21:42 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1) Report > Merging [#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.18%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2788 +/- ## = + Coverage 4.15% 45.33% +41.18% - Complexity 746 8863 +8117 = Files 1894 1894 Lines 7087770879+2 Branches 7793 7795+2 = + Hits 294632136+29190 + Misses6761735781-31836 - Partials314 2962 +2648 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | | [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: | | [...apache/gobblin/source/jdbc/SqlServerExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9qZGJjL1NxbFNlcnZlckV4dHJhY3Rvci5qYXZh) | `4.41% <0%> (+4.41%)` | `3% <0%> (+3%)` | :arrow_up: | |
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1) Report > Merging [#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.18%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2788 +/- ## = + Coverage 4.15% 45.33% +41.18% - Complexity 746 8863 +8117 = Files 1894 1894 Lines 7087770879+2 Branches 7793 7795+2 = + Hits 294632136+29190 + Misses6761735781-31836 - Partials314 2962 +2648 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | | [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: | | [...apache/gobblin/source/jdbc/SqlServerExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9qZGJjL1NxbFNlcnZlckV4dHJhY3Rvci5qYXZh) | `4.41% <0%> (+4.41%)` | `3% <0%> (+3%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/AbstractJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvQWJzdHJhY3RKb2JMYXVuY2hlci5qYXZh) | `59.13% <0%> (+5.1%)` | `33% <0%> (+6%)` | :arrow_up: | | ... and [1087 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree-more) | | -- [Continue to review full report at
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337064=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337064 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:30 Start Date: 31/Oct/19 21:30 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2791#issuecomment-548577059 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=h1) Report > Merging [#2791](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/f60409ef0b6768bf46ddd137333d8d56981798fc?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `36.36%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2791 +/- ## - Coverage 45.34% 45.33% -0.01% Complexity 8863 8863 Files 1894 1894 Lines 7087970911 +32 Branches 7795 7799 +4 + Hits 3213732148 +11 - Misses3577835799 +21 Partials 2964 2964 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh) | `51.59% <0%> (-4.66%)` | `23 <0> (ø)` | | | [...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==) | `58.36% <100%> (+0.17%)` | `15 <0> (ø)` | :arrow_down: | | [...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh) | `89.61% <16.66%> (-6.17%)` | `6 <0> (ø)` | | | [...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=) | `67.2% <76.92%> (+0.19%)` | `84 <1> (+1)` | :arrow_up: | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70% <0%> (-2.23%)` | `13% <0%> (ø)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0%> (+0.93%)` | `24% <0%> (ø)` | :arrow_down: | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=footer). Last update
[GitHub] [incubator-gobblin] codecov-io commented on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
codecov-io commented on issue #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2791#issuecomment-548577059 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=h1) Report > Merging [#2791](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/f60409ef0b6768bf46ddd137333d8d56981798fc?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `36.36%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2791 +/- ## - Coverage 45.34% 45.33% -0.01% Complexity 8863 8863 Files 1894 1894 Lines 7087970911 +32 Branches 7795 7799 +4 + Hits 3213732148 +11 - Misses3577835799 +21 Partials 2964 2964 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh) | `51.59% <0%> (-4.66%)` | `23 <0> (ø)` | | | [...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==) | `58.36% <100%> (+0.17%)` | `15 <0> (ø)` | :arrow_down: | | [...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh) | `89.61% <16.66%> (-6.17%)` | `6 <0> (ø)` | | | [...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=) | `67.2% <76.92%> (+0.19%)` | `84 <1> (+1)` | :arrow_up: | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `70% <0%> (-2.23%)` | `13% <0%> (ø)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0%> (+0.93%)` | `24% <0%> (ø)` | :arrow_down: | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2791/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=footer). Last update [f60409e...42a3dbe](https://codecov.io/gh/apache/incubator-gobblin/pull/2791?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service,
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337058
]
ASF GitHub Bot logged work on GOBBLIN-941:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 21:17
Start Date: 31/Oct/19 21:17
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on pull request #2790:
[GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving
schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341369711
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
ddl.append(generateAvroToHiveColumnMapping(schema,
Optional.of(hiveColumns), true, dbName + "." + tblName));
+ try {
Review comment:
I want to make sure only when schema evolution is enabled or there is no
existing table, we can set the columns. Because that will overwrite the schema.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337058)
Time Spent: 1h 40m (was: 1.5h)
> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which
> would enforce avro2orc output preserving correct casing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337059 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:19 Start Date: 31/Oct/19 21:19 Worklog Time Spent: 10m Work Description: ZihanLi58 commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341370332 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/materializer/HiveMaterializerFromEntityQueryGenerator.java ## @@ -118,11 +119,19 @@ public QueryBasedHivePublishEntity generatePublishQueries() throws DataConversio Map publishDirectories = publishEntity.getPublishDirectories(); List cleanupQueries = publishEntity.getCleanupQueries(); List cleanupDirectories = publishEntity.getCleanupDirectories(); +Optional avroSchema = Optional.absent(); Review comment: This is not called by Avro2Roc. I add this just to make sure it's consistent with that. But I can remove this one if needed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337059) Time Spent: 1h 50m (was: 1h 40m) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341370332 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/materializer/HiveMaterializerFromEntityQueryGenerator.java ## @@ -118,11 +119,19 @@ public QueryBasedHivePublishEntity generatePublishQueries() throws DataConversio Map publishDirectories = publishEntity.getPublishDirectories(); List cleanupQueries = publishEntity.getCleanupQueries(); List cleanupDirectories = publishEntity.getCleanupDirectories(); +Optional avroSchema = Optional.absent(); Review comment: This is not called by Avro2Roc. I add this just to make sure it's consistent with that. But I can remove this one if needed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance
DDL to add column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341369711
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
ddl.append(generateAvroToHiveColumnMapping(schema,
Optional.of(hiveColumns), true, dbName + "." + tblName));
+ try {
Review comment:
I want to make sure only when schema evolution is enabled or there is no
existing table, we can set the columns. Because that will overwrite the schema.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
Zihan Li created GOBBLIN-941: Summary: Enhance DDL to add column and column.types with case-preserving schema Key: GOBBLIN-941 URL: https://issues.apache.org/jira/browse/GOBBLIN-941 Project: Apache Gobblin Issue Type: Task Reporter: Zihan Li Enhance DDL to add column and column.types with case-preserving schema which would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337047 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:01 Start Date: 31/Oct/19 21:01 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341363858 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/materializer/HiveMaterializerFromEntityQueryGenerator.java ## @@ -118,11 +119,19 @@ public QueryBasedHivePublishEntity generatePublishQueries() throws DataConversio Map publishDirectories = publishEntity.getPublishDirectories(); List cleanupQueries = publishEntity.getCleanupQueries(); List cleanupDirectories = publishEntity.getCleanupDirectories(); +Optional avroSchema = Optional.absent(); Review comment: I might be wrong but is this method being called through the conversion jobs? By briefly looking at the code base, this `generatePublishQueries` method is only called in `org.apache.gobblin.data.management.conversion.hive.materializer.HiveMaterializer#generatePublishQueries` which is not related to Avro2ORC, just want to confirm. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337047) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337045=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337045
]
ASF GitHub Bot logged work on GOBBLIN-941:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 21:01
Start Date: 31/Oct/19 21:01
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2790:
[GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving
schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341361806
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
ddl.append(generateAvroToHiveColumnMapping(schema,
Optional.of(hiveColumns), true, dbName + "." + tblName));
+ try {
Review comment:
Why adding table properties only happens in this branch ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337045)
Time Spent: 1h 10m (was: 1h)
> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which
> would enforce avro2orc output preserving correct casing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337052=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337052 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:09 Start Date: 31/Oct/19 21:09 Worklog Time Spent: 10m Work Description: ZihanLi58 commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341366880 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/task/HiveConverterUtils.java ## @@ -151,6 +154,30 @@ public static String generateCreateDuplicateTableDDL( dbName, tblName, inputDbName, inputTblName, tblLocation); } + public static String generateAlterSchemaDML( + String tableName, + Optional optionalDbName, Review comment: Since I try to make it consistent with method generateCreateDuplicateTableDDL. In which the dbName is optional. I guess it's for testing. But not so sure about that This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337052) Time Spent: 1.5h (was: 1h 20m) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341366880 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/task/HiveConverterUtils.java ## @@ -151,6 +154,30 @@ public static String generateCreateDuplicateTableDDL( dbName, tblName, inputDbName, inputTblName, tblLocation); } + public static String generateAlterSchemaDML( + String tableName, + Optional optionalDbName, Review comment: Since I try to make it consistent with method generateCreateDuplicateTableDDL. In which the dbName is optional. I guess it's for testing. But not so sure about that This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337050 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:08 Start Date: 31/Oct/19 21:08 Worklog Time Spent: 10m Work Description: ZihanLi58 commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341366268 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java ## @@ -85,6 +86,7 @@ * Subdirectory within destination ORC table directory to publish data */ private static final String PUBLISHED_TABLE_SUBDIRECTORY = "final"; + public static final String OUTPUT_AVRO_SCHEMA_KEY = "output.avro.schema"; Review comment: This one will be access in HiveMaterializerFromEntityQueryGenerator when it try to get the schema. So I set it to be public static. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337050) Time Spent: 1h 20m (was: 1h 10m) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341366268 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java ## @@ -85,6 +86,7 @@ * Subdirectory within destination ORC table directory to publish data */ private static final String PUBLISHED_TABLE_SUBDIRECTORY = "final"; + public static final String OUTPUT_AVRO_SCHEMA_KEY = "output.avro.schema"; Review comment: This one will be access in HiveMaterializerFromEntityQueryGenerator when it try to get the schema. So I set it to be public static. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance
DDL to add column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341358581
##
File path:
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/orc/HiveOrcSerDeManager.java
##
@@ -61,8 +61,6 @@
*/
@Slf4j
public class HiveOrcSerDeManager extends HiveSerDeManager {
- // Schema is in the format of TypeDescriptor
- public static final String SCHEMA_LITERAL = "orc.schema.literal";
Review comment:
Do a rebase and push with force option to avoid old commits
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance
DDL to add column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341361806
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
ddl.append(generateAvroToHiveColumnMapping(schema,
Optional.of(hiveColumns), true, dbName + "." + tblName));
+ try {
Review comment:
Why adding table properties only happens in this branch ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance
DDL to add column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341357486
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
Review comment:
Do you still need it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341356298 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java ## @@ -85,6 +86,7 @@ * Subdirectory within destination ORC table directory to publish data */ private static final String PUBLISHED_TABLE_SUBDIRECTORY = "final"; + public static final String OUTPUT_AVRO_SCHEMA_KEY = "output.avro.schema"; Review comment: Does it need to be public static ? Limit access modifiers so that it won't be accidentally touched by irrelevant constructs in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341358264 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/task/HiveConverterUtils.java ## @@ -151,6 +154,30 @@ public static String generateCreateDuplicateTableDDL( dbName, tblName, inputDbName, inputTblName, tblLocation); } + public static String generateAlterSchemaDML( Review comment: Shall we use `generateAlterSerDePropsDML`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341363858 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/materializer/HiveMaterializerFromEntityQueryGenerator.java ## @@ -118,11 +119,19 @@ public QueryBasedHivePublishEntity generatePublishQueries() throws DataConversio Map publishDirectories = publishEntity.getPublishDirectories(); List cleanupQueries = publishEntity.getCleanupQueries(); List cleanupDirectories = publishEntity.getCleanupDirectories(); +Optional avroSchema = Optional.absent(); Review comment: I might be wrong but is this method being called through the conversion jobs? By briefly looking at the code base, this `generatePublishQueries` method is only called in `org.apache.gobblin.data.management.conversion.hive.materializer.HiveMaterializer#generatePublishQueries` which is not related to Avro2ORC, just want to confirm. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
autumnust commented on a change in pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341357961 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/task/HiveConverterUtils.java ## @@ -151,6 +154,30 @@ public static String generateCreateDuplicateTableDDL( dbName, tblName, inputDbName, inputTblName, tblLocation); } + public static String generateAlterSchemaDML( + String tableName, + Optional optionalDbName, Review comment: I am not sure why dbName is optional This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337037 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 20:54 Start Date: 31/Oct/19 20:54 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#issuecomment-548564313 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=h1) Report > Merging [#2790](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2790 +/- ## === - Coverage 4.15% 4.15% -0.01% Complexity 746 746 === Files 18941894 Lines 70877 70911 +34 Branches 77937799 +6 === Hits 29462946 - Misses67617 67651 +34 Partials314 314 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=footer). Last update [9ee4dca...50639f2](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337037) Time Spent: 40m (was: 0.5h) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 >
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337046 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:01 Start Date: 31/Oct/19 21:01 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341357961 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/task/HiveConverterUtils.java ## @@ -151,6 +154,30 @@ public static String generateCreateDuplicateTableDDL( dbName, tblName, inputDbName, inputTblName, tblLocation); } + public static String generateAlterSchemaDML( + String tableName, + Optional optionalDbName, Review comment: I am not sure why dbName is optional This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337046) Time Spent: 1h 10m (was: 1h) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337044
]
ASF GitHub Bot logged work on GOBBLIN-941:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 21:01
Start Date: 31/Oct/19 21:01
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2790:
[GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving
schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341357486
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##
@@ -202,7 +204,21 @@ public static String generateCreateTableDDL(Schema schema,
//.. use columns from destination schema
if (isEvolutionEnabled || !destinationTableMeta.isPresent()) {
log.info("Generating DDL using source schema");
+ System.out.println("Generating DDL using source schema");
Review comment:
Do you still need it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337044)
Time Spent: 1h 10m (was: 1h)
> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which
> would enforce avro2orc output preserving correct casing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337042
]
ASF GitHub Bot logged work on GOBBLIN-941:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 21:01
Start Date: 31/Oct/19 21:01
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2790:
[GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving
schema
URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341358581
##
File path:
gobblin-hive-registration/src/main/java/org/apache/gobblin/hive/orc/HiveOrcSerDeManager.java
##
@@ -61,8 +61,6 @@
*/
@Slf4j
public class HiveOrcSerDeManager extends HiveSerDeManager {
- // Schema is in the format of TypeDescriptor
- public static final String SCHEMA_LITERAL = "orc.schema.literal";
Review comment:
Do a rebase and push with force option to avoid old commits
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337042)
Time Spent: 1h (was: 50m)
> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which
> would enforce avro2orc output preserving correct casing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337043=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337043 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:01 Start Date: 31/Oct/19 21:01 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341356298 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java ## @@ -85,6 +86,7 @@ * Subdirectory within destination ORC table directory to publish data */ private static final String PUBLISHED_TABLE_SUBDIRECTORY = "final"; + public static final String OUTPUT_AVRO_SCHEMA_KEY = "output.avro.schema"; Review comment: Does it need to be public static ? Limit access modifiers so that it won't be accidentally touched by irrelevant constructs in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337043) Time Spent: 1h 10m (was: 1h) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] jhsenjaliya commented on a change in pull request #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
jhsenjaliya commented on a change in pull request #2788: [GOBBLIN-939]
Integrate usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341362631
##
File path: bin/gobblin.sh
##
@@ -283,18 +291,18 @@ elif [[ -f ${GOBBLIN_CONF}/log4j.properties ]]; then
LOG4J_OPTS="-Dlog4j.configuration=$LOG4J_FILE_PATH"
fi
+#Create required directories
if [[ ! -d "$GOBBLIN_LOGS" ]]; then
mkdir -p $GOBBLIN_LOGS
fi
-GC_OPTS=''
-if [[ ${ENABLE_GC_LOGS} -eq 1 ]]; then
-GC_OPTS+="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops "
-GC_OPTS+="-XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution "
-GC_OPTS+="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$GOBBLIN_LOGS/ "
-GC_OPTS+="-Xloggc:$GOBBLIN_LOGS/gobblin-$GOBBLIN_MODE-gc.log "
+if [[ ! -d "$GOBBLIN_WORK_DIR" ]]; then
+mkdir -p $GOBBLIN_WORK_DIR
fi
+if [[ ! -d "$GOBBLIN_JOB_CONFIG_DIR" ]]; then
+mkdir -p $GOBBLIN_JOB_CONFIG_DIR
+fi
Review comment:
do u mean if the change in env variables in `gobblin-env.sh` will get
reflected and get dir created accordingly? yes. since `gobblin.sh` includes the
`gobblin-env.sh`. I have added option to override it from user input as well, i
think it will be useful to maintain any backward compatibility.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337048 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 21:01 Start Date: 31/Oct/19 21:01 Worklog Time Spent: 10m Work Description: autumnust commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#discussion_r341358264 ## File path: gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/task/HiveConverterUtils.java ## @@ -151,6 +154,30 @@ public static String generateCreateDuplicateTableDDL( dbName, tblName, inputDbName, inputTblName, tblLocation); } + public static String generateAlterSchemaDML( Review comment: Shall we use `generateAlterSerDePropsDML`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337048) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337035=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337035
]
ASF GitHub Bot logged work on GOBBLIN-941:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 20:46
Start Date: 31/Oct/19 20:46
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on pull request #2791:
[GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving
schema
URL: https://github.com/apache/incubator-gobblin/pull/2791
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-941
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Enhance DDL to add column and column.types with case-preserving schema which
would enforce avro2orc output preserving correct casing
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Unit test and test the avroToOrc pipeline to make sure the output preserving
correct casing
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337035)
Time Spent: 0.5h (was: 20m)
> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which
> would enforce avro2orc output preserving correct casing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] jhsenjaliya commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
jhsenjaliya commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548566219 > I was wondering, would it be better if we were to utilise javaopts, so `-D` flags for these environment variables, rather than continuously modifying the scripts/configs? u mean like javaopts in bash ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337039
]
ASF GitHub Bot logged work on GOBBLIN-939:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 20:58
Start Date: 31/Oct/19 20:58
Worklog Time Spent: 10m
Work Description: jhsenjaliya commented on pull request #2788:
[GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341362631
##
File path: bin/gobblin.sh
##
@@ -283,18 +291,18 @@ elif [[ -f ${GOBBLIN_CONF}/log4j.properties ]]; then
LOG4J_OPTS="-Dlog4j.configuration=$LOG4J_FILE_PATH"
fi
+#Create required directories
if [[ ! -d "$GOBBLIN_LOGS" ]]; then
mkdir -p $GOBBLIN_LOGS
fi
-GC_OPTS=''
-if [[ ${ENABLE_GC_LOGS} -eq 1 ]]; then
-GC_OPTS+="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops "
-GC_OPTS+="-XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution "
-GC_OPTS+="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$GOBBLIN_LOGS/ "
-GC_OPTS+="-Xloggc:$GOBBLIN_LOGS/gobblin-$GOBBLIN_MODE-gc.log "
+if [[ ! -d "$GOBBLIN_WORK_DIR" ]]; then
+mkdir -p $GOBBLIN_WORK_DIR
fi
+if [[ ! -d "$GOBBLIN_JOB_CONFIG_DIR" ]]; then
+mkdir -p $GOBBLIN_JOB_CONFIG_DIR
+fi
Review comment:
do u mean if the change in env variables in `gobblin-env.sh` will get
reflected and get dir created accordingly? yes. since `gobblin.sh` includes the
`gobblin-env.sh`. I have added option to override it from user input as well, i
think it will be useful to maintain any backward compatibility.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337039)
Time Spent: 1h (was: 50m)
> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
> Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> 1. standardize config with ENV variables
> 2. define default env variables in gobblin-env.sh
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
codecov-io edited a comment on issue #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#issuecomment-548564313 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=h1) Report > Merging [#2790](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.17%`. > The diff coverage is `28.57%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2790 +/- ## = + Coverage 4.15% 45.33% +41.17% - Complexity 746 8864 +8118 = Files 1894 1894 Lines 7087770911 +34 Branches 7793 7799+6 = + Hits 294632148+29202 + Misses6761735797-31820 - Partials314 2966 +2652 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `63.95% <0%> (+63.95%)` | `12 <0> (+12)` | :arrow_up: | | [...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh) | `51.59% <0%> (+51.59%)` | `23 <0> (+23)` | :arrow_up: | | [...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==) | `58.36% <100%> (+58.36%)` | `15 <0> (+15)` | :arrow_up: | | [...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh) | `89.61% <16.66%> (+89.61%)` | `6 <0> (+6)` | :arrow_up: | | [...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=) | `67.2% <76.92%> (+67.2%)` | `84 <1> (+84)` | :arrow_up: | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%>
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337038 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 20:55 Start Date: 31/Oct/19 20:55 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#issuecomment-548564313 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=h1) Report > Merging [#2790](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.17%`. > The diff coverage is `28.57%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2790 +/- ## = + Coverage 4.15% 45.33% +41.17% - Complexity 746 8864 +8118 = Files 1894 1894 Lines 7087770911 +34 Branches 7793 7799+6 = + Hits 294632148+29202 + Misses6761735797-31820 - Partials314 2966 +2652 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `63.95% <0%> (+63.95%)` | `12 <0> (+12)` | :arrow_up: | | [...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh) | `51.59% <0%> (+51.59%)` | `23 <0> (+23)` | :arrow_up: | | [...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==) | `58.36% <100%> (+58.36%)` | `15 <0> (+15)` | :arrow_up: | | [...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh) | `89.61% <16.66%> (+89.61%)` | `6 <0> (+6)` | :arrow_up: | | [...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=) | `67.2% <76.92%> (+67.2%)` | `84 <1> (+84)` | :arrow_up: | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | |
[GitHub] [incubator-gobblin] codecov-io commented on issue #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
codecov-io commented on issue #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790#issuecomment-548564313 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=h1) Report > Merging [#2790](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2790 +/- ## === - Coverage 4.15% 4.15% -0.01% Complexity 746 746 === Files 18941894 Lines 70877 70911 +34 Branches 77937799 +6 === Hits 29462946 - Misses67617 67651 +34 Partials314 314 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ion/hive/converter/AbstractAvroToOrcConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9jb252ZXJ0ZXIvQWJzdHJhY3RBdnJvVG9PcmNDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...nversion/hive/query/HiveAvroORCQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9xdWVyeS9IaXZlQXZyb09SQ1F1ZXJ5R2VuZXJhdG9yLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...izer/HiveMaterializerFromEntityQueryGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9tYXRlcmlhbGl6ZXIvSGl2ZU1hdGVyaWFsaXplckZyb21FbnRpdHlRdWVyeUdlbmVyYXRvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ement/conversion/hive/task/HiveConverterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2790/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS90YXNrL0hpdmVDb252ZXJ0ZXJVdGlscy5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=footer). Last update [9ee4dca...50639f2](https://codecov.io/gh/apache/incubator-gobblin/pull/2790?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-895) Gobblin Standalone Script does not run properly
[ https://issues.apache.org/jira/browse/GOBBLIN-895?focusedWorklogId=337032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337032 ] ASF GitHub Bot logged work on GOBBLIN-895: -- Author: ASF GitHub Bot Created on: 31/Oct/19 20:37 Start Date: 31/Oct/19 20:37 Worklog Time Spent: 10m Work Description: jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate URL: https://github.com/apache/incubator-gobblin/pull/2751#issuecomment-548558371 Sure, let me updates the docs accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337032) Time Spent: 1h 50m (was: 1h 40m) > Gobblin Standalone Script does not run properly > --- > > Key: GOBBLIN-895 > URL: https://issues.apache.org/jira/browse/GOBBLIN-895 > Project: Apache Gobblin > Issue Type: Bug >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Gobblin.sh's changes do not support Gobblin standalone properly as specified > in the getting started docs. This is due to a regression in the > configurations of the app and the logger. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] ZihanLi58 closed pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 closed pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] ZihanLi58 opened a new pull request #2791: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 opened a new pull request #2791: [GOBBLIN-941] Enhance DDL to add
column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2791
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-941
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Enhance DDL to add column and column.types with case-preserving schema which
would enforce avro2orc output preserving correct casing
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Unit test and test the avroToOrc pipeline to make sure the output preserving
correct casing
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[ https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337034 ] ASF GitHub Bot logged work on GOBBLIN-941: -- Author: ASF GitHub Bot Created on: 31/Oct/19 20:45 Start Date: 31/Oct/19 20:45 Worklog Time Spent: 10m Work Description: ZihanLi58 commented on pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema URL: https://github.com/apache/incubator-gobblin/pull/2790 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337034) Time Spent: 20m (was: 10m) > Enhance DDL to add column and column.types with case-preserving schema > -- > > Key: GOBBLIN-941 > URL: https://issues.apache.org/jira/browse/GOBBLIN-941 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Enhance DDL to add column and column.types with case-preserving schema which > would enforce avro2orc output preserving correct casing -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate
jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate URL: https://github.com/apache/incubator-gobblin/pull/2751#issuecomment-548558371 Sure, let me updates the docs accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-924) Get rid of orc.schema.literal in ORC-ingestion and registration
[ https://issues.apache.org/jira/browse/GOBBLIN-924?focusedWorklogId=337031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337031 ] ASF GitHub Bot logged work on GOBBLIN-924: -- Author: ASF GitHub Bot Created on: 31/Oct/19 20:30 Start Date: 31/Oct/19 20:30 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2780: [GOBBLIN-924]Get rid of orc.schema.literal in ORC-ingestion and registration URL: https://github.com/apache/incubator-gobblin/pull/2780 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 337031) Time Spent: 2h (was: 1h 50m) > Get rid of orc.schema.literal in ORC-ingestion and registration > --- > > Key: GOBBLIN-924 > URL: https://issues.apache.org/jira/browse/GOBBLIN-924 > Project: Apache Gobblin > Issue Type: Task >Reporter: Zihan Li >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2780: [GOBBLIN-924]Get rid of orc.schema.literal in ORC-ingestion and registration
asfgit closed pull request #2780: [GOBBLIN-924]Get rid of orc.schema.literal in ORC-ingestion and registration URL: https://github.com/apache/incubator-gobblin/pull/2780 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-941) Enhance DDL to add column and column.types with case-preserving schema
[
https://issues.apache.org/jira/browse/GOBBLIN-941?focusedWorklogId=337022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337022
]
ASF GitHub Bot logged work on GOBBLIN-941:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 20:19
Start Date: 31/Oct/19 20:19
Worklog Time Spent: 10m
Work Description: ZihanLi58 commented on pull request #2790:
[GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving
schema
URL: https://github.com/apache/incubator-gobblin/pull/2790
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-941] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-941
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Enhance DDL to add column and column.types with case-preserving schema which
would enforce avro2orc output preserving correct casing
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Unit test and test the avroToOrc pipeline to make sure the output preserving
correct casing
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 337022)
Remaining Estimate: 0h
Time Spent: 10m
> Enhance DDL to add column and column.types with case-preserving schema
> --
>
> Key: GOBBLIN-941
> URL: https://issues.apache.org/jira/browse/GOBBLIN-941
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Zihan Li
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Enhance DDL to add column and column.types with case-preserving schema which
> would enforce avro2orc output preserving correct casing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] ZihanLi58 opened a new pull request #2790: [GOBBLIN-941] Enhance DDL to add column and column.types with case-preserving schema
ZihanLi58 opened a new pull request #2790: [GOBBLIN-941] Enhance DDL to add
column and column.types with case-preserving schema
URL: https://github.com/apache/incubator-gobblin/pull/2790
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-941] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-941
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Enhance DDL to add column and column.types with case-preserving schema which
would enforce avro2orc output preserving correct casing
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Unit test and test the avroToOrc pipeline to make sure the output preserving
correct casing
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-940) Synchronization between workunit persistency and Helix job launching
[ https://issues.apache.org/jira/browse/GOBBLIN-940?focusedWorklogId=337004=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337004 ] ASF GitHub Bot logged work on GOBBLIN-940: -- Author: ASF GitHub Bot Created on: 31/Oct/19 19:13 Start Date: 31/Oct/19 19:13 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2789: [GOBBLIN-940]Add synchronization on workunit persistency before Helix job launching URL: https://github.com/apache/incubator-gobblin/pull/2789#issuecomment-548527186 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=h1) Report > Merging [#2789](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.19%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2789 +/- ## = + Coverage 4.15% 45.35% +41.19% - Complexity 746 8867 +8121 = Files 1894 1894 Lines 7087770879+2 Branches 7793 7793 = + Hits 294632145+29199 + Misses6761735772-31845 - Partials314 2962 +2648 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `82% <100%> (+82%)` | `27 <0> (+27)` | :arrow_up: | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | |
[GitHub] [incubator-gobblin] codecov-io commented on issue #2789: [GOBBLIN-940]Add synchronization on workunit persistency before Helix job launching
codecov-io commented on issue #2789: [GOBBLIN-940]Add synchronization on workunit persistency before Helix job launching URL: https://github.com/apache/incubator-gobblin/pull/2789#issuecomment-548527186 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=h1) Report > Merging [#2789](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.19%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2789 +/- ## = + Coverage 4.15% 45.35% +41.19% - Complexity 746 8867 +8121 = Files 1894 1894 Lines 7087770879+2 Branches 7793 7793 = + Hits 294632145+29199 + Misses6761735772-31845 - Partials314 2962 +2648 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `82% <100%> (+82%)` | `27 <0> (+27)` | :arrow_up: | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | | [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: | | ... and [1089 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2789/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2789?src=pr=continue). >
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
codecov-io edited a comment on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1) Report > Merging [#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.19%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2788 +/- ## = + Coverage 4.15% 45.35% +41.19% - Complexity 746 8864 +8118 = Files 1894 1894 Lines 7087770877 Branches 7793 7793 = + Hits 294632143+29197 + Misses6761735773-31844 - Partials314 2961 +2647 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | | [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: | | [...apache/gobblin/source/jdbc/SqlServerExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9qZGJjL1NxbFNlcnZlckV4dHJhY3Rvci5qYXZh) | `4.41% <0%> (+4.41%)` | `3% <0%> (+3%)` | :arrow_up: | | ... and [1088 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree-more) | | -- [Continue to review full report at
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336998 ] ASF GitHub Bot logged work on GOBBLIN-939: -- Author: ASF GitHub Bot Created on: 31/Oct/19 18:47 Start Date: 31/Oct/19 18:47 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1) Report > Merging [#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **increase** coverage by `41.19%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2788 +/- ## = + Coverage 4.15% 45.35% +41.19% - Complexity 746 8864 +8118 = Files 1894 1894 Lines 7087770877 Branches 7793 7793 = + Hits 294632143+29197 + Misses6761735773-31844 - Partials314 2961 +2647 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==) | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: | | [...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh) | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==) | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: | | [...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==) | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: | | [...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh) | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: | | [...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh) | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: | | [.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=) | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: | |
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336997 ] ASF GitHub Bot logged work on GOBBLIN-939: -- Author: ASF GitHub Bot Created on: 31/Oct/19 18:46 Start Date: 31/Oct/19 18:46 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1) Report > Merging [#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2788 +/- ## === - Coverage 4.15% 4.15% -0.01% + Complexity 746 745 -1 === Files 18941894 Lines 70877 70877 Branches 77937793 === - Hits 29462945 -1 Misses67617 67617 - Partials314 315 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=footer). Last update [9ee4dca...9209128](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336997) Time Spent: 40m (was: 0.5h) > Integrate usage of env variables in gobblin scripts and configs > --- > > Key: GOBBLIN-939 > URL: https://issues.apache.org/jira/browse/GOBBLIN-939 > Project: Apache Gobblin > Issue Type: Improvement >Affects Versions: 0.14.0 >Reporter: Jay Sen >Priority: Major > Fix For: 0.15.0 > > Time Spent: 40m > Remaining Estimate: 0h > > 1. standardize config with ENV variables > 2. define default env variables in gobblin-env.sh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] codecov-io commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
codecov-io commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1) Report > Merging [#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2788 +/- ## === - Coverage 4.15% 4.15% -0.01% + Complexity 746 745 -1 === Files 18941894 Lines 70877 70877 Branches 77937793 === - Hits 29462945 -1 Misses67617 67617 - Partials314 315 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=footer). Last update [9ee4dca...9209128](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-940) Synchronization between workunit persistency and Helix job launching
[ https://issues.apache.org/jira/browse/GOBBLIN-940?focusedWorklogId=336987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336987 ] ASF GitHub Bot logged work on GOBBLIN-940: -- Author: ASF GitHub Bot Created on: 31/Oct/19 18:27 Start Date: 31/Oct/19 18:27 Worklog Time Spent: 10m Work Description: autumnust commented on issue #2789: [GOBBLIN-940]Add synchronization on workunit persistency before Helix job launching URL: https://github.com/apache/incubator-gobblin/pull/2789#issuecomment-548508801 @htran1 Can you take a look ? thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336987) Time Spent: 20m (was: 10m) > Synchronization between workunit persistency and Helix job launching > > > Key: GOBBLIN-940 > URL: https://issues.apache.org/jira/browse/GOBBLIN-940 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-940) Synchronization between workunit persistency and Helix job launching
[
https://issues.apache.org/jira/browse/GOBBLIN-940?focusedWorklogId=336986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336986
]
ASF GitHub Bot logged work on GOBBLIN-940:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 18:26
Start Date: 31/Oct/19 18:26
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2789:
[GOBBLIN-940]Add synchronization on workunit persistency before Helix job
launching
URL: https://github.com/apache/incubator-gobblin/pull/2789
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] https://issues.apache.org/jira/browse/GOBBLIN-940
### Description
- [x] Did some minor refactoring and unit test fixing, the major part is
just adding `waitForTasks ` before Helix job being returned and submitted. This
is causing problem when HDFS being very slow / parallel runner hitting CPU
bounded on working on workunit persistency that Helix task is activated before
WU is showing up.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 336986)
Remaining Estimate: 0h
Time Spent: 10m
> Synchronization between workunit persistency and Helix job launching
>
>
> Key: GOBBLIN-940
> URL: https://issues.apache.org/jira/browse/GOBBLIN-940
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust commented on issue #2789: [GOBBLIN-940]Add synchronization on workunit persistency before Helix job launching
autumnust commented on issue #2789: [GOBBLIN-940]Add synchronization on workunit persistency before Helix job launching URL: https://github.com/apache/incubator-gobblin/pull/2789#issuecomment-548508801 @htran1 Can you take a look ? thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust opened a new pull request #2789: [GOBBLIN-940]Add synchronization on workunit persistency before Helix job launching
autumnust opened a new pull request #2789: [GOBBLIN-940]Add synchronization on
workunit persistency before Helix job launching
URL: https://github.com/apache/incubator-gobblin/pull/2789
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] https://issues.apache.org/jira/browse/GOBBLIN-940
### Description
- [x] Did some minor refactoring and unit test fixing, the major part is
just adding `waitForTasks ` before Helix job being returned and submitted. This
is causing problem when HDFS being very slow / parallel runner hitting CPU
bounded on working on workunit persistency that Helix task is activated before
WU is showing up.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-940) Synchronization between workunit persistency and Helix job launching
Lei Sun created GOBBLIN-940: --- Summary: Synchronization between workunit persistency and Helix job launching Key: GOBBLIN-940 URL: https://issues.apache.org/jira/browse/GOBBLIN-940 Project: Apache Gobblin Issue Type: Improvement Reporter: Lei Sun -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336968 ] ASF GitHub Bot logged work on GOBBLIN-939: -- Author: ASF GitHub Bot Created on: 31/Oct/19 18:06 Start Date: 31/Oct/19 18:06 Worklog Time Spent: 10m Work Description: Will-Lo commented on pull request #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341287822 ## File path: bin/gobblin-env.sh ## @@ -17,4 +17,6 @@ # limitations under the License. # -# Set Gobblin specific environment variables here. +# export environment variables here to be used in gobblin.sh scripts and gobblin platform configs. +export GOBBLIN_WORK_DIR=$GOBBLIN_HOME/gobblin-work-dir +export GOBBLIN_JOB_CONFIG_DIR=$GOBBLIN_HOME/gobblin-job-config-dir Review comment: I think to keep it more consistent with our old naming conventions we should do `GOBBLIN_WORK_DIR=$GOBBLIN_HOME/work` `GOBBLIN_JOB_CONFIG_DIR=$GOBBLIN_HOME/jobs` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336968) Time Spent: 20m (was: 10m) > Integrate usage of env variables in gobblin scripts and configs > --- > > Key: GOBBLIN-939 > URL: https://issues.apache.org/jira/browse/GOBBLIN-939 > Project: Apache Gobblin > Issue Type: Improvement >Affects Versions: 0.14.0 >Reporter: Jay Sen >Priority: Major > Fix For: 0.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > > 1. standardize config with ENV variables > 2. define default env variables in gobblin-env.sh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336972 ] ASF GitHub Bot logged work on GOBBLIN-939: -- Author: ASF GitHub Bot Created on: 31/Oct/19 18:06 Start Date: 31/Oct/19 18:06 Worklog Time Spent: 10m Work Description: Will-Lo commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548500401 I was wondering, would it be better if we were to utilise javaopts, so `-D` flags for these environment variables, rather than continuously modifying the scripts/configs? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336972) Time Spent: 0.5h (was: 20m) > Integrate usage of env variables in gobblin scripts and configs > --- > > Key: GOBBLIN-939 > URL: https://issues.apache.org/jira/browse/GOBBLIN-939 > Project: Apache Gobblin > Issue Type: Improvement >Affects Versions: 0.14.0 >Reporter: Jay Sen >Priority: Major > Fix For: 0.15.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > 1. standardize config with ENV variables > 2. define default env variables in gobblin-env.sh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336969
]
ASF GitHub Bot logged work on GOBBLIN-939:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 18:06
Start Date: 31/Oct/19 18:06
Worklog Time Spent: 10m
Work Description: Will-Lo commented on pull request #2788: [GOBBLIN-939]
Integrate usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341290769
##
File path: bin/gobblin.sh
##
@@ -283,18 +291,18 @@ elif [[ -f ${GOBBLIN_CONF}/log4j.properties ]]; then
LOG4J_OPTS="-Dlog4j.configuration=$LOG4J_FILE_PATH"
fi
+#Create required directories
if [[ ! -d "$GOBBLIN_LOGS" ]]; then
mkdir -p $GOBBLIN_LOGS
fi
-GC_OPTS=''
-if [[ ${ENABLE_GC_LOGS} -eq 1 ]]; then
-GC_OPTS+="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops "
-GC_OPTS+="-XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution "
-GC_OPTS+="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$GOBBLIN_LOGS/ "
-GC_OPTS+="-Xloggc:$GOBBLIN_LOGS/gobblin-$GOBBLIN_MODE-gc.log "
+if [[ ! -d "$GOBBLIN_WORK_DIR" ]]; then
+mkdir -p $GOBBLIN_WORK_DIR
fi
+if [[ ! -d "$GOBBLIN_JOB_CONFIG_DIR" ]]; then
+mkdir -p $GOBBLIN_JOB_CONFIG_DIR
+fi
Review comment:
If these are initiated using `gobblin-env.sh`, and called in `gobblin.sh`
and never overwritten before the directories are created, would it be possible
to modify the env variables?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 336969)
> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
> Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> 1. standardize config with ENV variables
> 2. define default env variables in gobblin-env.sh
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] Will-Lo commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
Will-Lo commented on issue #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548500401 I was wondering, would it be better if we were to utilise javaopts, so `-D` flags for these environment variables, rather than continuously modifying the scripts/configs? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] Will-Lo commented on a change in pull request #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
Will-Lo commented on a change in pull request #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341287822 ## File path: bin/gobblin-env.sh ## @@ -17,4 +17,6 @@ # limitations under the License. # -# Set Gobblin specific environment variables here. +# export environment variables here to be used in gobblin.sh scripts and gobblin platform configs. +export GOBBLIN_WORK_DIR=$GOBBLIN_HOME/gobblin-work-dir +export GOBBLIN_JOB_CONFIG_DIR=$GOBBLIN_HOME/gobblin-job-config-dir Review comment: I think to keep it more consistent with our old naming conventions we should do `GOBBLIN_WORK_DIR=$GOBBLIN_HOME/work` `GOBBLIN_JOB_CONFIG_DIR=$GOBBLIN_HOME/jobs` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] Will-Lo commented on a change in pull request #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
Will-Lo commented on a change in pull request #2788: [GOBBLIN-939] Integrate
usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341290769
##
File path: bin/gobblin.sh
##
@@ -283,18 +291,18 @@ elif [[ -f ${GOBBLIN_CONF}/log4j.properties ]]; then
LOG4J_OPTS="-Dlog4j.configuration=$LOG4J_FILE_PATH"
fi
+#Create required directories
if [[ ! -d "$GOBBLIN_LOGS" ]]; then
mkdir -p $GOBBLIN_LOGS
fi
-GC_OPTS=''
-if [[ ${ENABLE_GC_LOGS} -eq 1 ]]; then
-GC_OPTS+="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops "
-GC_OPTS+="-XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution "
-GC_OPTS+="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$GOBBLIN_LOGS/ "
-GC_OPTS+="-Xloggc:$GOBBLIN_LOGS/gobblin-$GOBBLIN_MODE-gc.log "
+if [[ ! -d "$GOBBLIN_WORK_DIR" ]]; then
+mkdir -p $GOBBLIN_WORK_DIR
fi
+if [[ ! -d "$GOBBLIN_JOB_CONFIG_DIR" ]]; then
+mkdir -p $GOBBLIN_JOB_CONFIG_DIR
+fi
Review comment:
If these are initiated using `gobblin-env.sh`, and called in `gobblin.sh`
and never overwritten before the directories are created, would it be possible
to modify the env variables?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-895) Gobblin Standalone Script does not run properly
[ https://issues.apache.org/jira/browse/GOBBLIN-895?focusedWorklogId=336947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336947 ] ASF GitHub Bot logged work on GOBBLIN-895: -- Author: ASF GitHub Bot Created on: 31/Oct/19 17:41 Start Date: 31/Oct/19 17:41 Worklog Time Spent: 10m Work Description: jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate URL: https://github.com/apache/incubator-gobblin/pull/2751#issuecomment-548489825 not it looks much better with #2788, pls check it out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336947) Time Spent: 1.5h (was: 1h 20m) > Gobblin Standalone Script does not run properly > --- > > Key: GOBBLIN-895 > URL: https://issues.apache.org/jira/browse/GOBBLIN-895 > Project: Apache Gobblin > Issue Type: Bug >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Gobblin.sh's changes do not support Gobblin standalone properly as specified > in the getting started docs. This is due to a regression in the > configurations of the app and the logger. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (GOBBLIN-931) Fixing Gobblin Documentation typos
[ https://issues.apache.org/jira/browse/GOBBLIN-931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Sun updated GOBBLIN-931: Issue Type: Task (was: Improvement) > Fixing Gobblin Documentation typos > -- > > Key: GOBBLIN-931 > URL: https://issues.apache.org/jira/browse/GOBBLIN-931 > Project: Apache Gobblin > Issue Type: Task >Reporter: Lei Sun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] Will-Lo commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate
Will-Lo commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate URL: https://github.com/apache/incubator-gobblin/pull/2751#issuecomment-548491660 @jhsenjaliya sure we can go with that, I think the main thing is that the documentation should be updated to reflect the change in workflows that we introduce. I think following an approach with #2788 is good, we just need to test it to ensure we don't introduce regressions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-895) Gobblin Standalone Script does not run properly
[ https://issues.apache.org/jira/browse/GOBBLIN-895?focusedWorklogId=336949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336949 ] ASF GitHub Bot logged work on GOBBLIN-895: -- Author: ASF GitHub Bot Created on: 31/Oct/19 17:45 Start Date: 31/Oct/19 17:45 Worklog Time Spent: 10m Work Description: Will-Lo commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate URL: https://github.com/apache/incubator-gobblin/pull/2751#issuecomment-548491660 @jhsenjaliya sure we can go with that, I think the main thing is that the documentation should be updated to reflect the change in workflows that we introduce. I think following an approach with #2788 is good, we just need to test it to ensure we don't introduce regressions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336949) Time Spent: 1h 40m (was: 1.5h) > Gobblin Standalone Script does not run properly > --- > > Key: GOBBLIN-895 > URL: https://issues.apache.org/jira/browse/GOBBLIN-895 > Project: Apache Gobblin > Issue Type: Bug >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Gobblin.sh's changes do not support Gobblin standalone properly as specified > in the getting started docs. This is due to a regression in the > configurations of the app and the logger. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust commented on issue #2783: Update Hive-Avro-To-ORC-Converter.md for typos
autumnust commented on issue #2783: Update Hive-Avro-To-ORC-Converter.md for typos URL: https://github.com/apache/incubator-gobblin/pull/2783#issuecomment-548491406 @bstaudacher Just out of curiosity: Which tool are you planning to address typos given the scale of this code base? I am more than happy to assist on that if you already have a plan. I updated https://issues.apache.org/jira/browse/GOBBLIN-931 to be not specific to Avro-to-ORC. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (GOBBLIN-931) Fixing Gobblin Documentation typos
[ https://issues.apache.org/jira/browse/GOBBLIN-931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Sun updated GOBBLIN-931: Summary: Fixing Gobblin Documentation typos (was: Update Hive-Avro-To-ORC-Converter.md for typos) > Fixing Gobblin Documentation typos > -- > > Key: GOBBLIN-931 > URL: https://issues.apache.org/jira/browse/GOBBLIN-931 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] jhsenjaliya opened a new pull request #2788: [GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
jhsenjaliya opened a new pull request #2788: [GOBBLIN-939] Integrate usage of
env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-939
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
1. standardize config with ENV variables
2. define default env variables in gobblin-env.sh
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: N/A
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate
jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate URL: https://github.com/apache/incubator-gobblin/pull/2751#issuecomment-548489825 not it looks much better with #2788, pls check it out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336946
]
ASF GitHub Bot logged work on GOBBLIN-939:
--
Author: ASF GitHub Bot
Created on: 31/Oct/19 17:40
Start Date: 31/Oct/19 17:40
Worklog Time Spent: 10m
Work Description: jhsenjaliya commented on pull request #2788:
[GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-939
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
1. standardize config with ENV variables
2. define default env variables in gobblin-env.sh
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason: N/A
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 336946)
Remaining Estimate: 0h
Time Spent: 10m
> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
> Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> 1. standardize config with ENV variables
> 2. define default env variables in gobblin-env.sh
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Sen updated GOBBLIN-939: Description: 1. standardize config with ENV variables 2. define default env variables in gobblin-env.sh was:standardize the ability to override gobblin variables via env variable with new gobblin.sh script. > Integrate usage of env variables in gobblin scripts and configs > --- > > Key: GOBBLIN-939 > URL: https://issues.apache.org/jira/browse/GOBBLIN-939 > Project: Apache Gobblin > Issue Type: Improvement >Affects Versions: 0.14.0 >Reporter: Jay Sen >Priority: Major > Fix For: 0.15.0 > > > 1. standardize config with ENV variables > 2. define default env variables in gobblin-env.sh -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs
[ https://issues.apache.org/jira/browse/GOBBLIN-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Sen updated GOBBLIN-939: Summary: Integrate usage of env variables in gobblin scripts and configs (was: standardize the ability to override gobblin variables via env variable with new gobblin.sh script) > Integrate usage of env variables in gobblin scripts and configs > --- > > Key: GOBBLIN-939 > URL: https://issues.apache.org/jira/browse/GOBBLIN-939 > Project: Apache Gobblin > Issue Type: Improvement >Affects Versions: 0.14.0 >Reporter: Jay Sen >Priority: Major > Fix For: 0.15.0 > > > standardize the ability to override gobblin variables via env variable with > new gobblin.sh script. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-939) standardize the ability to override gobblin variables via env variable with new gobblin.sh script
Jay Sen created GOBBLIN-939: --- Summary: standardize the ability to override gobblin variables via env variable with new gobblin.sh script Key: GOBBLIN-939 URL: https://issues.apache.org/jira/browse/GOBBLIN-939 Project: Apache Gobblin Issue Type: Improvement Affects Versions: 0.14.0 Reporter: Jay Sen Fix For: 0.15.0 standardize the ability to override gobblin variables via env variable with new gobblin.sh script. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-895) Gobblin Standalone Script does not run properly
[ https://issues.apache.org/jira/browse/GOBBLIN-895?focusedWorklogId=336925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336925 ] ASF GitHub Bot logged work on GOBBLIN-895: -- Author: ASF GitHub Bot Created on: 31/Oct/19 16:53 Start Date: 31/Oct/19 16:53 Worklog Time Spent: 10m Work Description: jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate URL: https://github.com/apache/incubator-gobblin/pull/2751#issuecomment-548469068 @Will-Lo , I am trying to see how can we make this consistent, I think with addition of GOBBLIN_WORK_DIR and GOBBLIN_JOB_CONFIG_DIR envrionment varialble support with ability to override from command line would perfectly work. what do you think? I will make the changes accordingly. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336925) Time Spent: 1h 20m (was: 1h 10m) > Gobblin Standalone Script does not run properly > --- > > Key: GOBBLIN-895 > URL: https://issues.apache.org/jira/browse/GOBBLIN-895 > Project: Apache Gobblin > Issue Type: Bug >Affects Versions: 0.15.0 >Reporter: William Lo >Priority: Major > Fix For: 0.15.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Gobblin.sh's changes do not support Gobblin standalone properly as specified > in the getting started docs. This is due to a regression in the > configurations of the app and the logger. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate
jhsenjaliya commented on issue #2751: [GOBBLIN-895] Fixes Gobblin Standalone configs and scripts so that user guide is accurate URL: https://github.com/apache/incubator-gobblin/pull/2751#issuecomment-548469068 @Will-Lo , I am trying to see how can we make this consistent, I think with addition of GOBBLIN_WORK_DIR and GOBBLIN_JOB_CONFIG_DIR envrionment varialble support with ability to override from command line would perfectly work. what do you think? I will make the changes accordingly. Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-938) Extend template-resolution in all JobLauncher
[ https://issues.apache.org/jira/browse/GOBBLIN-938?focusedWorklogId=336795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336795 ] ASF GitHub Bot logged work on GOBBLIN-938: -- Author: ASF GitHub Bot Created on: 31/Oct/19 14:21 Start Date: 31/Oct/19 14:21 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2787: [GOBBLIN-938] Make job-template resolution available in all JobLaunchers URL: https://github.com/apache/incubator-gobblin/pull/2787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 336795) Time Spent: 0.5h (was: 20m) > Extend template-resolution in all JobLauncher > - > > Key: GOBBLIN-938 > URL: https://issues.apache.org/jira/browse/GOBBLIN-938 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2787: [GOBBLIN-938] Make job-template resolution available in all JobLaunchers
asfgit closed pull request #2787: [GOBBLIN-938] Make job-template resolution available in all JobLaunchers URL: https://github.com/apache/incubator-gobblin/pull/2787 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
