subject:"\[jira\] \[Work logged\] \(GOBBLIN\-939\) Integrate usage of env variables in gobblin scripts and configs"

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2020-03-10 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=400781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400781
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 10/Mar/20 15:43
Start Date: 10/Mar/20 15:43
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-597157726
 
 
   @Will-Lo , @sv2000 , Can u pls take a look?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 400781)
Time Spent: 4h 10m  (was: 4h)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-12-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=353049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353049
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 03/Dec/19 22:24
Start Date: 03/Dec/19 22:24
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/48d2eb56e0a6b1b570c7cf67489b566976574a66?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `46.96%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2788  +/-   ##
   
   - Coverage 45.62%   45.61%   -0.01% 
   - Complexity 8970 8975   +5 
   
 Files  1902 1903   +1 
 Lines 7127271315  +43 
 Branches   7862 7868   +6 
   
   + Hits  3251832531  +13 
   - Misses3576235790  +28 
   - Partials   2992 2994   +2
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `31.69% <ø> (ø)` | `12 <0> (ø)` | :arrow_down: |
   | 
[...ache/gobblin/kafka/client/KafkaConsumerRecord.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL2NsaWVudC9LYWZrYUNvbnN1bWVyUmVjb3JkLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (?)` | |
   | 
[...bblin/data/management/trash/TrashCollectorJob.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9kYXRhL21hbmFnZW1lbnQvdHJhc2gvVHJhc2hDb2xsZWN0b3JKb2IuamF2YQ==)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...lin/hive/metastore/HiveMetaStoreBasedRegister.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlQmFzZWRSZWdpc3Rlci5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...g/apache/gobblin/azkaban/AzkabanGobblinDaemon.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluRGFlbW9uLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...ava/org/apache/gobblin/util/hadoop/TokenUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvaGFkb29wL1Rva2VuVXRpbHMuamF2YQ==)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[.../apache/gobblin/runtime/task/BaseAbstractTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvdGFzay9CYXNlQWJzdHJhY3RUYXNrLmphdmE=)
 | `61.11% <0%> (-3.6%)` | `6 <0> (ø)` | |
   |

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-12-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=353047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-353047
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 03/Dec/19 22:22
Start Date: 03/Dec/19 22:22
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/48d2eb56e0a6b1b570c7cf67489b566976574a66?src=pr=desc)
 will **decrease** coverage by `41.47%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #2788   +/-   ##
   
   - Coverage 45.62%   4.15%   -41.48% 
   + Complexity 8970 747 -8223 
   
 Files  19021903+1 
 Lines 71272   71315   +43 
 Branches   78627868+6 
   
   - Hits  325182960-29558 
   - Misses35762   68036+32274 
   + Partials   2992 319 -2673
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh)
 | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...che/gobblin/hive/metastore/HiveMetaStoreUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL21ldGFzdG9yZS9IaXZlTWV0YVN0b3JlVXRpbHMuamF2YQ==)
 | `0% <ø> (-31.7%)` | `0 <0> (-12)` | |
   | 
[...ache/gobblin/kafka/client/KafkaConsumerRecord.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL2NsaWVudC9LYWZrYUNvbnN1bWVyUmVjb3JkLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (?)` | |
   | 
[...bblin/data/management/trash/TrashCollectorJob.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9kYXRhL21hbmFnZW1lbnQvdHJhc2gvVHJhc2hDb2xsZWN0b3JKb2IuamF2YQ==)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...org/apache/gobblin/config/client/ConfigClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jb25maWctbWFuYWdlbWVudC9nb2JibGluLWNvbmZpZy1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlnL2NsaWVudC9Db25maWdDbGllbnQuamF2YQ==)
 | `0% <0%> (-44.65%)` | `0 <0> (-15)` | |
   | 
[.../management/conversion/hive/source/HiveSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvbnZlcnNpb24vaGl2ZS9zb3VyY2UvSGl2ZVNvdXJjZS5qYXZh)
 | `0% <0%> (-65.91%)` | `0 <0> (-24)` | |
   | 
[...blin/config/client/ConfigStoreFactoryRegister.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jb25maWctbWFuYWdlbWVudC9nb2JibGluLWNvbmZpZy1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlnL2NsaWVudC9Db25maWdTdG9yZUZhY3RvcnlSZWdpc3Rlci5qYXZh)
 | `0% <0%> (-7.7%)` | `0 <0> (-1)` | |
   | 
[.../service/monitoring/KafkaAvroJobStatusMonitor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUF2cm9Kb2JTdGF0dXNNb25pdG9yLmphdmE=)
 | `0% <0%> (-51.67%)` | `0 <0> (-12)` | |
   |

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-12-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=352963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352963
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 03/Dec/19 20:31
Start Date: 03/Dec/19 20:31
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on pull request #2788: 
[GOBBLIN-939] Integrate usage of env variables in gobblin scripts and 
standardize configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r353406342
 
 

 ##
 File path: conf/gobblin-as-service/application.conf
 ##
 @@ -16,11 +16,12 @@
 #
 
 # Gobblin-As-Service configuration properties
-gobblin.service.work.dir=/tmp/gobblin-as-service
+gobblin.work.dir=${GOBBLIN_WORK_DIR}
 
 Review comment:
   looks like there were some merged recently, will find those and replace with 
`gobblin.work.dir` 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352963)
Time Spent: 3h 40m  (was: 3.5h)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-12-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=352961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352961
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 03/Dec/19 20:30
Start Date: 03/Dec/19 20:30
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on pull request #2788: 
[GOBBLIN-939] Integrate usage of env variables in gobblin scripts and 
standardize configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r353406054
 
 

 ##
 File path: gobblin-docs/user-guide/Gobblin-CLI.md
 ##
 @@ -35,23 +35,27 @@ options:
 keystoreExamine JCE Keystore files
 watermarks  Inspect streaming watermarks
 job-store-schema-managerDatabase job history store schema 
manager
-
---conf-dir  Gobblon config path. default is 
'$GOBBLIN_HOME/conf/'.
---log4j-conf   default is 
'//log4j.properties'.
---jvmopts   String containing JVM flags to include, 
in addition to "-Xmx1g -Xms512m".
---jars Column-separated list of extra jars to 
put on the CLASSPATH.
---enable-gc-logs   enables gc logs & dumps.
---show-classpath   prints gobblin runtime classpath.
---help Display this help.
---verbose  Display full command used to start the 
process.
-   Gobblin Version: 0.15.0
+gobblin-classpath   shows the constructed gobblin 
classpath
+
+--conf-dir Gobblon config path. default is 
'$GOBBLIN_HOME/conf/cli'.
+--log4j-conf  default is 
'/cli/log4j.properties'.
+--work-dir  Gobblin work dir. default is 
/Users/jsenjaliya/tools/gobblin-dist/gobblin-cluster-data.
+--job-conf-dirJob configuration dir to pick up the 
jobs to run. default is /Users/jsenjaliya/tools/gobblin-dist/gobblin-jobs.
+--jvmopts  JVM or GC parameters for the java 
process to append to the default params: "-Xmx1g -Xms512m".
+--jarsColumn-separated list of extra jars 
to put on the CLASSPATH.
+--enable-gc-logs  enables gc logs & dumps.
+--show-classpath  prints gobblin runtime classpath.
+--helpDisplay this help.
+--verbose Display full command used to start 
the process.
 ```
 
 
 Argument details:
-* `--conf-dir`: specifies the path to directory containing gobblin system 
configuration files, like `application.conf` or `reference.conf`, 
`log4j.properties` and `quartz.properties`.
+* `--conf-dir`: specifies the path to directory containing gobblin system 
configuration files, like `application.conf` or `reference.conf`, 
`log4j.properties` and `quartz.properties`, as this is provided as classpath 
for config file look up.
 
 Review comment:
   meant `--conf-dir` value is supplied as classpath for any config file to be 
looked up. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352961)
Time Spent: 3.5h  (was: 3h 20m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-12-03 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=352870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-352870
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 03/Dec/19 18:37
Start Date: 03/Dec/19 18:37
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on pull request #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r353352604
 
 

 ##
 File path: conf/gobblin-as-service/application.conf
 ##
 @@ -16,11 +16,12 @@
 #
 
 # Gobblin-As-Service configuration properties
-gobblin.service.work.dir=/tmp/gobblin-as-service
+gobblin.work.dir=${GOBBLIN_WORK_DIR}
 
 Review comment:
   So I tested the scripts and some execution modes still seem to have 
references to the old variable of the old directory 
e.g.`gobblin.service.work.dir`. Could we fix these to ensure the scripts are 
all working?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 352870)
Time Spent: 3h 20m  (was: 3h 10m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=350848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350848
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 28/Nov/19 07:45
Start Date: 28/Nov/19 07:45
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-559379162
 
 
   fixed the typo.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 350848)
Time Spent: 3h 10m  (was: 3h)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=350601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350601
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 27/Nov/19 18:03
Start Date: 27/Nov/19 18:03
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-559194748
 
 
   Going to test this out first later today, thanks for the additions!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 350601)
Time Spent: 3h  (was: 2h 50m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=350599=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350599
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 27/Nov/19 18:03
Start Date: 27/Nov/19 18:03
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on pull request #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r351430223
 
 

 ##
 File path: gobblin-docs/user-guide/Gobblin-CLI.md
 ##
 @@ -35,23 +35,27 @@ options:
 keystoreExamine JCE Keystore files
 watermarks  Inspect streaming watermarks
 job-store-schema-managerDatabase job history store schema 
manager
-
---conf-dir  Gobblon config path. default is 
'$GOBBLIN_HOME/conf/'.
---log4j-conf   default is 
'//log4j.properties'.
---jvmopts   String containing JVM flags to include, 
in addition to "-Xmx1g -Xms512m".
---jars Column-separated list of extra jars to 
put on the CLASSPATH.
---enable-gc-logs   enables gc logs & dumps.
---show-classpath   prints gobblin runtime classpath.
---help Display this help.
---verbose  Display full command used to start the 
process.
-   Gobblin Version: 0.15.0
+gobblin-classpath   shows the constructed gobblin 
classpath
+
+--conf-dir Gobblon config path. default is 
'$GOBBLIN_HOME/conf/cli'.
+--log4j-conf  default is 
'/cli/log4j.properties'.
+--work-dir  Gobblin work dir. default is 
/Users/jsenjaliya/tools/gobblin-dist/gobblin-cluster-data.
+--job-conf-dirJob configuration dir to pick up the 
jobs to run. default is /Users/jsenjaliya/tools/gobblin-dist/gobblin-jobs.
+--jvmopts  JVM or GC parameters for the java 
process to append to the default params: "-Xmx1g -Xms512m".
+--jarsColumn-separated list of extra jars 
to put on the CLASSPATH.
+--enable-gc-logs  enables gc logs & dumps.
+--show-classpath  prints gobblin runtime classpath.
+--helpDisplay this help.
+--verbose Display full command used to start 
the process.
 ```
 
 
 Argument details:
-* `--conf-dir`: specifies the path to directory containing gobblin system 
configuration files, like `application.conf` or `reference.conf`, 
`log4j.properties` and `quartz.properties`.
+* `--conf-dir`: specifies the path to directory containing gobblin system 
configuration files, like `application.conf` or `reference.conf`, 
`log4j.properties` and `quartz.properties`, as this is provided as classpath 
for config file look up.
 
 Review comment:
   I find this addition confusing, what do you mean by `provided as classpath?`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 350599)
Time Spent: 2h 50m  (was: 2h 40m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=350598=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350598
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 27/Nov/19 18:03
Start Date: 27/Nov/19 18:03
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on pull request #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r351428413
 
 

 ##
 File path: bin/gobblin.sh
 ##
 @@ -106,9 +106,11 @@ function print_gobblin_cli_usage() {
 job-store-schema-managerDatabase job history store schema 
manager
 gobblin-classpath   shows the constructed gobblin 
classpath"
 echo ""
-echo "--conf-dir Gobblon config path. 
default is '\$GOBBLIN_HOME/conf/cli'."
-echo "--log4j-conf  default is 
'/cli/log4j.properties'."
-echo "--jvmopts  JVM or GC parameters for 
the java process to append to the default params: \"$JVM_OPTS\"."
+echo "--conf-dir Gobblon config path. 
default is '\$GOBBLIN_HOME/conf/'."
 
 Review comment:
   typo: Gobblin config path
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 350598)
Time Spent: 2h 40m  (was: 2.5h)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=350600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350600
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 27/Nov/19 18:03
Start Date: 27/Nov/19 18:03
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on pull request #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r351429988
 
 

 ##
 File path: gobblin-docs/user-guide/Gobblin-CLI.md
 ##
 @@ -35,23 +35,27 @@ options:
 keystoreExamine JCE Keystore files
 watermarks  Inspect streaming watermarks
 job-store-schema-managerDatabase job history store schema 
manager
-
---conf-dir  Gobblon config path. default is 
'$GOBBLIN_HOME/conf/'.
---log4j-conf   default is 
'//log4j.properties'.
---jvmopts   String containing JVM flags to include, 
in addition to "-Xmx1g -Xms512m".
---jars Column-separated list of extra jars to 
put on the CLASSPATH.
---enable-gc-logs   enables gc logs & dumps.
---show-classpath   prints gobblin runtime classpath.
---help Display this help.
---verbose  Display full command used to start the 
process.
-   Gobblin Version: 0.15.0
+gobblin-classpath   shows the constructed gobblin 
classpath
+
+--conf-dir Gobblon config path. default is 
'$GOBBLIN_HOME/conf/cli'.
 
 Review comment:
   typo: Gobblin config path
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 350600)
Time Spent: 2h 50m  (was: 2h 40m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-27 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=350587=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-350587
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 27/Nov/19 17:55
Start Date: 27/Nov/19 17:55
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-559191983
 
 
   I'll take a look, thanks!
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 350587)
Time Spent: 2.5h  (was: 2h 20m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-26 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=349961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-349961
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 26/Nov/19 19:19
Start Date: 26/Nov/19 19:19
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-558779247
 
 
   @Will-Lo,can u pls review this ? 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 349961)
Time Spent: 2h 20m  (was: 2h 10m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-22 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=348436=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-348436
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 23/Nov/19 01:21
Start Date: 23/Nov/19 01:21
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/48d2eb56e0a6b1b570c7cf67489b566976574a66?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2788  +/-   ##
   
   - Coverage 45.62%   45.62%   -0.01% 
   + Complexity 8970 8969   -1 
   
 Files  1902 1902  
 Lines 7127271272  
 Branches   7862 7862  
   
   - Hits  3251832516   -2 
   - Misses3576235764   +2 
 Partials   2992 2992
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `35.51% <0%> (-3.74%)` | `12% <0%> (-1%)` | |
   | 
[...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==)
 | `30.53% <0%> (+0.33%)` | `24% <0%> (ø)` | :arrow_down: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=footer).
 Last update 
[48d2eb5...af58c46](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 348436)
Time Spent: 2h 10m  (was: 2h)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-22 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=348432=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-348432
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 23/Nov/19 00:26
Start Date: 23/Nov/19 00:26
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-557742549
 
 
   @Will-Lo , @sv2000  can u pls take a look, This will make all exec mode 
config very standard with the gobblin script changes. Thanks
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 348432)
Time Spent: 2h  (was: 1h 50m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337575
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 21:49
Start Date: 01/Nov/19 21:49
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr=desc)
 will **decrease** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2788  +/-   ##
   
   - Coverage 45.34%   45.32%   -0.03% 
   + Complexity 8864 8859   -5 
   
 Files  1894 1894  
 Lines 7087970879  
 Branches   7795 7795  
   
   - Hits  3214132126  -15 
   - Misses3577635788  +12 
   - Partials   2962 2965   +3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/async/AsyncDataDispatcher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2FzeW5jL0FzeW5jRGF0YURpc3BhdGNoZXIuamF2YQ==)
 | `79.66% <0%> (-8.48%)` | `13% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `92.85% <0%> (-7.15%)` | `3% <0%> (-1%)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `35.51% <0%> (-3.74%)` | `12% <0%> (-1%)` | |
   | 
[.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==)
 | `63.42% <0%> (-1.86%)` | `27% <0%> (-1%)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=footer).
 Last update 
[60e3ab5...8c7f1be](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337575)
Time Spent: 1h 50m  (was: 1h 40m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-11-01 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337490
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 01/Nov/19 19:08
Start Date: 01/Nov/19 19:08
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and standardize configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/60e3ab56a6a5bd1fd7c7c31586d6739581fe354f?src=pr=desc)
 will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master#2788  +/-   ##
   
   - Coverage 45.34%   45.32%   -0.02% 
   + Complexity 8864 8861   -3 
   
 Files  1894 1894  
 Lines 7087970879  
 Branches   7795 7795  
   
   - Hits  3214132129  -12 
   - Misses3577635787  +11 
   - Partials   2962 2963   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...a/org/apache/gobblin/util/limiter/NoopLimiter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvbGltaXRlci9Ob29wTGltaXRlci5qYXZh)
 | `40% <0%> (-20%)` | `2% <0%> (-1%)` | |
   | 
[...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=)
 | `85.71% <0%> (-14.29%)` | `3% <0%> (-1%)` | |
   | 
[...in/java/org/apache/gobblin/cluster/SingleTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFzay5qYXZh)
 | `73.58% <0%> (-7.55%)` | `9% <0%> (ø)` | |
   | 
[...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh)
 | `32.71% <0%> (-6.55%)` | `11% <0%> (-2%)` | |
   | 
[...a/org/apache/gobblin/cluster/GobblinHelixTask.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4VGFzay5qYXZh)
 | `76.08% <0%> (-4.35%)` | `5% <0%> (ø)` | |
   | 
[.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=)
 | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | |
   | 
[...main/java/org/apache/gobblin/util/HadoopUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvSGFkb29wVXRpbHMuamF2YQ==)
 | `30.87% <0%> (+0.67%)` | `25% <0%> (+1%)` | :arrow_up: |
   | 
[...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh)
 | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: |
   | 
[...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=)
 | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337081
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 22:37
Start Date: 31/Oct/19 22:37
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc)
 will **increase** coverage by `41.18%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@  Coverage Diff  @@
   ## master#2788   +/-   ##
   =
   + Coverage  4.15%   45.34%   +41.18% 
   - Complexity  746 8863 +8117 
   =
 Files  1894 1894   
 Lines 7087770879+2 
 Branches   7793 7795+2 
   =
   + Hits   294632139+29193 
   + Misses6761735778-31839 
   - Partials314 2962 +2648
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | |
   | 
[...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==)
 | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: |
   | 
[...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh)
 | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: |
   | 
[...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=)
 | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: |
   | 
[...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==)
 | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: |
   | 
[...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==)
 | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: |
   | 
[...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh)
 | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: |
   | 
[...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh)
 | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: |
   | 
[.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=)
 | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: |
   |

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337040
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 20:59
Start Date: 31/Oct/19 20:59
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548566219
 
 
   > I was wondering, would it be better if we were to utilise javaopts, so 
`-D` flags for these environment variables, rather than continuously modifying 
the scripts/configs?
   
   u mean like javaopts in bash ?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337040)
Time Spent: 1h 10m  (was: 1h)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337068
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 21:42
Start Date: 31/Oct/19 21:42
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc)
 will **increase** coverage by `41.18%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@  Coverage Diff  @@
   ## master#2788   +/-   ##
   =
   + Coverage  4.15%   45.33%   +41.18% 
   - Complexity  746 8863 +8117 
   =
 Files  1894 1894   
 Lines 7087770879+2 
 Branches   7793 7795+2 
   =
   + Hits   294632136+29190 
   + Misses6761735781-31836 
   - Partials314 2962 +2648
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==)
 | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: |
   | 
[...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh)
 | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: |
   | 
[...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=)
 | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: |
   | 
[...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==)
 | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: |
   | 
[...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==)
 | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: |
   | 
[...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh)
 | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: |
   | 
[...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh)
 | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: |
   | 
[.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=)
 | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: |
   | 
[...apache/gobblin/source/jdbc/SqlServerExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9qZGJjL1NxbFNlcnZlckV4dHJhY3Rvci5qYXZh)
 | `4.41% <0%> (+4.41%)` | `3% <0%> (+3%)` | :arrow_up: |
   |

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=337039=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-337039
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 20:58
Start Date: 31/Oct/19 20:58
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on pull request #2788: 
[GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341362631
 
 

 ##
 File path: bin/gobblin.sh
 ##
 @@ -283,18 +291,18 @@ elif [[ -f ${GOBBLIN_CONF}/log4j.properties ]]; then
 LOG4J_OPTS="-Dlog4j.configuration=$LOG4J_FILE_PATH"
 fi
 
+#Create required directories
 if [[ ! -d "$GOBBLIN_LOGS" ]]; then
 mkdir -p $GOBBLIN_LOGS
 fi
 
-GC_OPTS=''
-if [[ ${ENABLE_GC_LOGS} -eq 1 ]]; then
-GC_OPTS+="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops "
-GC_OPTS+="-XX:+PrintGCDetails -XX:+PrintGCDateStamps 
-XX:+PrintTenuringDistribution "
-GC_OPTS+="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$GOBBLIN_LOGS/ "
-GC_OPTS+="-Xloggc:$GOBBLIN_LOGS/gobblin-$GOBBLIN_MODE-gc.log "
+if [[ ! -d "$GOBBLIN_WORK_DIR" ]]; then
+mkdir -p $GOBBLIN_WORK_DIR
 fi
 
+if [[ ! -d "$GOBBLIN_JOB_CONFIG_DIR" ]]; then
+mkdir -p $GOBBLIN_JOB_CONFIG_DIR
+fi
 
 
 Review comment:
   do u mean if the change in env variables in `gobblin-env.sh` will get 
reflected and get dir created accordingly? yes. since `gobblin.sh` includes the 
`gobblin-env.sh`. I have added option to override it from user input as well, i 
think it will be useful to maintain any backward compatibility.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 337039)
Time Spent: 1h  (was: 50m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336998
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 18:47
Start Date: 31/Oct/19 18:47
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc)
 will **increase** coverage by `41.19%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@  Coverage Diff  @@
   ## master#2788   +/-   ##
   =
   + Coverage  4.15%   45.35%   +41.19% 
   - Complexity  746 8864 +8118 
   =
 Files  1894 1894   
 Lines 7087770877   
 Branches   7793 7793   
   =
   + Hits   294632143+29197 
   + Misses6761735773-31844 
   - Partials314 2961 +2647
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | |
   | 
[...gobblin/service/monitoring/JobStatusRetriever.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXNSZXRyaWV2ZXIuamF2YQ==)
 | `0% <0%> (ø)` | `2% <0%> (+2%)` | :arrow_up: |
   | 
[...ata/management/copy/hive/HivePartitionFileSet.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlUGFydGl0aW9uRmlsZVNldC5qYXZh)
 | `0.95% <0%> (+0.95%)` | `1% <0%> (+1%)` | :arrow_up: |
   | 
[...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=)
 | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: |
   | 
[...pache/gobblin/runtime/GobblinMultiTaskAttempt.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvR29iYmxpbk11bHRpVGFza0F0dGVtcHQuamF2YQ==)
 | `56.3% <0%> (+1.35%)` | `27% <0%> (+2%)` | :arrow_up: |
   | 
[...ava/org/apache/gobblin/runtime/MultiConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvTXVsdGlDb252ZXJ0ZXIuamF2YQ==)
 | `83.6% <0%> (+1.63%)` | `9% <0%> (+1%)` | :arrow_up: |
   | 
[...rg/apache/gobblin/runtime/FsDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvRnNEYXRhc2V0U3RhdGVTdG9yZS5qYXZh)
 | `73.8% <0%> (+1.78%)` | `35% <0%> (+1%)` | :arrow_up: |
   | 
[...a/org/apache/gobblin/cluster/SingleTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvU2luZ2xlVGFza1J1bm5lci5qYXZh)
 | `1.85% <0%> (+1.85%)` | `1% <0%> (+1%)` | :arrow_up: |
   | 
[.../java/org/apache/gobblin/runtime/TaskExecutor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza0V4ZWN1dG9yLmphdmE=)
 | `45.05% <0%> (+2.19%)` | `9% <0%> (+1%)` | :arrow_up: |
   |

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336997
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 18:46
Start Date: 31/Oct/19 18:46
Worklog Time Spent: 10m 
  Work Description: codecov-io commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548517189
 
 
   # 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=h1)
 Report
   > Merging 
[#2788](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=desc)
 into 
[master](https://codecov.io/gh/apache/incubator-gobblin/commit/9ee4dcaf66257b6e2926cf1470b16b912cd343ff?src=pr=desc)
 will **decrease** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/graphs/tree.svg?width=650=4MgURJ0bGc=150=pr)](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree)
   
   ```diff
   @@ Coverage Diff @@
   ## master   #2788  +/-   ##
   ===
   - Coverage  4.15%   4.15%   -0.01% 
   + Complexity  746 745   -1 
   ===
 Files  18941894  
 Lines 70877   70877  
 Branches   77937793  
   ===
   - Hits   29462945   -1 
 Misses67617   67617  
   - Partials314 315   +1
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=tree) 
| Coverage Δ | Complexity Δ | |
   |---|---|---|---|
   | 
[...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2788/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=)
 | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=footer).
 Last update 
[9ee4dca...9209128](https://codecov.io/gh/apache/incubator-gobblin/pull/2788?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 336997)
Time Spent: 40m  (was: 0.5h)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336968
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 18:06
Start Date: 31/Oct/19 18:06
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on pull request #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341287822
 
 

 ##
 File path: bin/gobblin-env.sh
 ##
 @@ -17,4 +17,6 @@
 # limitations under the License.
 #
 
-# Set Gobblin specific environment variables here.
+# export environment variables here to be used in gobblin.sh scripts and 
gobblin platform configs.
+export GOBBLIN_WORK_DIR=$GOBBLIN_HOME/gobblin-work-dir
+export GOBBLIN_JOB_CONFIG_DIR=$GOBBLIN_HOME/gobblin-job-config-dir
 
 Review comment:
   I think to keep it more consistent with our old naming conventions we should 
do
`GOBBLIN_WORK_DIR=$GOBBLIN_HOME/work`
   `GOBBLIN_JOB_CONFIG_DIR=$GOBBLIN_HOME/jobs`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 336968)
Time Spent: 20m  (was: 10m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336972
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 18:06
Start Date: 31/Oct/19 18:06
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on issue #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and configs
URL: 
https://github.com/apache/incubator-gobblin/pull/2788#issuecomment-548500401
 
 
   I was wondering, would it be better if we were to utilise javaopts, so `-D` 
flags for these environment variables, rather than continuously modifying the 
scripts/configs?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 336972)
Time Spent: 0.5h  (was: 20m)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336969
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 18:06
Start Date: 31/Oct/19 18:06
Worklog Time Spent: 10m 
  Work Description: Will-Lo commented on pull request #2788: [GOBBLIN-939] 
Integrate usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788#discussion_r341290769
 
 

 ##
 File path: bin/gobblin.sh
 ##
 @@ -283,18 +291,18 @@ elif [[ -f ${GOBBLIN_CONF}/log4j.properties ]]; then
 LOG4J_OPTS="-Dlog4j.configuration=$LOG4J_FILE_PATH"
 fi
 
+#Create required directories
 if [[ ! -d "$GOBBLIN_LOGS" ]]; then
 mkdir -p $GOBBLIN_LOGS
 fi
 
-GC_OPTS=''
-if [[ ${ENABLE_GC_LOGS} -eq 1 ]]; then
-GC_OPTS+="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCompressedOops "
-GC_OPTS+="-XX:+PrintGCDetails -XX:+PrintGCDateStamps 
-XX:+PrintTenuringDistribution "
-GC_OPTS+="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$GOBBLIN_LOGS/ "
-GC_OPTS+="-Xloggc:$GOBBLIN_LOGS/gobblin-$GOBBLIN_MODE-gc.log "
+if [[ ! -d "$GOBBLIN_WORK_DIR" ]]; then
+mkdir -p $GOBBLIN_WORK_DIR
 fi
 
+if [[ ! -d "$GOBBLIN_JOB_CONFIG_DIR" ]]; then
+mkdir -p $GOBBLIN_JOB_CONFIG_DIR
+fi
 
 
 Review comment:
   If these are initiated using `gobblin-env.sh`, and called in `gobblin.sh` 
and never overwritten before the directories are created, would it be possible 
to modify the env variables?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 336969)

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

2019-10-31 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/GOBBLIN-939?focusedWorklogId=336946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-336946
 ]

ASF GitHub Bot logged work on GOBBLIN-939:
--

Author: ASF GitHub Bot
Created on: 31/Oct/19 17:40
Start Date: 31/Oct/19 17:40
Worklog Time Spent: 10m 
  Work Description: jhsenjaliya commented on pull request #2788: 
[GOBBLIN-939] Integrate usage of env variables in gobblin scripts and configs
URL: https://github.com/apache/incubator-gobblin/pull/2788
 
 
   Dear Gobblin maintainers,
   
   Please accept this PR. I understand that it will not be reviewed until I 
have checked off all the steps below!
   
   
   ### JIRA
   - [x] My PR addresses the following [Gobblin 
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references 
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
   - https://issues.apache.org/jira/browse/GOBBLIN-939
   
   
   ### Description
   - [x] Here are some details about my PR, including screenshots (if 
applicable):
   1. standardize config with ENV variables 
   2. define default env variables in gobblin-env.sh
   
   ### Tests
   - [x] My PR adds the following unit tests __OR__ does not need testing for 
this extremely good reason: N/A
   
   
   ### Commits
   - [x] My commits all reference JIRA issues in their subject lines, and I 
have squashed multiple commits if they address the same issue. In addition, my 
commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
   1. Subject is separated from body by a blank line
   2. Subject is limited to 50 characters
   3. Subject does not end with a period
   4. Subject uses the imperative mood ("add", not "adding")
   5. Body wraps at 72 characters
   6. Body explains "what" and "why", not "how"
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 336946)
Remaining Estimate: 0h
Time Spent: 10m

> Integrate usage of env variables in gobblin scripts and configs
> ---
>
> Key: GOBBLIN-939
> URL: https://issues.apache.org/jira/browse/GOBBLIN-939
> Project: Apache Gobblin
>  Issue Type: Improvement
>Affects Versions: 0.14.0
>Reporter: Jay Sen
>Priority: Major
> Fix For: 0.15.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 1. standardize config with ENV variables 
> 2. define default env variables in gobblin-env.sh



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

[jira] [Work logged] (GOBBLIN-939) Integrate usage of env variables in gobblin scripts and configs

27 matches

Site Navigation

Mail list logo

Footer information