[jira] [Work logged] (GOBBLIN-1060) Fix YarnAppLauncher resource existence checking with wrong fs object
[ https://issues.apache.org/jira/browse/GOBBLIN-1060?focusedWorklogId=393142=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-393142 ] ASF GitHub Bot logged work on GOBBLIN-1060: --- Author: ASF GitHub Bot Created on: 26/Feb/20 02:34 Start Date: 26/Feb/20 02:34 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher URL: https://github.com/apache/incubator-gobblin/pull/2899#issuecomment-591042125 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=h1) Report > Merging [#2899](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/0de109b6941e636f4a35069752ef1e3c4188054d?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2899 +/- ## - Coverage 45.86% 45.85% -0.01% + Complexity 9170 9167 -3 Files 1932 1932 Lines 7278972792 +3 Branches 8022 8022 - Hits 3338633382 -4 - Misses3634236352 +10 + Partials 3061 3058 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `21.02% <0%> (-0.16%)` | `8 <0> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-3.48%)` | `14% <0%> (-1%)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==) | `15.32% <0%> (-0.84%)` | `4% <0%> (-1%)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0%> (+0.93%)` | `24% <0%> (ø)` | :arrow_down: | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `79.68% <0%> (+1.56%)` | `16% <0%> (+1%)` | :arrow_up: | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=footer). Last update [0de109b...5c71800](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher
codecov-io edited a comment on issue #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher URL: https://github.com/apache/incubator-gobblin/pull/2899#issuecomment-591042125 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=h1) Report > Merging [#2899](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/0de109b6941e636f4a35069752ef1e3c4188054d?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2899 +/- ## - Coverage 45.86% 45.85% -0.01% + Complexity 9170 9167 -3 Files 1932 1932 Lines 7278972792 +3 Branches 8022 8022 - Hits 3338633382 -4 - Misses3634236352 +10 + Partials 3061 3058 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `21.02% <0%> (-0.16%)` | `8 <0> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-3.48%)` | `14% <0%> (-1%)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | [...main/java/org/apache/gobblin/yarn/YarnService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vWWFyblNlcnZpY2UuamF2YQ==) | `15.32% <0%> (-0.84%)` | `4% <0%> (-1%)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `80.37% <0%> (+0.93%)` | `24% <0%> (ø)` | :arrow_down: | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `79.68% <0%> (+1.56%)` | `16% <0%> (+1%)` | :arrow_up: | | [...lin/restli/throttling/ZookeeperLeaderElection.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2UvZ29iYmxpbi10aHJvdHRsaW5nLXNlcnZpY2Utc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3Jlc3RsaS90aHJvdHRsaW5nL1pvb2tlZXBlckxlYWRlckVsZWN0aW9uLmphdmE=) | `72.22% <0%> (+2.22%)` | `13% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=footer). Last update [0de109b...5c71800](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1050) Verify requester when updating/deleting FlowConfig
[ https://issues.apache.org/jira/browse/GOBBLIN-1050?focusedWorklogId=393115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-393115 ] ASF GitHub Bot logged work on GOBBLIN-1050: --- Author: ASF GitHub Bot Created on: 26/Feb/20 01:42 Start Date: 26/Feb/20 01:42 Worklog Time Spent: 10m Work Description: arjun4084346 commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig URL: https://github.com/apache/incubator-gobblin/pull/2890#issuecomment-591191796 I did not like the idea of updating signature of update at multiple places. It made the code non-uniform. Place of requesterList as a parameter in update() is not quite justified in my opinion. Is it possible to do checkRequester() in FlowConfigsResource::update/delete instead of doing in resource handlers' update/delete ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 393115) Time Spent: 40m (was: 0.5h) > Verify requester when updating/deleting FlowConfig > -- > > Key: GOBBLIN-1050 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1050 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig
arjun4084346 commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig URL: https://github.com/apache/incubator-gobblin/pull/2890#issuecomment-591191796 I did not like the idea of updating signature of update at multiple places. It made the code non-uniform. Place of requesterList as a parameter in update() is not quite justified in my opinion. Is it possible to do checkRequester() in FlowConfigsResource::update/delete instead of doing in resource handlers' update/delete ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher
sv2000 commented on a change in pull request #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher URL: https://github.com/apache/incubator-gobblin/pull/2899#discussion_r384128391 ## File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java ## @@ -608,21 +608,22 @@ private Resource prepareContainerResource(GetNewApplicationResponse newApplicati Path appMasterWorkDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.APP_MASTER_WORK_DIR_NAME); Map appMasterResources = Maps.newHashMap(); +FileSystem localFs = FileSystem.getLocal(this.yarnConfiguration); Review comment: Maybe it is safer to use FileSystem.getLocal(new Configuration()) since we are dependent of fs.defaultFS not being configured in yarnConfiguration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1060) Fix YarnAppLauncher resource existence checking with wrong fs object
[ https://issues.apache.org/jira/browse/GOBBLIN-1060?focusedWorklogId=392932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392932 ] ASF GitHub Bot logged work on GOBBLIN-1060: --- Author: ASF GitHub Bot Created on: 25/Feb/20 21:11 Start Date: 25/Feb/20 21:11 Worklog Time Spent: 10m Work Description: sv2000 commented on pull request #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher URL: https://github.com/apache/incubator-gobblin/pull/2899#discussion_r384128391 ## File path: gobblin-yarn/src/main/java/org/apache/gobblin/yarn/GobblinYarnAppLauncher.java ## @@ -608,21 +608,22 @@ private Resource prepareContainerResource(GetNewApplicationResponse newApplicati Path appMasterWorkDir = new Path(appWorkDir, GobblinYarnConfigurationKeys.APP_MASTER_WORK_DIR_NAME); Map appMasterResources = Maps.newHashMap(); +FileSystem localFs = FileSystem.getLocal(this.yarnConfiguration); Review comment: Maybe it is safer to use FileSystem.getLocal(new Configuration()) since we are dependent of fs.defaultFS not being configured in yarnConfiguration. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392932) Time Spent: 0.5h (was: 20m) > Fix YarnAppLauncher resource existence checking with wrong fs object > > > Key: GOBBLIN-1060 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1060 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Lei Sun >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1060) Fix YarnAppLauncher resource existence checking with wrong fs object
[ https://issues.apache.org/jira/browse/GOBBLIN-1060?focusedWorklogId=392881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392881 ] ASF GitHub Bot logged work on GOBBLIN-1060: --- Author: ASF GitHub Bot Created on: 25/Feb/20 20:03 Start Date: 25/Feb/20 20:03 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher URL: https://github.com/apache/incubator-gobblin/pull/2899#issuecomment-591042125 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=h1) Report > Merging [#2899](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/0de109b6941e636f4a35069752ef1e3c4188054d?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2899 +/- ## === - Coverage 4.12% 4.12% -0.01% Complexity 750 750 === Files 19321932 Lines 72789 72792 +3 Branches 80228022 === Hits 30023002 - Misses69469 69472 +3 Partials318 318 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=footer). Last update [0de109b...0abc430](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392881) Time Spent: 20m (was: 10m) > Fix YarnAppLauncher resource existence checking with wrong fs object > > > Key: GOBBLIN-1060 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1060 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Lei Sun >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] codecov-io commented on issue #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher
codecov-io commented on issue #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher URL: https://github.com/apache/incubator-gobblin/pull/2899#issuecomment-591042125 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=h1) Report > Merging [#2899](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/0de109b6941e636f4a35069752ef1e3c4188054d?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #2899 +/- ## === - Coverage 4.12% 4.12% -0.01% Complexity 750 750 === Files 19321932 Lines 72789 72792 +3 Branches 80228022 === Hits 30023002 - Misses69469 69472 +3 Partials318 318 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...rg/apache/gobblin/yarn/GobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2899/diff?src=pr=tree#diff-Z29iYmxpbi15YXJuL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3lhcm4vR29iYmxpbllhcm5BcHBMYXVuY2hlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=footer). Last update [0de109b...0abc430](https://codecov.io/gh/apache/incubator-gobblin/pull/2899?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust opened a new pull request #2899: [GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher
autumnust opened a new pull request #2899: [GOBBLIN-1060]Fix wrong fileSystem
object in YarnApplauncher
URL: https://github.com/apache/incubator-gobblin/pull/2899
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1060
### Description
The existence checking of files to be localized in yarn appmaster /
container need to happen against local fs object. Currently it is using HDFS fs
object which is incorrect.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1060) Fix YarnAppLauncher resource existence checking with wrong fs object
[
https://issues.apache.org/jira/browse/GOBBLIN-1060?focusedWorklogId=392835=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392835
]
ASF GitHub Bot logged work on GOBBLIN-1060:
---
Author: ASF GitHub Bot
Created on: 25/Feb/20 19:00
Start Date: 25/Feb/20 19:00
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2899:
[GOBBLIN-1060]Fix wrong fileSystem object in YarnApplauncher
URL: https://github.com/apache/incubator-gobblin/pull/2899
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1060
### Description
The existence checking of files to be localized in yarn appmaster /
container need to happen against local fs object. Currently it is using HDFS fs
object which is incorrect.
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 392835)
Remaining Estimate: 0h
Time Spent: 10m
> Fix YarnAppLauncher resource existence checking with wrong fs object
>
>
> Key: GOBBLIN-1060
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1060
> Project: Apache Gobblin
> Issue Type: Bug
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1058) Make emitTrackingEvents method accept additional PartitionsToTags map for ease of extension on metrics
[ https://issues.apache.org/jira/browse/GOBBLIN-1058?focusedWorklogId=392832=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392832 ] ASF GitHub Bot logged work on GOBBLIN-1058: --- Author: ASF GitHub Bot Created on: 25/Feb/20 18:58 Start Date: 25/Feb/20 18:58 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags URL: https://github.com/apache/incubator-gobblin/pull/2898 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 392832) Time Spent: 50m (was: 40m) > Make emitTrackingEvents method accept additional PartitionsToTags map for > ease of extension on metrics > -- > > Key: GOBBLIN-1058 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1058 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1060) Fix YarnAppLauncher resource existence checking with wrong fs object
Lei Sun created GOBBLIN-1060: Summary: Fix YarnAppLauncher resource existence checking with wrong fs object Key: GOBBLIN-1060 URL: https://issues.apache.org/jira/browse/GOBBLIN-1060 Project: Apache Gobblin Issue Type: Bug Reporter: Lei Sun -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags
asfgit closed pull request #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags URL: https://github.com/apache/incubator-gobblin/pull/2898 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1058) Make emitTrackingEvents method accept additional PartitionsToTags map for ease of extension on metrics
[ https://issues.apache.org/jira/browse/GOBBLIN-1058?focusedWorklogId=392811=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-392811 ] ASF GitHub Bot logged work on GOBBLIN-1058: --- Author: ASF GitHub Bot Created on: 25/Feb/20 18:33 Start Date: 25/Feb/20 18:33 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags URL: https://github.com/apache/incubator-gobblin/pull/2898#issuecomment-590601056 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=h1) Report > Merging [#2898](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a580d6e4beb0ea53e7971caa489f1fe508159ec2?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `25%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2898 +/- ## + Coverage 45.84% 45.85% +0.01% - Complexity 9151 9168 +17 Files 1929 1932 +3 Lines 7264972789 +140 Branches 7999 8022 +23 + Hits 3330433379 +75 - Misses3630536351 +46 - Partials 3040 3059 +19 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...source/extractor/extract/kafka/KafkaExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `85.97% <28.57%> (+31.48%)` | `29 <0> (+4)` | :arrow_up: | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `73.43% <0%> (-6.25%)` | `15% <0%> (-1%)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-4.22%)` | `14% <0%> (-1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecConsumer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY0NvbnN1bWVyLmphdmE=) | `60% <0%> (-3.83%)` | `6% <0%> (+1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY1Byb2R1Y2VyLmphdmE=) | `72.54% <0%> (-3.07%)` | `9% <0%> (+1%)` | | | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `68.87% <0%> (-1.68%)` | `10% <0%> (ø)` | | |
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags
codecov-io edited a comment on issue #2898: [GOBBLIN-1058] Refactor method emitting GTE for ease of adding new tags URL: https://github.com/apache/incubator-gobblin/pull/2898#issuecomment-590601056 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=h1) Report > Merging [#2898](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/a580d6e4beb0ea53e7971caa489f1fe508159ec2?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `25%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2898 +/- ## + Coverage 45.84% 45.85% +0.01% - Complexity 9151 9168 +17 Files 1929 1932 +3 Lines 7264972789 +140 Branches 7999 8022 +23 + Hits 3330433379 +75 - Misses3630536351 +46 - Partials 3040 3059 +19 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2898?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...source/extractor/extract/kafka/KafkaExtractor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3Rvci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...in/source/extractor/extract/kafka/KafkaSource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYVNvdXJjZS5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ctor/extract/kafka/KafkaExtractorStatsTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9rYWZrYS9LYWZrYUV4dHJhY3RvclN0YXRzVHJhY2tlci5qYXZh) | `85.97% <28.57%> (+31.48%)` | `29 <0> (+4)` | :arrow_up: | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `73.43% <0%> (-6.25%)` | `15% <0%> (-1%)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-4.22%)` | `14% <0%> (-1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecConsumer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY0NvbnN1bWVyLmphdmE=) | `60% <0%> (-3.83%)` | `6% <0%> (+1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY1Byb2R1Y2VyLmphdmE=) | `72.54% <0%> (-3.07%)` | `9% <0%> (+1%)` | | | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `68.87% <0%> (-1.68%)` | `10% <0%> (ø)` | | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `63.33% <0%> (-1.12%)` | `15% <0%> (-1%)` | | | ... and [19 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2898/diff?src=pr=tree-more) | | --
[jira] [Updated] (GOBBLIN-1059) Failed to run job for MySQL to HDFS ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chhaya Vankhede updated GOBBLIN-1059: - Priority: Trivial (was: Critical) > Failed to run job for MySQL to HDFS ingestion > - > > Key: GOBBLIN-1059 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1059 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-sql > Environment: linux subsystem, hadoop 2.3.0, >Reporter: Chhaya Vankhede >Assignee: Shirshanka Das >Priority: Trivial > Labels: beginner > > I am tring to ingest data from mysql to hdfs but getting below error. The > job file and properties are as below > mysql.pull > > #Job properties > job.name=GobblinMySql > job.group=MySql > job.description=Data pull from MySql > #Extract properties > extract.namespace=demodb > extract.table.type=snapshot_only > extract.table.name=user > #Property to consider the extract as full dump > extract.is.full=true > #Source properties > source.querybased.schema=user > source.entity=user > source.querybased.extract.type=snapshot > ``` > mysql.propertes > ``` > #Source properties - source class to extract data from Mysql Source > source.class=org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > #Source properties > source.max.number.of.partitions=1 > source.querybased.partition.interval=1 > source.querybased.is.compression=true > source.querybased.watermark.type=timestamp > #Source connection properties > source.conn.driver=com.mysql.jdbc.Driver > source.conn.username=root > source.conn.password=root > source.conn.host=localhost > source.conn.port=3306 > source.conn.timeout=1500 > #Converter properties - Record from mysql source will be processed by the > below series of converters > > converter.classes=org.apache.gobblin.converter.avro.JsonIntermediateToAvroConverter > #date columns format > converter.avro.timestamp.format=-MM-dd HH:mm:ss'.0' > converter.avro.date.format=-MM-dd > converter.avro.time.format=HH:mm:ss > #Qualitychecker properties > > qualitychecker.task.policies=org.apache.gobblin.policies.count.RowCountPolicy,gobblin.policies.schema.SchemaCompatibilityPolicy > qualitychecker.task.policy.types=OPTIONAL,OPTIONAL > #Publisher properties > data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher > below is error log > ``` > GMT ERROR [JobScheduler-0] > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed > to run job GobblinMySql > org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql > at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) > at > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: Failed to create job launcher: > java.lang.ClassNotFoundException: > org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68) > at > org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:450) > at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:443) > ... 4 more > Caused by: java.lang.ClassNotFoundException: > org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:419) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:352) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:240) > at org.apache.gobblin.runtime.JobContext.(JobContext.java:172) > at > org.apache.gobblin.runtime.AbstractJobLauncher.(AbstractJobLauncher.java:201) > at > org.apache.gobblin.runtime.local.LocalJobLauncher.(LocalJobLauncher.java:86) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144) > ... 9 more > 2020-02-25 08:02:45 GMT ERROR [JobScheduler-0] > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed > to run job GobblinMySql >
[jira] [Updated] (GOBBLIN-1059) Failed to run job for MySQL to HDFS ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chhaya Vankhede updated GOBBLIN-1059: - Priority: Critical (was: Major) > Failed to run job for MySQL to HDFS ingestion > - > > Key: GOBBLIN-1059 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1059 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-sql > Environment: linux subsystem, hadoop 2.3.0, >Reporter: Chhaya Vankhede >Assignee: Shirshanka Das >Priority: Critical > Labels: beginner > > I am tring to ingest data from mysql to hdfs but getting below error. The > job file and properties are as below > mysql.pull > > #Job properties > job.name=GobblinMySql > job.group=MySql > job.description=Data pull from MySql > #Extract properties > extract.namespace=demodb > extract.table.type=snapshot_only > extract.table.name=user > #Property to consider the extract as full dump > extract.is.full=true > #Source properties > source.querybased.schema=user > source.entity=user > source.querybased.extract.type=snapshot > ``` > mysql.propertes > ``` > #Source properties - source class to extract data from Mysql Source > source.class=org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > #Source properties > source.max.number.of.partitions=1 > source.querybased.partition.interval=1 > source.querybased.is.compression=true > source.querybased.watermark.type=timestamp > #Source connection properties > source.conn.driver=com.mysql.jdbc.Driver > source.conn.username=root > source.conn.password=root > source.conn.host=localhost > source.conn.port=3306 > source.conn.timeout=1500 > #Converter properties - Record from mysql source will be processed by the > below series of converters > > converter.classes=org.apache.gobblin.converter.avro.JsonIntermediateToAvroConverter > #date columns format > converter.avro.timestamp.format=-MM-dd HH:mm:ss'.0' > converter.avro.date.format=-MM-dd > converter.avro.time.format=HH:mm:ss > #Qualitychecker properties > > qualitychecker.task.policies=org.apache.gobblin.policies.count.RowCountPolicy,gobblin.policies.schema.SchemaCompatibilityPolicy > qualitychecker.task.policy.types=OPTIONAL,OPTIONAL > #Publisher properties > data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher > below is error log > ``` > GMT ERROR [JobScheduler-0] > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed > to run job GobblinMySql > org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql > at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) > at > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: Failed to create job launcher: > java.lang.ClassNotFoundException: > org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68) > at > org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:450) > at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:443) > ... 4 more > Caused by: java.lang.ClassNotFoundException: > org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:419) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:352) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:240) > at org.apache.gobblin.runtime.JobContext.(JobContext.java:172) > at > org.apache.gobblin.runtime.AbstractJobLauncher.(AbstractJobLauncher.java:201) > at > org.apache.gobblin.runtime.local.LocalJobLauncher.(LocalJobLauncher.java:86) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144) > ... 9 more > 2020-02-25 08:02:45 GMT ERROR [JobScheduler-0] > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed > to run job GobblinMySql >
[jira] [Updated] (GOBBLIN-1059) Failed to run job for MySQL to HDFS ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chhaya Vankhede updated GOBBLIN-1059: - Summary: Failed to run job for MySQL to HDFS ingestion (was: MySQL to HDFS ingestion) > Failed to run job for MySQL to HDFS ingestion > - > > Key: GOBBLIN-1059 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1059 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-sql > Environment: linux subsystem, hadoop 2.3.0, >Reporter: Chhaya Vankhede >Assignee: Shirshanka Das >Priority: Major > Labels: beginner > > I am tring to ingest data from mysql to hdfs but getting below error. The > job file and properties are as below > mysql.pull > > #Job properties > job.name=GobblinMySql > job.group=MySql > job.description=Data pull from MySql > #Extract properties > extract.namespace=demodb > extract.table.type=snapshot_only > extract.table.name=user > #Property to consider the extract as full dump > extract.is.full=true > #Source properties > source.querybased.schema=user > source.entity=user > source.querybased.extract.type=snapshot > ``` > mysql.propertes > ``` > #Source properties - source class to extract data from Mysql Source > source.class=org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > #Source properties > source.max.number.of.partitions=1 > source.querybased.partition.interval=1 > source.querybased.is.compression=true > source.querybased.watermark.type=timestamp > #Source connection properties > source.conn.driver=com.mysql.jdbc.Driver > source.conn.username=root > source.conn.password=root > source.conn.host=localhost > source.conn.port=3306 > source.conn.timeout=1500 > #Converter properties - Record from mysql source will be processed by the > below series of converters > > converter.classes=org.apache.gobblin.converter.avro.JsonIntermediateToAvroConverter > #date columns format > converter.avro.timestamp.format=-MM-dd HH:mm:ss'.0' > converter.avro.date.format=-MM-dd > converter.avro.time.format=HH:mm:ss > #Qualitychecker properties > > qualitychecker.task.policies=org.apache.gobblin.policies.count.RowCountPolicy,gobblin.policies.schema.SchemaCompatibilityPolicy > qualitychecker.task.policy.types=OPTIONAL,OPTIONAL > #Publisher properties > data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher > below is error log > ``` > GMT ERROR [JobScheduler-0] > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed > to run job GobblinMySql > org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql > at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) > at > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: Failed to create job launcher: > java.lang.ClassNotFoundException: > org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68) > at > org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:450) > at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:443) > ... 4 more > Caused by: java.lang.ClassNotFoundException: > org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:419) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:352) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:240) > at org.apache.gobblin.runtime.JobContext.(JobContext.java:172) > at > org.apache.gobblin.runtime.AbstractJobLauncher.(AbstractJobLauncher.java:201) > at > org.apache.gobblin.runtime.local.LocalJobLauncher.(LocalJobLauncher.java:86) > at > org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144) > ... 9 more > 2020-02-25 08:02:45 GMT ERROR [JobScheduler-0] > org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637
[jira] [Updated] (GOBBLIN-1059) MySQL to HDFS ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chhaya Vankhede updated GOBBLIN-1059: - Docs Text: (was: I am tring to ingest data from mysql to hdfs but getting below error. The job file and properties are as below mysql.pull ``` # Job properties job.name=GobblinMySql job.group=MySql job.description=Data pull from MySql # Extract properties extract.namespace=demodb extract.table.type=snapshot_only extract.table.name=user # Property to consider the extract as full dump extract.is.full=true # Source properties source.querybased.schema=user source.entity=user source.querybased.extract.type=snapshot ``` mysql.propertes ``` # Source properties - source class to extract data from Mysql Source source.class=org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource # Source properties source.max.number.of.partitions=1 source.querybased.partition.interval=1 source.querybased.is.compression=true source.querybased.watermark.type=timestamp # Source connection properties source.conn.driver=com.mysql.jdbc.Driver source.conn.username=root source.conn.password=root source.conn.host=localhost source.conn.port=3306 source.conn.timeout=1500 # Converter properties - Record from mysql source will be processed by the below series of converters converter.classes=org.apache.gobblin.converter.avro.JsonIntermediateToAvroConverter # date columns format converter.avro.timestamp.format=-MM-dd HH:mm:ss'.0' converter.avro.date.format=-MM-dd converter.avro.time.format=HH:mm:ss # Qualitychecker properties qualitychecker.task.policies=org.apache.gobblin.policies.count.RowCountPolicy,gobblin.policies.schema.SchemaCompatibilityPolicy qualitychecker.task.policy.types=OPTIONAL,OPTIONAL # Publisher properties data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher ``` below is error log ``` GMT ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed to run job GobblinMySql org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Failed to create job launcher: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68) at org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:450) at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:443) ... 4 more Caused by: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:240) at org.apache.gobblin.runtime.JobContext.(JobContext.java:172) at org.apache.gobblin.runtime.AbstractJobLauncher.(AbstractJobLauncher.java:201) at org.apache.gobblin.runtime.local.LocalJobLauncher.(LocalJobLauncher.java:86) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144) ... 9 more 2020-02-25 08:02:45 GMT ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed to run job GobblinMySql org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Failed to create job launcher:
[jira] [Updated] (GOBBLIN-1059) MySQL to HDFS ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chhaya Vankhede updated GOBBLIN-1059: - Description: (was: using command `bin/gobblin.sh service standalone start --jars /mnt/C/Users/name/incubator-gobblin/build/gobblin-sql/libs/gobblin-sql-0.15.0.jar` How to solve it?) > MySQL to HDFS ingestion > --- > > Key: GOBBLIN-1059 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1059 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-sql > Environment: linux subsystem, hadoop 2.3.0, >Reporter: Chhaya Vankhede >Assignee: Shirshanka Das >Priority: Blocker > Labels: beginner > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (GOBBLIN-1059) MySQL to HDFS ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chhaya Vankhede updated GOBBLIN-1059: - Priority: Major (was: Blocker) > MySQL to HDFS ingestion > --- > > Key: GOBBLIN-1059 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1059 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-sql > Environment: linux subsystem, hadoop 2.3.0, >Reporter: Chhaya Vankhede >Assignee: Shirshanka Das >Priority: Major > Labels: beginner > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (GOBBLIN-1059) MySQL to HDFS ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chhaya Vankhede updated GOBBLIN-1059: - Description: using command `bin/gobblin.sh service standalone start --jars /mnt/C/Users/name/incubator-gobblin/build/gobblin-sql/libs/gobblin-sql-0.15.0.jar` How to solve it? > MySQL to HDFS ingestion > --- > > Key: GOBBLIN-1059 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1059 > Project: Apache Gobblin > Issue Type: Bug > Components: gobblin-sql > Environment: linux subsystem, hadoop 2.3.0, >Reporter: Chhaya Vankhede >Assignee: Shirshanka Das >Priority: Blocker > Labels: beginner > > using command > `bin/gobblin.sh service standalone start --jars > /mnt/C/Users/name/incubator-gobblin/build/gobblin-sql/libs/gobblin-sql-0.15.0.jar` > How to solve it? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (GOBBLIN-1059) MySQL to HDFS ingestion
[ https://issues.apache.org/jira/browse/GOBBLIN-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chhaya Vankhede updated GOBBLIN-1059: - Docs Text: I am tring to ingest data from mysql to hdfs but getting below error. The job file and properties are as below mysql.pull ``` # Job properties job.name=GobblinMySql job.group=MySql job.description=Data pull from MySql # Extract properties extract.namespace=demodb extract.table.type=snapshot_only extract.table.name=user # Property to consider the extract as full dump extract.is.full=true # Source properties source.querybased.schema=user source.entity=user source.querybased.extract.type=snapshot ``` mysql.propertes ``` # Source properties - source class to extract data from Mysql Source source.class=org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource # Source properties source.max.number.of.partitions=1 source.querybased.partition.interval=1 source.querybased.is.compression=true source.querybased.watermark.type=timestamp # Source connection properties source.conn.driver=com.mysql.jdbc.Driver source.conn.username=root source.conn.password=root source.conn.host=localhost source.conn.port=3306 source.conn.timeout=1500 # Converter properties - Record from mysql source will be processed by the below series of converters converter.classes=org.apache.gobblin.converter.avro.JsonIntermediateToAvroConverter # date columns format converter.avro.timestamp.format=-MM-dd HH:mm:ss'.0' converter.avro.date.format=-MM-dd converter.avro.time.format=HH:mm:ss # Qualitychecker properties qualitychecker.task.policies=org.apache.gobblin.policies.count.RowCountPolicy,gobblin.policies.schema.SchemaCompatibilityPolicy qualitychecker.task.policy.types=OPTIONAL,OPTIONAL # Publisher properties data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher ``` below is error log ``` GMT ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed to run job GobblinMySql org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Failed to create job launcher: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68) at org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:450) at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:443) ... 4 more Caused by: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:240) at org.apache.gobblin.runtime.JobContext.(JobContext.java:172) at org.apache.gobblin.runtime.AbstractJobLauncher.(AbstractJobLauncher.java:201) at org.apache.gobblin.runtime.local.LocalJobLauncher.(LocalJobLauncher.java:86) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144) ... 9 more 2020-02-25 08:02:45 GMT ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed to run job GobblinMySql org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Failed to create job launcher:
[jira] [Created] (GOBBLIN-1059) MySQL to HDFS ingestion
Chhaya Vankhede created GOBBLIN-1059: Summary: MySQL to HDFS ingestion Key: GOBBLIN-1059 URL: https://issues.apache.org/jira/browse/GOBBLIN-1059 Project: Apache Gobblin Issue Type: Bug Components: gobblin-sql Environment: linux subsystem, hadoop 2.3.0, Reporter: Chhaya Vankhede Assignee: Shirshanka Das I am tring to ingest data from mysql to hdfs but getting below error. The job file and properties are as below mysql.pull ``` # Job properties job.name=GobblinMySql job.group=MySql job.description=Data pull from MySql # Extract properties extract.namespace=demodb extract.table.type=snapshot_only extract.table.name=user # Property to consider the extract as full dump extract.is.full=true # Source properties source.querybased.schema=user source.entity=user source.querybased.extract.type=snapshot ``` mysql.propertes ``` # Source properties - source class to extract data from Mysql Source source.class=org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource # Source properties source.max.number.of.partitions=1 source.querybased.partition.interval=1 source.querybased.is.compression=true source.querybased.watermark.type=timestamp # Source connection properties source.conn.driver=com.mysql.jdbc.Driver source.conn.username=root source.conn.password=root source.conn.host=localhost source.conn.port=3306 source.conn.timeout=1500 # Converter properties - Record from mysql source will be processed by the below series of converters converter.classes=org.apache.gobblin.converter.avro.JsonIntermediateToAvroConverter # date columns format converter.avro.timestamp.format=-MM-dd HH:mm:ss'.0' converter.avro.date.format=-MM-dd converter.avro.time.format=HH:mm:ss # Qualitychecker properties qualitychecker.task.policies=org.apache.gobblin.policies.count.RowCountPolicy,gobblin.policies.schema.SchemaCompatibilityPolicy qualitychecker.task.policy.types=OPTIONAL,OPTIONAL # Publisher properties data.publisher.type=org.apache.gobblin.publisher.BaseDataPublisher ``` below is error log ``` GMT ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed to run job GobblinMySql org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: Failed to create job launcher: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68) at org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:450) at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:443) ... 4 more Caused by: java.lang.ClassNotFoundException: org.apache.gobblin.source.extractor.extract.jdbc.MysqlSource at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.gobblin.runtime.JobContext.createSource(JobContext.java:240) at org.apache.gobblin.runtime.JobContext.(JobContext.java:172) at org.apache.gobblin.runtime.AbstractJobLauncher.(AbstractJobLauncher.java:201) at org.apache.gobblin.runtime.local.LocalJobLauncher.(LocalJobLauncher.java:86) at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144) ... 9 more 2020-02-25 08:02:45 GMT ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 637 - Failed to run job GobblinMySql org.apache.gobblin.runtime.JobException: Failed to run job GobblinMySql at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:445) at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:635) at
