[jira] [Work logged] (GOBBLIN-1132) move logic of requester list validation to RequesterService implementation
[ https://issues.apache.org/jira/browse/GOBBLIN-1132?focusedWorklogId=427927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-427927 ] ASF GitHub Bot logged work on GOBBLIN-1132: --- Author: ASF GitHub Bot Created on: 28/Apr/20 02:01 Start Date: 28/Apr/20 02:01 Worklog Time Spent: 10m Work Description: codecov-io edited a comment on pull request #2969: URL: https://github.com/apache/incubator-gobblin/pull/2969#issuecomment-620168499 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=h1) Report > Merging [#2969](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/d8d579a42ec901dd74b6f453bd334c77e9498195=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `17.85%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2969 +/- ## - Coverage 45.64% 45.63% -0.02% - Complexity 9197 9198 +1 Files 1940 1940 Lines 7357473581 +7 Branches 8128 8128 - Hits 3358633577 -9 - Misses3686436880 +16 Partials 3124 3124 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ce/extractor/extract/restapi/RestApiConnector.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9yZXN0YXBpL1Jlc3RBcGlDb25uZWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...rg/apache/gobblin/service/FlowConfigsResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1Jlc291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [.../apache/gobblin/service/FlowConfigsV2Resource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1YyUmVzb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...g/apache/gobblin/service/NoopRequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9Ob29wUmVxdWVzdGVyU2VydmljZS5qYXZh) | `50.00% <0.00%> (-16.67%)` | `1.00 <0.00> (ø)` | | | [...apache/gobblin/salesforce/SalesforceConnector.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbm5lY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...a/org/apache/gobblin/service/RequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9SZXF1ZXN0ZXJTZXJ2aWNlLmphdmE=) | `100.00% <100.00%> (ø)` | `4.00 <1.00> (ø)` | | | [...e/gobblin/runtime/app/ServiceBasedAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBwL1NlcnZpY2VCYXNlZEFwcExhdW5jaGVyLmphdmE=) | `43.68% <0.00%> (-5.83%)` | `12.00% <0.00%> (ø%)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `34.92% <0.00%> (-5.56%)` | `14.00% <0.00%> (-2.00%)` | | |
[GitHub] [incubator-gobblin] codecov-io edited a comment on pull request #2969: [GOBBLIN-1132] move the logic of requester list verification to RequesterService implementation
codecov-io edited a comment on pull request #2969: URL: https://github.com/apache/incubator-gobblin/pull/2969#issuecomment-620168499 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=h1) Report > Merging [#2969](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/d8d579a42ec901dd74b6f453bd334c77e9498195=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `17.85%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2969 +/- ## - Coverage 45.64% 45.63% -0.02% - Complexity 9197 9198 +1 Files 1940 1940 Lines 7357473581 +7 Branches 8128 8128 - Hits 3358633577 -9 - Misses3686436880 +16 Partials 3124 3124 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ce/extractor/extract/restapi/RestApiConnector.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9yZXN0YXBpL1Jlc3RBcGlDb25uZWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...rg/apache/gobblin/service/FlowConfigsResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1Jlc291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [.../apache/gobblin/service/FlowConfigsV2Resource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1YyUmVzb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...g/apache/gobblin/service/NoopRequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9Ob29wUmVxdWVzdGVyU2VydmljZS5qYXZh) | `50.00% <0.00%> (-16.67%)` | `1.00 <0.00> (ø)` | | | [...apache/gobblin/salesforce/SalesforceConnector.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbm5lY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...a/org/apache/gobblin/service/RequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9SZXF1ZXN0ZXJTZXJ2aWNlLmphdmE=) | `100.00% <100.00%> (ø)` | `4.00 <1.00> (ø)` | | | [...e/gobblin/runtime/app/ServiceBasedAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBwL1NlcnZpY2VCYXNlZEFwcExhdW5jaGVyLmphdmE=) | `43.68% <0.00%> (-5.83%)` | `12.00% <0.00%> (ø%)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `34.92% <0.00%> (-5.56%)` | `14.00% <0.00%> (-2.00%)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0.00%> (-0.94%)` | `24.00% <0.00%> (ø%)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `67.53% <0.00%> (-0.38%)` | `33.00% <0.00%> (ø%)` | | | ... and [3
[jira] [Created] (GOBBLIN-1132) move logic of requester list validation to RequesterService implementation
Arjun Singh Bora created GOBBLIN-1132: - Summary: move logic of requester list validation to RequesterService implementation Key: GOBBLIN-1132 URL: https://issues.apache.org/jira/browse/GOBBLIN-1132 Project: Apache Gobblin Issue Type: Task Reporter: Arjun Singh Bora -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1131) Bump up ORC version to pick up ORC-616
[
https://issues.apache.org/jira/browse/GOBBLIN-1131?focusedWorklogId=427846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-427846
]
ASF GitHub Bot logged work on GOBBLIN-1131:
---
Author: ASF GitHub Bot
Created on: 27/Apr/20 20:34
Start Date: 27/Apr/20 20:34
Worklog Time Spent: 10m
Work Description: autumnust opened a new pull request #2972:
URL: https://github.com/apache/incubator-gobblin/pull/2972
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin JIRA]
https://issues.apache.org/jira/browse/GOBBLIN-1131
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 427846)
Remaining Estimate: 0h
Time Spent: 10m
> Bump up ORC version to pick up ORC-616
> --
>
> Key: GOBBLIN-1131
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1131
> Project: Apache Gobblin
> Issue Type: Bug
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (GOBBLIN-1131) Bump up ORC version to pick up ORC-616
Lei Sun created GOBBLIN-1131: Summary: Bump up ORC version to pick up ORC-616 Key: GOBBLIN-1131 URL: https://issues.apache.org/jira/browse/GOBBLIN-1131 Project: Apache Gobblin Issue Type: Bug Reporter: Lei Sun -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust opened a new pull request #2972: [GOBBLIN-1131] Bump up ORC deps to 1.6.3 to pick ORC-616
autumnust opened a new pull request #2972:
URL: https://github.com/apache/incubator-gobblin/pull/2972
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin JIRA]
https://issues.apache.org/jira/browse/GOBBLIN-1131
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [incubator-gobblin] arjun4084346 commented on pull request #2971: [GOBBLIN-1130] Add API for adding proxy user to azkaban project
arjun4084346 commented on pull request #2971: URL: https://github.com/apache/incubator-gobblin/pull/2971#issuecomment-620217933 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1130) Add API for adding proxy user to azkaban project
[
https://issues.apache.org/jira/browse/GOBBLIN-1130?focusedWorklogId=427843=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-427843
]
ASF GitHub Bot logged work on GOBBLIN-1130:
---
Author: ASF GitHub Bot
Created on: 27/Apr/20 20:30
Start Date: 27/Apr/20 20:30
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#2971:
URL: https://github.com/apache/incubator-gobblin/pull/2971#discussion_r416126824
##
File path:
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/service/modules/orchestration/AzkabanMultiCallables.java
##
@@ -375,4 +375,44 @@ public AzkabanClientStatus call()
}
}
}
+
+ /**
+ * A callable that will add a proxy user to a project on Azkaban
+ */
+ @Builder
+ static class addProxyUserCallable implements Callable {
+private AzkabanClient client;
+private String projectName;
+private String proxyUserName;
+private boolean invalidSession = false;
+
+@Override
+public AzkabanClientStatus call()
+throws AzkabanClientException {
+ try (Closer closer = Closer.create()) {
+client.refreshSession(this.invalidSession);
+List nvps = new ArrayList<>();
+nvps.add(new BasicNameValuePair(AzkabanClientParams.AJAX,
"addProxyUser"));
+nvps.add(new BasicNameValuePair(AzkabanClientParams.SESSION_ID,
client.sessionId));
+nvps.add(new BasicNameValuePair(AzkabanClientParams.PROJECT,
projectName));
+nvps.add(new BasicNameValuePair(AzkabanClientParams.NAME,
proxyUserName));
+
+Header contentType = new BasicHeader(HttpHeaders.CONTENT_TYPE,
"application/x-www-form-urlencoded");
+Header requestType = new BasicHeader("X-Requested-With",
"XMLHttpRequest");
+
+HttpGet httpGet = new HttpGet(client.url + "/manager?" +
URLEncodedUtils.format(nvps, "UTF-8"));
+httpGet.setHeaders(new Header[]{contentType, requestType});
+
+CloseableHttpResponse response = client.httpClient.execute(httpGet);
+closer.register(response);
+client.handleResponse(response);
Review comment:
this can be called as AzkabanClient.handleResponse()
intellij shows codestyle issue
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 427843)
Time Spent: 0.5h (was: 20m)
> Add API for adding proxy user to azkaban project
>
>
> Key: GOBBLIN-1130
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1130
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #2971: [GOBBLIN-1130] Add API for adding proxy user to azkaban project
arjun4084346 commented on a change in pull request #2971:
URL: https://github.com/apache/incubator-gobblin/pull/2971#discussion_r416126824
##
File path:
gobblin-modules/gobblin-azkaban/src/main/java/org/apache/gobblin/service/modules/orchestration/AzkabanMultiCallables.java
##
@@ -375,4 +375,44 @@ public AzkabanClientStatus call()
}
}
}
+
+ /**
+ * A callable that will add a proxy user to a project on Azkaban
+ */
+ @Builder
+ static class addProxyUserCallable implements Callable {
+private AzkabanClient client;
+private String projectName;
+private String proxyUserName;
+private boolean invalidSession = false;
+
+@Override
+public AzkabanClientStatus call()
+throws AzkabanClientException {
+ try (Closer closer = Closer.create()) {
+client.refreshSession(this.invalidSession);
+List nvps = new ArrayList<>();
+nvps.add(new BasicNameValuePair(AzkabanClientParams.AJAX,
"addProxyUser"));
+nvps.add(new BasicNameValuePair(AzkabanClientParams.SESSION_ID,
client.sessionId));
+nvps.add(new BasicNameValuePair(AzkabanClientParams.PROJECT,
projectName));
+nvps.add(new BasicNameValuePair(AzkabanClientParams.NAME,
proxyUserName));
+
+Header contentType = new BasicHeader(HttpHeaders.CONTENT_TYPE,
"application/x-www-form-urlencoded");
+Header requestType = new BasicHeader("X-Requested-With",
"XMLHttpRequest");
+
+HttpGet httpGet = new HttpGet(client.url + "/manager?" +
URLEncodedUtils.format(nvps, "UTF-8"));
+httpGet.setHeaders(new Header[]{contentType, requestType});
+
+CloseableHttpResponse response = client.httpClient.execute(httpGet);
+closer.register(response);
+client.handleResponse(response);
Review comment:
this can be called as AzkabanClient.handleResponse()
intellij shows codestyle issue
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1130) Add API for adding proxy user to azkaban project
[ https://issues.apache.org/jira/browse/GOBBLIN-1130?focusedWorklogId=427837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-427837 ] ASF GitHub Bot logged work on GOBBLIN-1130: --- Author: ASF GitHub Bot Created on: 27/Apr/20 20:19 Start Date: 27/Apr/20 20:19 Worklog Time Spent: 10m Work Description: jack-moseley commented on pull request #2971: URL: https://github.com/apache/incubator-gobblin/pull/2971#issuecomment-620211198 @arjun4084346 please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 427837) Time Spent: 20m (was: 10m) > Add API for adding proxy user to azkaban project > > > Key: GOBBLIN-1130 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1130 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] jack-moseley opened a new pull request #2971: [GOBBLIN-1130] Add API for adding proxy user to azkaban project
jack-moseley opened a new pull request #2971:
URL: https://github.com/apache/incubator-gobblin/pull/2971
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1130
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Add method in `AzkabanClient` that adds given user as a proxy user in given
azkaban project
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Work logged] (GOBBLIN-1130) Add API for adding proxy user to azkaban project
[
https://issues.apache.org/jira/browse/GOBBLIN-1130?focusedWorklogId=427836=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-427836
]
ASF GitHub Bot logged work on GOBBLIN-1130:
---
Author: ASF GitHub Bot
Created on: 27/Apr/20 20:18
Start Date: 27/Apr/20 20:18
Worklog Time Spent: 10m
Work Description: jack-moseley opened a new pull request #2971:
URL: https://github.com/apache/incubator-gobblin/pull/2971
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1130
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Add method in `AzkabanClient` that adds given user as a proxy user in given
azkaban project
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 427836)
Remaining Estimate: 0h
Time Spent: 10m
> Add API for adding proxy user to azkaban project
>
>
> Key: GOBBLIN-1130
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1130
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] jack-moseley commented on pull request #2971: [GOBBLIN-1130] Add API for adding proxy user to azkaban project
jack-moseley commented on pull request #2971: URL: https://github.com/apache/incubator-gobblin/pull/2971#issuecomment-620211198 @arjun4084346 please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (GOBBLIN-1130) Add API for adding proxy user to azkaban project
Jack Moseley created GOBBLIN-1130: - Summary: Add API for adding proxy user to azkaban project Key: GOBBLIN-1130 URL: https://issues.apache.org/jira/browse/GOBBLIN-1130 Project: Apache Gobblin Issue Type: Improvement Reporter: Jack Moseley -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1129) clean up staging table created by avro2orc pipeline
[
https://issues.apache.org/jira/browse/GOBBLIN-1129?focusedWorklogId=427793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-427793
]
ASF GitHub Bot logged work on GOBBLIN-1129:
---
Author: ASF GitHub Bot
Created on: 27/Apr/20 19:06
Start Date: 27/Apr/20 19:06
Worklog Time Spent: 10m
Work Description: yukuai518 opened a new pull request #2970:
URL: https://github.com/apache/incubator-gobblin/pull/2970
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1129
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
During the avro2orc conversion, many staging tables are created, they are
not cleaned up due to the pipeline failure, and next execution doesn't take
care of the clean up, which caused the staging table taking many spaces in hive
metastore.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 427793)
Remaining Estimate: 0h
Time Spent: 10m
> clean up staging table created by avro2orc pipeline
> ---
>
> Key: GOBBLIN-1129
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1129
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Kuai Yu
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> During the avro2orc conversion, many staging tables are created, they are not
> cleaned up due to the pipeline failure, and next execution doesn't take care
> of the clean up, which caused the staging table taking many spaces in hive
> metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] yukuai518 opened a new pull request #2970: [GOBBLIN-1129] Add staging table cleanup logic for avro2orc pipeline.
yukuai518 opened a new pull request #2970:
URL: https://github.com/apache/incubator-gobblin/pull/2970
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1129
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
During the avro2orc conversion, many staging tables are created, they are
not cleaned up due to the pipeline failure, and next execution doesn't take
care of the clean up, which caused the staging table taking many spaces in hive
metastore.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (GOBBLIN-1129) clean up staging table created by avro2orc pipeline
Kuai Yu created GOBBLIN-1129: Summary: clean up staging table created by avro2orc pipeline Key: GOBBLIN-1129 URL: https://issues.apache.org/jira/browse/GOBBLIN-1129 Project: Apache Gobblin Issue Type: Improvement Reporter: Kuai Yu During the avro2orc conversion, many staging tables are created, they are not cleaned up due to the pipeline failure, and next execution doesn't take care of the clean up, which caused the staging table taking many spaces in hive metastore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] codecov-io commented on pull request #2969: change the logic of requester list verification
codecov-io commented on pull request #2969: URL: https://github.com/apache/incubator-gobblin/pull/2969#issuecomment-620168499 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=h1) Report > Merging [#2969](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/d8d579a42ec901dd74b6f453bd334c77e9498195=desc) will **decrease** coverage by `0.94%`. > The diff coverage is `35.29%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#2969 +/- ## - Coverage 45.64% 44.70% -0.95% + Complexity 9197 9026 -171 Files 1940 1940 Lines 7357473573 -1 Branches 8128 8128 - Hits 3358632888 -698 - Misses3686437601 +737 + Partials 3124 3084 -40 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2969?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...ce/extractor/extract/restapi/RestApiConnector.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NvdXJjZS9leHRyYWN0b3IvZXh0cmFjdC9yZXN0YXBpL1Jlc3RBcGlDb25uZWN0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...rg/apache/gobblin/service/FlowConfigsResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1Jlc291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [.../apache/gobblin/service/FlowConfigsV2Resource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnc1YyUmVzb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...apache/gobblin/salesforce/SalesforceConnector.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1zYWxlc2ZvcmNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NhbGVzZm9yY2UvU2FsZXNmb3JjZUNvbm5lY3Rvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...a/org/apache/gobblin/service/RequesterService.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9SZXF1ZXN0ZXJTZXJ2aWNlLmphdmE=) | `100.00% <100.00%> (ø)` | `4.00 <1.00> (ø)` | | | [...gobblin/runtime/mapreduce/GobblinOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL0dvYmJsaW5PdXRwdXRGb3JtYXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...askStateCollectorServiceHiveRegHandlerFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlQ29sbGVjdG9yU2VydmljZUhpdmVSZWdIYW5kbGVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | | | [...re/filesystem/FsDatasetStateStoreEntryManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWV0YXN0b3JlL2ZpbGVzeXN0ZW0vRnNEYXRhc2V0U3RhdGVTdG9yZUVudHJ5TWFuYWdlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | | | [...in/runtime/mapreduce/CustomizedProgresserBase.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbWFwcmVkdWNlL0N1c3RvbWl6ZWRQcm9ncmVzc2VyQmFzZS5qYXZh) | `0.00% <0.00%> (-83.34%)` | `0.00% <0.00%> (-1.00%)` | | | [...rg/apache/gobblin/runtime/ZkDatasetStateStore.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2969/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4taGVsaXgvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vcnVudGltZS9aa0RhdGFzZXRTdGF0ZVN0b3JlLmphdmE=) | `0.00% <0.00%> (-80.77%)` | `0.00% <0.00%> (-7.00%)` | |
[GitHub] [incubator-gobblin] arjun4084346 opened a new pull request #2969: change the logic of requester list verification
arjun4084346 opened a new pull request #2969:
URL: https://github.com/apache/incubator-gobblin/pull/2969
codestyle changes
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-XXX
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
