[jira] [Work logged] (GOBBLIN-790) Mysql Implementation of DagStateStore
[
https://issues.apache.org/jira/browse/GOBBLIN-790?focusedWorklogId=251510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251510
]
ASF GitHub Bot logged work on GOBBLIN-790:
--
Author: ASF GitHub Bot
Created on: 31/May/19 05:54
Start Date: 31/May/19 05:54
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2656:
[GOBBLIN-790]DagStateStore MySQL
URL: https://github.com/apache/incubator-gobblin/pull/2656
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin JIRA]
- https://issues.apache.org/jira/browse/GOBBLIN-790
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
This is the implementation of `DagStateStore` backed by MySQL.
Instead of implementing its own MySQLl-related logic, it instantiate a
`MySQLStateStore` inside the DagStateStore and delegate all interactions
between MySQL to that. To archive this, also added:
- Conversion from `Dag` to `State` by having `Dag` serialized first and put
it as a value in State's k-v pair.
- `getAll` method in `MysqlStateStore` where it used to strictly require
`storeName` to show up.
- Model the schema of `MysqlStateStore` in the `Dag'gy way: `FlowId` <-->
`storeName`, `FlowExecutionId` <--> `tableName`.
- Refactor some of test methods to reduce code redundancy.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- Relevant unit tests are added.
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251510)
Time Spent: 10m
Remaining Estimate: 0h
> Mysql Implementation of DagStateStore
> -
>
> Key: GOBBLIN-790
> URL: https://issues.apache.org/jira/browse/GOBBLIN-790
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [incubator-gobblin] autumnust commented on issue #2656: [GOBBLIN-790]DagStateStore MySQL
autumnust commented on issue #2656: [GOBBLIN-790]DagStateStore MySQL URL: https://github.com/apache/incubator-gobblin/pull/2656#issuecomment-497585428 @htran1 @sv2000 Please help take a look, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-790) Mysql Implementation of DagStateStore
[ https://issues.apache.org/jira/browse/GOBBLIN-790?focusedWorklogId=251511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251511 ] ASF GitHub Bot logged work on GOBBLIN-790: -- Author: ASF GitHub Bot Created on: 31/May/19 05:54 Start Date: 31/May/19 05:54 Worklog Time Spent: 10m Work Description: autumnust commented on issue #2656: [GOBBLIN-790]DagStateStore MySQL URL: https://github.com/apache/incubator-gobblin/pull/2656#issuecomment-497585428 @htran1 @sv2000 Please help take a look, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251511) Time Spent: 20m (was: 10m) > Mysql Implementation of DagStateStore > - > > Key: GOBBLIN-790 > URL: https://issues.apache.org/jira/browse/GOBBLIN-790 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [incubator-gobblin] autumnust opened a new pull request #2656: [GOBBLIN-790]DagStateStore MySQL
autumnust opened a new pull request #2656: [GOBBLIN-790]DagStateStore MySQL
URL: https://github.com/apache/incubator-gobblin/pull/2656
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin JIRA]
- https://issues.apache.org/jira/browse/GOBBLIN-790
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
This is the implementation of `DagStateStore` backed by MySQL.
Instead of implementing its own MySQLl-related logic, it instantiate a
`MySQLStateStore` inside the DagStateStore and delegate all interactions
between MySQL to that. To archive this, also added:
- Conversion from `Dag` to `State` by having `Dag` serialized first and put
it as a value in State's k-v pair.
- `getAll` method in `MysqlStateStore` where it used to strictly require
`storeName` to show up.
- Model the schema of `MysqlStateStore` in the `Dag'gy way: `FlowId` <-->
`storeName`, `FlowExecutionId` <--> `tableName`.
- Refactor some of test methods to reduce code redundancy.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
- Relevant unit tests are added.
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-790) Mysql Implementation of DagStateStore
Lei Sun created GOBBLIN-790: --- Summary: Mysql Implementation of DagStateStore Key: GOBBLIN-790 URL: https://issues.apache.org/jira/browse/GOBBLIN-790 Project: Apache Gobblin Issue Type: Improvement Reporter: Lei Sun -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (GOBBLIN-781) Clean Flow execution state when DR is enabled: Skeleton
[
https://issues.apache.org/jira/browse/GOBBLIN-781?focusedWorklogId=251390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251390
]
ASF GitHub Bot logged work on GOBBLIN-781:
--
Author: ASF GitHub Bot
Created on: 30/May/19 23:01
Start Date: 30/May/19 23:01
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2647: [GOBBLIN-781]
Skeleton for GaaS DR mode clean transition
URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289204291
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java
##
@@ -170,6 +195,19 @@ private void scheduleSpecsFromCatalog() {
}
}
+ /**
+ * In DR-mode, the running {@link FlowSpec} will all be cancelled and
rescheduled.
+ * We will need to make sure that running {@link FlowSpec}s' state are
cleared, and corresponding running jobs are
+ * killed before rescheduling them.
+ * @param drUris The uris that applicable for DR discovered from FlowCatalog.
+ */
+ private void clearRunningFlowState(Iterator drUris) {
Review comment:
Should the DR related failover handling be moved to a separate class? For
now, you are triggering it based on config. In the future, we might have a REST
end-point or some other mechanism for triggering a fail over, without needing a
service restart.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251390)
Time Spent: 1h (was: 50m)
> Clean Flow execution state when DR is enabled: Skeleton
> ---
>
> Key: GOBBLIN-781
> URL: https://issues.apache.org/jira/browse/GOBBLIN-781
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (GOBBLIN-781) Clean Flow execution state when DR is enabled: Skeleton
[ https://issues.apache.org/jira/browse/GOBBLIN-781?focusedWorklogId=251386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251386 ] ASF GitHub Bot logged work on GOBBLIN-781: -- Author: ASF GitHub Bot Created on: 30/May/19 23:01 Start Date: 30/May/19 23:01 Worklog Time Spent: 10m Work Description: sv2000 commented on pull request #2647: [GOBBLIN-781] Skeleton for GaaS DR mode clean transition URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289199437 ## File path: gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/SpecStore.java ## @@ -113,6 +113,8 @@ */ Iterator getSpecURIs() throws IOException; + Iterator getSpecURIsWithTag(String tag) throws IOException; Review comment: Javadoc here will be useful. Perhaps an example too? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251386) Time Spent: 40m (was: 0.5h) > Clean Flow execution state when DR is enabled: Skeleton > --- > > Key: GOBBLIN-781 > URL: https://issues.apache.org/jira/browse/GOBBLIN-781 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (GOBBLIN-781) Clean Flow execution state when DR is enabled: Skeleton
[ https://issues.apache.org/jira/browse/GOBBLIN-781?focusedWorklogId=251385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251385 ] ASF GitHub Bot logged work on GOBBLIN-781: -- Author: ASF GitHub Bot Created on: 30/May/19 23:01 Start Date: 30/May/19 23:01 Worklog Time Spent: 10m Work Description: sv2000 commented on pull request #2647: [GOBBLIN-781] Skeleton for GaaS DR mode clean transition URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289198551 ## File path: gobblin-restli/gobblin-throttling-service/gobblin-throttling-service-api/src/main/snapshot/org.apache.gobblin.restli.throttling.permits.snapshot.json ## @@ -17,6 +17,18 @@ "type" : "long", Review comment: I am assuming this file was not intended to be in the commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251385) Time Spent: 0.5h (was: 20m) > Clean Flow execution state when DR is enabled: Skeleton > --- > > Key: GOBBLIN-781 > URL: https://issues.apache.org/jira/browse/GOBBLIN-781 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (GOBBLIN-781) Clean Flow execution state when DR is enabled: Skeleton
[
https://issues.apache.org/jira/browse/GOBBLIN-781?focusedWorklogId=251389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251389
]
ASF GitHub Bot logged work on GOBBLIN-781:
--
Author: ASF GitHub Bot
Created on: 30/May/19 23:01
Start Date: 30/May/19 23:01
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2647: [GOBBLIN-781]
Skeleton for GaaS DR mode clean transition
URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289201334
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java
##
@@ -135,28 +158,30 @@ public void run() {
* Load all {@link FlowSpec}s from {@link FlowCatalog} as one of the
initialization step,
* and make schedulers be aware of that.
*
+ * If it is newly brought up as the DR handler, will load additional
FlowSpecs and handle transition properly.
*/
private void scheduleSpecsFromCatalog() {
Iterator specUris = null;
long startTime = System.currentTimeMillis();
try {
specUris = this.flowCatalog.get().getSpecURIs();
+
+ // If current instances nominated as DR handler, will take additional
URIS from FlowCatalog.
+ if (isNominatedDRHanlder) {
Review comment:
isNominatedDRHandler?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251389)
> Clean Flow execution state when DR is enabled: Skeleton
> ---
>
> Key: GOBBLIN-781
> URL: https://issues.apache.org/jira/browse/GOBBLIN-781
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (GOBBLIN-781) Clean Flow execution state when DR is enabled: Skeleton
[
https://issues.apache.org/jira/browse/GOBBLIN-781?focusedWorklogId=251387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251387
]
ASF GitHub Bot logged work on GOBBLIN-781:
--
Author: ASF GitHub Bot
Created on: 30/May/19 23:01
Start Date: 30/May/19 23:01
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2647: [GOBBLIN-781]
Skeleton for GaaS DR mode clean transition
URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289199973
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/FSSpecStore.java
##
@@ -263,6 +263,12 @@ public URI next() {
};
}
+ @Override
+ public Iterator getSpecURIsWithTag(String tag) throws IOException {
Review comment:
Maybe this method should just throw an UnsupportedOperationException(),
instead of returning a wrong result.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251387)
Time Spent: 40m (was: 0.5h)
> Clean Flow execution state when DR is enabled: Skeleton
> ---
>
> Key: GOBBLIN-781
> URL: https://issues.apache.org/jira/browse/GOBBLIN-781
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (GOBBLIN-781) Clean Flow execution state when DR is enabled: Skeleton
[
https://issues.apache.org/jira/browse/GOBBLIN-781?focusedWorklogId=251388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251388
]
ASF GitHub Bot logged work on GOBBLIN-781:
--
Author: ASF GitHub Bot
Created on: 30/May/19 23:01
Start Date: 30/May/19 23:01
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2647: [GOBBLIN-781]
Skeleton for GaaS DR mode clean transition
URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289200696
##
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStore.java
##
@@ -47,32 +47,33 @@
import org.apache.gobblin.runtime.api.SpecNotFoundException;
import org.apache.gobblin.runtime.api.SpecSerDe;
import org.apache.gobblin.runtime.api.SpecStore;
-import org.apache.gobblin.util.ConfigUtils;
+
/**
* Implementation of {@link SpecStore} that stores specs as serialized java
objects in MySQL. Note that versions are not
* supported, so the version parameter will be ignored in methods that have it.
+ *
+ * A tag column is added into implementation to serve certain filtering
purposes in MySQL-based SpecStore.
Review comment:
Maybe provide an example of how the tag column could be used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251388)
Time Spent: 50m (was: 40m)
> Clean Flow execution state when DR is enabled: Skeleton
> ---
>
> Key: GOBBLIN-781
> URL: https://issues.apache.org/jira/browse/GOBBLIN-781
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2647: [GOBBLIN-781] Skeleton for GaaS DR mode clean transition
sv2000 commented on a change in pull request #2647: [GOBBLIN-781] Skeleton for GaaS DR mode clean transition URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289198551 ## File path: gobblin-restli/gobblin-throttling-service/gobblin-throttling-service-api/src/main/snapshot/org.apache.gobblin.restli.throttling.permits.snapshot.json ## @@ -17,6 +17,18 @@ "type" : "long", Review comment: I am assuming this file was not intended to be in the commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2647: [GOBBLIN-781] Skeleton for GaaS DR mode clean transition
sv2000 commented on a change in pull request #2647: [GOBBLIN-781] Skeleton for
GaaS DR mode clean transition
URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289204291
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java
##
@@ -170,6 +195,19 @@ private void scheduleSpecsFromCatalog() {
}
}
+ /**
+ * In DR-mode, the running {@link FlowSpec} will all be cancelled and
rescheduled.
+ * We will need to make sure that running {@link FlowSpec}s' state are
cleared, and corresponding running jobs are
+ * killed before rescheduling them.
+ * @param drUris The uris that applicable for DR discovered from FlowCatalog.
+ */
+ private void clearRunningFlowState(Iterator drUris) {
Review comment:
Should the DR related failover handling be moved to a separate class? For
now, you are triggering it based on config. In the future, we might have a REST
end-point or some other mechanism for triggering a fail over, without needing a
service restart.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2647: [GOBBLIN-781] Skeleton for GaaS DR mode clean transition
sv2000 commented on a change in pull request #2647: [GOBBLIN-781] Skeleton for
GaaS DR mode clean transition
URL: https://github.com/apache/incubator-gobblin/pull/2647#discussion_r289201334
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java
##
@@ -135,28 +158,30 @@ public void run() {
* Load all {@link FlowSpec}s from {@link FlowCatalog} as one of the
initialization step,
* and make schedulers be aware of that.
*
+ * If it is newly brought up as the DR handler, will load additional
FlowSpecs and handle transition properly.
*/
private void scheduleSpecsFromCatalog() {
Iterator specUris = null;
long startTime = System.currentTimeMillis();
try {
specUris = this.flowCatalog.get().getSpecURIs();
+
+ // If current instances nominated as DR handler, will take additional
URIS from FlowCatalog.
+ if (isNominatedDRHanlder) {
Review comment:
isNominatedDRHandler?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-789) Implement a FileSystem based SpecProducer
[ https://issues.apache.org/jira/browse/GOBBLIN-789?focusedWorklogId=251300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251300 ] ASF GitHub Bot logged work on GOBBLIN-789: -- Author: ASF GitHub Bot Created on: 30/May/19 20:28 Start Date: 30/May/19 20:28 Worklog Time Spent: 10m Work Description: sv2000 commented on issue #2655: GOBBLIN-789: Implement a FileSystem based SpecProducer. URL: https://github.com/apache/incubator-gobblin/pull/2655#issuecomment-497474258 @autumnust this class implements the SpecProducer interface for producing JobSpecs which will be consumed by a corresponding SpecConsumer. The target use case is to trigger restarts of jobs running in Gobblin cluster via JobSpecs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251300) Time Spent: 0.5h (was: 20m) > Implement a FileSystem based SpecProducer > - > > Key: GOBBLIN-789 > URL: https://issues.apache.org/jira/browse/GOBBLIN-789 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Implement a FSSpecProducer class that produces JobSpecs for consumption by > FsSpecConsumer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [incubator-gobblin] sv2000 commented on issue #2655: GOBBLIN-789: Implement a FileSystem based SpecProducer.
sv2000 commented on issue #2655: GOBBLIN-789: Implement a FileSystem based SpecProducer. URL: https://github.com/apache/incubator-gobblin/pull/2655#issuecomment-497474258 @autumnust this class implements the SpecProducer interface for producing JobSpecs which will be consumed by a corresponding SpecConsumer. The target use case is to trigger restarts of jobs running in Gobblin cluster via JobSpecs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-789) Implement a FileSystem based SpecProducer
[ https://issues.apache.org/jira/browse/GOBBLIN-789?focusedWorklogId=251297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251297 ] ASF GitHub Bot logged work on GOBBLIN-789: -- Author: ASF GitHub Bot Created on: 30/May/19 20:25 Start Date: 30/May/19 20:25 Worklog Time Spent: 10m Work Description: autumnust commented on issue #2655: GOBBLIN-789: Implement a FileSystem based SpecProducer. URL: https://github.com/apache/incubator-gobblin/pull/2655#issuecomment-497473120 Can you give a little bit more context of why we need this ? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251297) Time Spent: 20m (was: 10m) > Implement a FileSystem based SpecProducer > - > > Key: GOBBLIN-789 > URL: https://issues.apache.org/jira/browse/GOBBLIN-789 > Project: Apache Gobblin > Issue Type: Improvement > Components: gobblin-cluster >Affects Versions: 0.15.0 >Reporter: Sudarshan Vasudevan >Assignee: Hung Tran >Priority: Major > Fix For: 0.15.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Implement a FSSpecProducer class that produces JobSpecs for consumption by > FsSpecConsumer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [incubator-gobblin] autumnust commented on issue #2655: GOBBLIN-789: Implement a FileSystem based SpecProducer.
autumnust commented on issue #2655: GOBBLIN-789: Implement a FileSystem based SpecProducer. URL: https://github.com/apache/incubator-gobblin/pull/2655#issuecomment-497473120 Can you give a little bit more context of why we need this ? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-786) GaaS SerDe Libaray
[ https://issues.apache.org/jira/browse/GOBBLIN-786?focusedWorklogId=251294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251294 ] ASF GitHub Bot logged work on GOBBLIN-786: -- Author: ASF GitHub Bot Created on: 30/May/19 20:23 Start Date: 30/May/19 20:23 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2651: [GOBBLIN-786]Separate SerDe library in DagStateStore out for GaaS-wide sharing URL: https://github.com/apache/incubator-gobblin/pull/2651 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 251294) Time Spent: 50m (was: 40m) > GaaS SerDe Libaray > -- > > Key: GOBBLIN-786 > URL: https://issues.apache.org/jira/browse/GOBBLIN-786 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2651: [GOBBLIN-786]Separate SerDe library in DagStateStore out for GaaS-wide sharing
asfgit closed pull request #2651: [GOBBLIN-786]Separate SerDe library in DagStateStore out for GaaS-wide sharing URL: https://github.com/apache/incubator-gobblin/pull/2651 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-789) Implement a FileSystem based SpecProducer
[
https://issues.apache.org/jira/browse/GOBBLIN-789?focusedWorklogId=251235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-251235
]
ASF GitHub Bot logged work on GOBBLIN-789:
--
Author: ASF GitHub Bot
Created on: 30/May/19 18:39
Start Date: 30/May/19 18:39
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2655: GOBBLIN-789:
Implement a FileSystem based SpecProducer.
URL: https://github.com/apache/incubator-gobblin/pull/2655
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-789
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
Implement a FSSpecProducer class that produces JobSpecs for consumption by
FsSpecConsumer.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
FsSpecProducerTest and updated existing tests in
FsScheduledJobConfigurationManagerTest and IntegrationJobRestartViaSpecSuite.
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 251235)
Time Spent: 10m
Remaining Estimate: 0h
> Implement a FileSystem based SpecProducer
> -
>
> Key: GOBBLIN-789
> URL: https://issues.apache.org/jira/browse/GOBBLIN-789
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
>Affects Versions: 0.15.0
>Reporter: Sudarshan Vasudevan
>Assignee: Hung Tran
>Priority: Major
> Fix For: 0.15.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Implement a FSSpecProducer class that produces JobSpecs for consumption by
> FsSpecConsumer.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [incubator-gobblin] sv2000 opened a new pull request #2655: GOBBLIN-789: Implement a FileSystem based SpecProducer.
sv2000 opened a new pull request #2655: GOBBLIN-789: Implement a FileSystem
based SpecProducer.
URL: https://github.com/apache/incubator-gobblin/pull/2655
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-789
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
Implement a FSSpecProducer class that produces JobSpecs for consumption by
FsSpecConsumer.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
FsSpecProducerTest and updated existing tests in
FsScheduledJobConfigurationManagerTest and IntegrationJobRestartViaSpecSuite.
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-789) Implement a FileSystem based SpecProducer
Sudarshan Vasudevan created GOBBLIN-789: --- Summary: Implement a FileSystem based SpecProducer Key: GOBBLIN-789 URL: https://issues.apache.org/jira/browse/GOBBLIN-789 Project: Apache Gobblin Issue Type: Improvement Components: gobblin-cluster Affects Versions: 0.15.0 Reporter: Sudarshan Vasudevan Assignee: Hung Tran Fix For: 0.15.0 Implement a FSSpecProducer class that produces JobSpecs for consumption by FsSpecConsumer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (GOBBLIN-786) GaaS SerDe Libaray
[ https://issues.apache.org/jira/browse/GOBBLIN-786?focusedWorklogId=25=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-25 ] ASF GitHub Bot logged work on GOBBLIN-786: -- Author: ASF GitHub Bot Created on: 30/May/19 17:03 Start Date: 30/May/19 17:03 Worklog Time Spent: 10m Work Description: autumnust commented on issue #2651: [GOBBLIN-786]Separate SerDe library in DagStateStore out for GaaS-wide sharing URL: https://github.com/apache/incubator-gobblin/pull/2651#issuecomment-497404405 @jack-moseley Can you take another look at this? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 25) Time Spent: 40m (was: 0.5h) > GaaS SerDe Libaray > -- > > Key: GOBBLIN-786 > URL: https://issues.apache.org/jira/browse/GOBBLIN-786 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [incubator-gobblin] autumnust commented on issue #2651: [GOBBLIN-786]Separate SerDe library in DagStateStore out for GaaS-wide sharing
autumnust commented on issue #2651: [GOBBLIN-786]Separate SerDe library in DagStateStore out for GaaS-wide sharing URL: https://github.com/apache/incubator-gobblin/pull/2651#issuecomment-497404405 @jack-moseley Can you take another look at this? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] sv2000 opened a new pull request #2654: GOBBLIN-776: Add a utility method to return Helix WorflowId given a Gobblin job name.
sv2000 opened a new pull request #2654: GOBBLIN-776: Add a utility method to
return Helix WorflowId given a Gobblin job name.
URL: https://github.com/apache/incubator-gobblin/pull/2654
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-776
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
This task adds a utility method to return Helix WorflowId given a Gobblin
job name. This allows cancelling Gobblin jobs running on a Gobblin Helix
cluster using a Spec containing the job name, instead of requiring the spec to
provide the Helix Workflow Id.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
ClusterIntegrationTest
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
