[jira] [Work logged] (GOBBLIN-894) Add option to combine datasets into a single flow
[ https://issues.apache.org/jira/browse/GOBBLIN-894?focusedWorklogId=322400&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322400 ] ASF GitHub Bot logged work on GOBBLIN-894: -- Author: ASF GitHub Bot Created on: 03/Oct/19 01:51 Start Date: 03/Oct/19 01:51 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2749: [GOBBLIN-894] Add option to combine datasets into a single flow URL: https://github.com/apache/incubator-gobblin/pull/2749#issuecomment-537752157 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2749?src=pr&el=h1) Report > Merging [#2749](https://codecov.io/gh/apache/incubator-gobblin/pull/2749?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/5b8af8cf6e932e8031ef060c79260a4a8f47974c?src=pr&el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2749?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2749 +/- ## + Coverage 45.27% 45.29% +0.01% - Complexity 8828 8837 +9 Files 1889 1889 Lines 7063470660 +26 Branches 7757 7762 +5 + Hits 3197932003 +24 - Misses3570135704 +3 + Partials 2954 2953 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2749?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...modules/flowgraph/DatasetDescriptorConfigKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvRGF0YXNldERlc2NyaXB0b3JDb25maWdLZXlzLmphdmE=) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../service/modules/dataset/SqlDatasetDescriptor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L1NxbERhdGFzZXREZXNjcmlwdG9yLmphdmE=) | `63.88% <100%> (+1.03%)` | `7 <2> (ø)` | :arrow_down: | | [...lin/service/modules/flow/MultiHopFlowCompiler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93L011bHRpSG9wRmxvd0NvbXBpbGVyLmphdmE=) | `72.72% <100%> (+3.56%)` | `16 <1> (+2)` | :arrow_up: | | [...service/modules/dataset/BaseDatasetDescriptor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L0Jhc2VEYXRhc2V0RGVzY3JpcHRvci5qYXZh) | `72.41% <100%> (ø)` | `13 <0> (ø)` | :arrow_down: | | [...odules/template\_catalog/FSFlowTemplateCatalog.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy90ZW1wbGF0ZV9jYXRhbG9nL0ZTRmxvd1RlbXBsYXRlQ2F0YWxvZy5qYXZh) | `58.69% <100%> (-0.88%)` | `8 <0> (ø)` | | | [...n/service/modules/dataset/FSDatasetDescriptor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L0ZTRGF0YXNldERlc2NyaXB0b3IuamF2YQ==) | `77.08% <100%> (+7.63%)` | `18 <5> (+6)` | :arrow_up: | | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `63.09% <0%> (-3.58%)` | `12% <0%> (ø)` | | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `78.12% <0%> (-1.57%)` | `15% <0%> (-
[GitHub] [incubator-gobblin] codecov-io commented on issue #2749: [GOBBLIN-894] Add option to combine datasets into a single flow
codecov-io commented on issue #2749: [GOBBLIN-894] Add option to combine datasets into a single flow URL: https://github.com/apache/incubator-gobblin/pull/2749#issuecomment-537752157 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2749?src=pr&el=h1) Report > Merging [#2749](https://codecov.io/gh/apache/incubator-gobblin/pull/2749?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/5b8af8cf6e932e8031ef060c79260a4a8f47974c?src=pr&el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2749?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2749 +/- ## + Coverage 45.27% 45.29% +0.01% - Complexity 8828 8837 +9 Files 1889 1889 Lines 7063470660 +26 Branches 7757 7762 +5 + Hits 3197932003 +24 - Misses3570135704 +3 + Partials 2954 2953 -1 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2749?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...pache/gobblin/configuration/ConfigurationKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vY29uZmlndXJhdGlvbi9Db25maWd1cmF0aW9uS2V5cy5qYXZh) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...modules/flowgraph/DatasetDescriptorConfigKeys.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvRGF0YXNldERlc2NyaXB0b3JDb25maWdLZXlzLmphdmE=) | `0% <ø> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../service/modules/dataset/SqlDatasetDescriptor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L1NxbERhdGFzZXREZXNjcmlwdG9yLmphdmE=) | `63.88% <100%> (+1.03%)` | `7 <2> (ø)` | :arrow_down: | | [...lin/service/modules/flow/MultiHopFlowCompiler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93L011bHRpSG9wRmxvd0NvbXBpbGVyLmphdmE=) | `72.72% <100%> (+3.56%)` | `16 <1> (+2)` | :arrow_up: | | [...service/modules/dataset/BaseDatasetDescriptor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L0Jhc2VEYXRhc2V0RGVzY3JpcHRvci5qYXZh) | `72.41% <100%> (ø)` | `13 <0> (ø)` | :arrow_down: | | [...odules/template\_catalog/FSFlowTemplateCatalog.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy90ZW1wbGF0ZV9jYXRhbG9nL0ZTRmxvd1RlbXBsYXRlQ2F0YWxvZy5qYXZh) | `58.69% <100%> (-0.88%)` | `8 <0> (ø)` | | | [...n/service/modules/dataset/FSDatasetDescriptor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9kYXRhc2V0L0ZTRGF0YXNldERlc2NyaXB0b3IuamF2YQ==) | `77.08% <100%> (+7.63%)` | `18 <5> (+6)` | :arrow_up: | | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `63.09% <0%> (-3.58%)` | `12% <0%> (ø)` | | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `78.12% <0%> (-1.57%)` | `15% <0%> (-1%)` | | | [...dules/flowgraph/pathfinder/AbstractPathFinder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2749/diff?src=pr&el=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvcGF0aGZpbmRlci9BYnN0cmFjdFBhdGhGaW5kZXIuamF2YQ==) | `92.23% <0%> (+0.97%)` | `29% <0%> (+1%)` | :arrow_up: | | ... and [1 more](https://codecov.io/gh/apache/incubator-gobblin
[jira] [Work logged] (GOBBLIN-894) Add option to combine datasets into a single flow
[
https://issues.apache.org/jira/browse/GOBBLIN-894?focusedWorklogId=322391&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322391
]
ASF GitHub Bot logged work on GOBBLIN-894:
--
Author: ASF GitHub Bot
Created on: 03/Oct/19 01:17
Start Date: 03/Oct/19 01:17
Worklog Time Spent: 10m
Work Description: jack-moseley commented on pull request #2749:
[GOBBLIN-894] Add option to combine datasets into a single flow
URL: https://github.com/apache/incubator-gobblin/pull/2749
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-894
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
This change allows GaaS to combine datasets using a glob pattern. Classes
changed:
- Modify flow `MultiHopFlowCompiler` to set subpath config in the flow spec
if combining is specified
- Modify `FSDatasetDescriptor` to concatenate subpaths with base path and
check each against a glob pattern
- Modify `FSFlowTemplateCatalog` to not resolve config until later (this is
unnecessary and causes issues with optional configs)
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Updated tests
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322391)
Remaining Estimate: 0h
Time Spent: 10m
> Add option to combine datasets into a single flow
> -
>
> Key: GOBBLIN-894
> URL: https://issues.apache.org/jira/browse/GOBBLIN-894
> Project: Apache Gobblin
> Issue Type: Bug
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] jack-moseley opened a new pull request #2749: [GOBBLIN-894] Add option to combine datasets into a single flow
jack-moseley opened a new pull request #2749: [GOBBLIN-894] Add option to
combine datasets into a single flow
URL: https://github.com/apache/incubator-gobblin/pull/2749
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-894
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
This change allows GaaS to combine datasets using a glob pattern. Classes
changed:
- Modify flow `MultiHopFlowCompiler` to set subpath config in the flow spec
if combining is specified
- Modify `FSDatasetDescriptor` to concatenate subpaths with base path and
check each against a glob pattern
- Modify `FSFlowTemplateCatalog` to not resolve config until later (this is
unnecessary and causes issues with optional configs)
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
Updated tests
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-894) Add option to combine datasets into a single flow
Jack Moseley created GOBBLIN-894: Summary: Add option to combine datasets into a single flow Key: GOBBLIN-894 URL: https://issues.apache.org/jira/browse/GOBBLIN-894 Project: Apache Gobblin Issue Type: Bug Reporter: Jack Moseley -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-888) datePattern yyyy-MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
[
https://issues.apache.org/jira/browse/GOBBLIN-888?focusedWorklogId=322356&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322356
]
ASF GitHub Bot logged work on GOBBLIN-888:
--
Author: ASF GitHub Bot
Created on: 02/Oct/19 23:22
Start Date: 02/Oct/19 23:22
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2744: [GOBBLIN-888]
Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330640174
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
Review comment:
Instead of having 2 booleans, we can use an Enum (DAILY, HOURLY, MINUTELY).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322356)
Time Spent: 1h 20m (was: 1h 10m)
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
>
>
> Key: GOBBLIN-888
> URL: https://issues.apache.org/jira/browse/GOBBLIN-888
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Wei-Hsiang Lin
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-888) datePattern yyyy-MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
[ https://issues.apache.org/jira/browse/GOBBLIN-888?focusedWorklogId=322358&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322358 ] ASF GitHub Bot logged work on GOBBLIN-888: -- Author: ASF GitHub Bot Created on: 02/Oct/19 23:24 Start Date: 02/Oct/19 23:24 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2744: [GOBBLIN-888] Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset URL: https://github.com/apache/incubator-gobblin/pull/2744 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322358) Time Spent: 1.5h (was: 1h 20m) > datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset > > > Key: GOBBLIN-888 > URL: https://issues.apache.org/jira/browse/GOBBLIN-888 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Wei-Hsiang Lin >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2744: [GOBBLIN-888] Make yyyy-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
asfgit closed pull request #2744: [GOBBLIN-888] Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset URL: https://github.com/apache/incubator-gobblin/pull/2744 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make yyyy-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make
-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330640174
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
Review comment:
Instead of having 2 booleans, we can use an Enum (DAILY, HOURLY, MINUTELY).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2744: [GOBBLIN-888] Make yyyy-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
codecov-io edited a comment on issue #2744: [GOBBLIN-888] Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset URL: https://github.com/apache/incubator-gobblin/pull/2744#issuecomment-535327650 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2744?src=pr&el=h1) Report > Merging [#2744](https://codecov.io/gh/apache/incubator-gobblin/pull/2744?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/483d42c365ff727d5e6c08a2095ef570f9e22482?src=pr&el=desc) will **increase** coverage by `0.09%`. > The diff coverage is `90.9%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2744?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2744 +/- ## + Coverage 45.18% 45.27% +0.09% - Complexity 8803 8828 +25 Files 1889 1889 Lines 7059570634 +39 Branches 7746 7757 +11 + Hits 3189731983 +86 + Misses3574635699 -47 Partials 2952 2952 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2744?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...gement/copy/TimeAwareRecursiveCopyableDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvVGltZUF3YXJlUmVjdXJzaXZlQ29weWFibGVEYXRhc2V0LmphdmE=) | `91.17% <90.9%> (-1.85%)` | `15 <3> (+8)` | | | [...he/gobblin/kafka/client/Kafka09ConsumerClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4va2Fma2EvY2xpZW50L0thZmthMDlDb25zdW1lckNsaWVudC5qYXZh) | `76.81% <0%> (-14.42%)` | `8% <0%> (ø)` | | | [...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=) | `74.19% <0%> (-7.95%)` | `11% <0%> (ø)` | | | [...e/gobblin/data/management/copy/hive/HiveUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlVXRpbHMuamF2YQ==) | `21.95% <0%> (-2.44%)` | `3% <0%> (-1%)` | | | [...g/apache/gobblin/service/monitoring/JobStatus.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXMuamF2YQ==) | `22.22% <0%> (-1.31%)` | `4% <0%> (ø)` | | | [...main/java/org/apache/gobblin/util/WriterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvV3JpdGVyVXRpbHMuamF2YQ==) | `38.52% <0%> (-0.32%)` | `15% <0%> (ø)` | | | [...main/java/org/apache/gobblin/runtime/JobState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iU3RhdGUuamF2YQ==) | `75.28% <0%> (-0.29%)` | `58% <0%> (ø)` | | | [...ava/org/apache/gobblin/runtime/cli/GobblinCli.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvY2xpL0dvYmJsaW5DbGkuamF2YQ==) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...a/org/apache/gobblin/service/FlowStatusClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzQ2xpZW50LmphdmE=) | `72.34% <0%> (ø)` | `7% <0%> (ø)` | :arrow_down: | | [...org/apache/gobblin/service/FlowStatusResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzUmVzb3VyY2UuamF2YQ==) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | ... and [25 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/dif
[jira] [Work logged] (GOBBLIN-888) datePattern yyyy-MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
[ https://issues.apache.org/jira/browse/GOBBLIN-888?focusedWorklogId=322349&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322349 ] ASF GitHub Bot logged work on GOBBLIN-888: -- Author: ASF GitHub Bot Created on: 02/Oct/19 23:10 Start Date: 02/Oct/19 23:10 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2744: [GOBBLIN-888] Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset URL: https://github.com/apache/incubator-gobblin/pull/2744#issuecomment-535327650 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2744?src=pr&el=h1) Report > Merging [#2744](https://codecov.io/gh/apache/incubator-gobblin/pull/2744?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/483d42c365ff727d5e6c08a2095ef570f9e22482?src=pr&el=desc) will **increase** coverage by `0.09%`. > The diff coverage is `90.9%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2744?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2744 +/- ## + Coverage 45.18% 45.27% +0.09% - Complexity 8803 8828 +25 Files 1889 1889 Lines 7059570634 +39 Branches 7746 7757 +11 + Hits 3189731983 +86 + Misses3574635699 -47 Partials 2952 2952 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2744?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...gement/copy/TimeAwareRecursiveCopyableDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvVGltZUF3YXJlUmVjdXJzaXZlQ29weWFibGVEYXRhc2V0LmphdmE=) | `91.17% <90.9%> (-1.85%)` | `15 <3> (+8)` | | | [...he/gobblin/kafka/client/Kafka09ConsumerClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtMDkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4va2Fma2EvY2xpZW50L0thZmthMDlDb25zdW1lckNsaWVudC5qYXZh) | `76.81% <0%> (-14.42%)` | `8% <0%> (ø)` | | | [...obblin/service/monitoring/FlowStatusGenerator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9GbG93U3RhdHVzR2VuZXJhdG9yLmphdmE=) | `74.19% <0%> (-7.95%)` | `11% <0%> (ø)` | | | [...e/gobblin/data/management/copy/hive/HiveUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvaGl2ZS9IaXZlVXRpbHMuamF2YQ==) | `21.95% <0%> (-2.44%)` | `3% <0%> (-1%)` | | | [...g/apache/gobblin/service/monitoring/JobStatus.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9Kb2JTdGF0dXMuamF2YQ==) | `22.22% <0%> (-1.31%)` | `4% <0%> (ø)` | | | [...main/java/org/apache/gobblin/util/WriterUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvV3JpdGVyVXRpbHMuamF2YQ==) | `38.52% <0%> (-0.32%)` | `15% <0%> (ø)` | | | [...main/java/org/apache/gobblin/runtime/JobState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iU3RhdGUuamF2YQ==) | `75.28% <0%> (-0.29%)` | `58% <0%> (ø)` | | | [...ava/org/apache/gobblin/runtime/cli/GobblinCli.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvY2xpL0dvYmJsaW5DbGkuamF2YQ==) | `0% <0%> (ø)` | `0% <0%> (ø)` | :arrow_down: | | [...a/org/apache/gobblin/service/FlowStatusClient.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2744/diff?src=pr&el=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93U3RhdHVzQ2xpZW50LmphdmE=) | `72.34% <0%> (ø)` | `7% <0%> (ø)` | :arrow_down: | | [...org/apache/gobblin/service/FlowStatus
[jira] [Work logged] (GOBBLIN-893) Make ORC-format checking optional in ORC-registration
[ https://issues.apache.org/jira/browse/GOBBLIN-893?focusedWorklogId=322258&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322258 ] ASF GitHub Bot logged work on GOBBLIN-893: -- Author: ASF GitHub Bot Created on: 02/Oct/19 21:00 Start Date: 02/Oct/19 21:00 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled URL: https://github.com/apache/incubator-gobblin/pull/2748 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322258) Time Spent: 40m (was: 0.5h) > Make ORC-format checking optional in ORC-registration > - > > Key: GOBBLIN-893 > URL: https://issues.apache.org/jira/browse/GOBBLIN-893 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled
asfgit closed pull request #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled URL: https://github.com/apache/incubator-gobblin/pull/2748 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-893) Make ORC-format checking optional in ORC-registration
[ https://issues.apache.org/jira/browse/GOBBLIN-893?focusedWorklogId=322257&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322257 ] ASF GitHub Bot logged work on GOBBLIN-893: -- Author: ASF GitHub Bot Created on: 02/Oct/19 20:58 Start Date: 02/Oct/19 20:58 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled URL: https://github.com/apache/incubator-gobblin/pull/2748#issuecomment-537677782 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=h1) Report > Merging [#2748](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/37078ed6a40f73b2560863e3f73e0e1d67c05c87?src=pr&el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2748 +/- ## + Coverage 45.22% 45.25% +0.03% - Complexity 8815 8822 +7 Files 1889 1889 Lines 7061270609 -3 Branches 7753 7753 + Hits 3193331957 +24 + Misses3572535701 -24 + Partials 2954 2951 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `66.66% <100%> (+0.4%)` | `12 <0> (ø)` | :arrow_down: | | [...main/java/org/apache/gobblin/runtime/JobState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iU3RhdGUuamF2YQ==) | `75.28% <0%> (-0.29%)` | `58% <0%> (ø)` | | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: | | [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `66.43% <0%> (+4.79%)` | `25% <0%> (+3%)` | :arrow_up: | | [.../org/apache/gobblin/runtime/SafeDatasetCommit.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU2FmZURhdGFzZXRDb21taXQuamF2YQ==) | `47.31% <0%> (+7.6%)` | `29% <0%> (+4%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=footer). Last update [37078ed...ccc018e](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (wa
[jira] [Work logged] (GOBBLIN-893) Make ORC-format checking optional in ORC-registration
[ https://issues.apache.org/jira/browse/GOBBLIN-893?focusedWorklogId=322256&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322256 ] ASF GitHub Bot logged work on GOBBLIN-893: -- Author: ASF GitHub Bot Created on: 02/Oct/19 20:58 Start Date: 02/Oct/19 20:58 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled URL: https://github.com/apache/incubator-gobblin/pull/2748#issuecomment-537677782 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=h1) Report > Merging [#2748](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/37078ed6a40f73b2560863e3f73e0e1d67c05c87?src=pr&el=desc) will **decrease** coverage by `41.07%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #2748 +/- ## - Coverage 45.22% 4.14% -41.08% + Complexity 8815 742 -8073 Files 18891889 Lines 70612 70609-3 Branches 77537753 - Hits 319332930-29003 - Misses35725 67371+31646 + Partials 2954 308 -2646 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `0% <0%> (-66.27%)` | `0 <0> (-12)` | | | [...n/converter/AvroStringFieldDecryptorConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9BdnJvU3RyaW5nRmllbGREZWNyeXB0b3JDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...he/gobblin/cluster/TaskRunnerSuiteThreadModel.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvVGFza1J1bm5lclN1aXRlVGhyZWFkTW9kZWwuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-5%)` | | | [...n/mapreduce/avro/AvroKeyCompactorOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL2F2cm8vQXZyb0tleUNvbXBhY3Rvck91dHB1dEZvcm1hdC5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...apache/gobblin/fork/CopyNotSupportedException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Db3B5Tm90U3VwcG9ydGVkRXhjZXB0aW9uLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-1%)` | | | [.../gobblin/kafka/writer/KafkaWriterCommonConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3dyaXRlci9LYWZrYVdyaXRlckNvbW1vbkNvbmZpZy5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-7%)` | | | [...ker/task/TaskLevelPolicyCheckerBuilderFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3F1YWxpdHljaGVja2VyL3Rhc2svVGFza0xldmVsUG9saWN5Q2hlY2tlckJ1aWxkZXJGYWN0b3J5LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...bblin/data/management/copy/AllEqualComparator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQWxsRXF1YWxDb21wYXJhdG9yLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...blin/converter/string/ObjectToStringConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9zdHJpbmcvT2JqZWN0VG9TdHJpbmdDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...rquet/JsonIn
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled
codecov-io edited a comment on issue #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled URL: https://github.com/apache/incubator-gobblin/pull/2748#issuecomment-537677782 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=h1) Report > Merging [#2748](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/37078ed6a40f73b2560863e3f73e0e1d67c05c87?src=pr&el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2748 +/- ## + Coverage 45.22% 45.25% +0.03% - Complexity 8815 8822 +7 Files 1889 1889 Lines 7061270609 -3 Branches 7753 7753 + Hits 3193331957 +24 + Misses3572535701 -24 + Partials 2954 2951 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `66.66% <100%> (+0.4%)` | `12 <0> (ø)` | :arrow_down: | | [...main/java/org/apache/gobblin/runtime/JobState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iU3RhdGUuamF2YQ==) | `75.28% <0%> (-0.29%)` | `58% <0%> (ø)` | | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [...lin/elasticsearch/writer/FutureCallbackHolder.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tZWxhc3RpY3NlYXJjaC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9lbGFzdGljc2VhcmNoL3dyaXRlci9GdXR1cmVDYWxsYmFja0hvbGRlci5qYXZh) | `62.85% <0%> (+1.42%)` | `4% <0%> (ø)` | :arrow_down: | | [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `66.43% <0%> (+4.79%)` | `25% <0%> (+3%)` | :arrow_up: | | [.../org/apache/gobblin/runtime/SafeDatasetCommit.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU2FmZURhdGFzZXRDb21taXQuamF2YQ==) | `47.31% <0%> (+7.6%)` | `29% <0%> (+4%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=footer). Last update [37078ed...ccc018e](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] codecov-io commented on issue #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled
codecov-io commented on issue #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled URL: https://github.com/apache/incubator-gobblin/pull/2748#issuecomment-537677782 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=h1) Report > Merging [#2748](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/37078ed6a40f73b2560863e3f73e0e1d67c05c87?src=pr&el=desc) will **decrease** coverage by `41.07%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #2748 +/- ## - Coverage 45.22% 4.14% -41.08% + Complexity 8815 742 -8073 Files 18891889 Lines 70612 70609-3 Branches 77537753 - Hits 319332930-29003 - Misses35725 67371+31646 + Partials 2954 308 -2646 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2748?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...g/apache/gobblin/hive/orc/HiveOrcSerDeManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1oaXZlLXJlZ2lzdHJhdGlvbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9oaXZlL29yYy9IaXZlT3JjU2VyRGVNYW5hZ2VyLmphdmE=) | `0% <0%> (-66.27%)` | `0 <0> (-12)` | | | [...n/converter/AvroStringFieldDecryptorConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tY3J5cHRvLXByb3ZpZGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9BdnJvU3RyaW5nRmllbGREZWNyeXB0b3JDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...he/gobblin/cluster/TaskRunnerSuiteThreadModel.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvVGFza1J1bm5lclN1aXRlVGhyZWFkTW9kZWwuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-5%)` | | | [...n/mapreduce/avro/AvroKeyCompactorOutputFormat.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb21wYWN0aW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbXBhY3Rpb24vbWFwcmVkdWNlL2F2cm8vQXZyb0tleUNvbXBhY3Rvck91dHB1dEZvcm1hdC5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...apache/gobblin/fork/CopyNotSupportedException.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZm9yay9Db3B5Tm90U3VwcG9ydGVkRXhjZXB0aW9uLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-1%)` | | | [.../gobblin/kafka/writer/KafkaWriterCommonConfig.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4ta2Fma2EtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2thZmthL3dyaXRlci9LYWZrYVdyaXRlckNvbW1vbkNvbmZpZy5qYXZh) | `0% <0%> (-100%)` | `0% <0%> (-7%)` | | | [...ker/task/TaskLevelPolicyCheckerBuilderFactory.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3F1YWxpdHljaGVja2VyL3Rhc2svVGFza0xldmVsUG9saWN5Q2hlY2tlckJ1aWxkZXJGYWN0b3J5LmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...bblin/data/management/copy/AllEqualComparator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvQWxsRXF1YWxDb21wYXJhdG9yLmphdmE=) | `0% <0%> (-100%)` | `0% <0%> (-2%)` | | | [...blin/converter/string/ObjectToStringConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9zdHJpbmcvT2JqZWN0VG9TdHJpbmdDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | [...rquet/JsonIntermediateToParquetGroupConverter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tcGFycXVldC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9jb252ZXJ0ZXIvcGFycXVldC9Kc29uSW50ZXJtZWRpYXRlVG9QYXJxdWV0R3JvdXBDb252ZXJ0ZXIuamF2YQ==) | `0% <0%> (-100%)` | `0% <0%> (-3%)` | | | ... and [1082 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2748/diff?src=p
[jira] [Work logged] (GOBBLIN-892) fix a bug where we are setting/getting props in dataset state. it is not a supported operation.
[ https://issues.apache.org/jira/browse/GOBBLIN-892?focusedWorklogId=322205&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322205 ] ASF GitHub Bot logged work on GOBBLIN-892: -- Author: ASF GitHub Bot Created on: 02/Oct/19 20:13 Start Date: 02/Oct/19 20:13 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720 URL: https://github.com/apache/incubator-gobblin/pull/2747 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322205) Time Spent: 1h 40m (was: 1.5h) > fix a bug where we are setting/getting props in dataset state. it is not a > supported operation. > --- > > Key: GOBBLIN-892 > URL: https://issues.apache.org/jira/browse/GOBBLIN-892 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Arjun Singh Bora >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720
asfgit closed pull request #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720 URL: https://github.com/apache/incubator-gobblin/pull/2747 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (GOBBLIN-893) Make ORC-format checking optional in ORC-registration
Lei Sun created GOBBLIN-893: --- Summary: Make ORC-format checking optional in ORC-registration Key: GOBBLIN-893 URL: https://issues.apache.org/jira/browse/GOBBLIN-893 Project: Apache Gobblin Issue Type: Improvement Reporter: Lei Sun -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-893) Make ORC-format checking optional in ORC-registration
[
https://issues.apache.org/jira/browse/GOBBLIN-893?focusedWorklogId=322166&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322166
]
ASF GitHub Bot logged work on GOBBLIN-893:
--
Author: ASF GitHub Bot
Created on: 02/Oct/19 19:14
Start Date: 02/Oct/19 19:14
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2748:
[GOBBLIN-893]Make format-check in ORC-registration optional and by-default
disabled
URL: https://github.com/apache/incubator-gobblin/pull/2748
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] https://issues.apache.org/jira/browse/GOBBLIN-893
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322166)
Remaining Estimate: 0h
Time Spent: 10m
> Make ORC-format checking optional in ORC-registration
> -
>
> Key: GOBBLIN-893
> URL: https://issues.apache.org/jira/browse/GOBBLIN-893
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust opened a new pull request #2748: [GOBBLIN-893]Make format-check in ORC-registration optional and by-default disabled
autumnust opened a new pull request #2748: [GOBBLIN-893]Make format-check in
ORC-registration optional and by-default disabled
URL: https://github.com/apache/incubator-gobblin/pull/2748
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] https://issues.apache.org/jira/browse/GOBBLIN-893
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-892) fix a bug where we are setting/getting props in dataset state. it is not a supported operation.
[ https://issues.apache.org/jira/browse/GOBBLIN-892?focusedWorklogId=322088&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322088 ] ASF GitHub Bot logged work on GOBBLIN-892: -- Author: ASF GitHub Bot Created on: 02/Oct/19 17:42 Start Date: 02/Oct/19 17:42 Worklog Time Spent: 10m Work Description: sv2000 commented on issue #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720 URL: https://github.com/apache/incubator-gobblin/pull/2747#issuecomment-537603375 @arjun4084346 can you please add a description of what this PR is trying to address? Please describe what changes are being reverted. Can you also reference the JIRA ticket in the JIRA section? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322088) Time Spent: 1.5h (was: 1h 20m) > fix a bug where we are setting/getting props in dataset state. it is not a > supported operation. > --- > > Key: GOBBLIN-892 > URL: https://issues.apache.org/jira/browse/GOBBLIN-892 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Arjun Singh Bora >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-853) Support multiple paths specified in flow config
[ https://issues.apache.org/jira/browse/GOBBLIN-853?focusedWorklogId=322093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322093 ] ASF GitHub Bot logged work on GOBBLIN-853: -- Author: ASF GitHub Bot Created on: 02/Oct/19 17:49 Start Date: 02/Oct/19 17:49 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2709: [GOBBLIN-853] Support multiple paths specified in flow config URL: https://github.com/apache/incubator-gobblin/pull/2709 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 322093) Time Spent: 1h 20m (was: 1h 10m) > Support multiple paths specified in flow config > --- > > Key: GOBBLIN-853 > URL: https://issues.apache.org/jira/browse/GOBBLIN-853 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Jack Moseley >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2709: [GOBBLIN-853] Support multiple paths specified in flow config
asfgit closed pull request #2709: [GOBBLIN-853] Support multiple paths specified in flow config URL: https://github.com/apache/incubator-gobblin/pull/2709 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on issue #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720
sv2000 commented on issue #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720 URL: https://github.com/apache/incubator-gobblin/pull/2747#issuecomment-537603375 @arjun4084346 can you please add a description of what this PR is trying to address? Please describe what changes are being reverted. Can you also reference the JIRA ticket in the JIRA section? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-892) fix a bug where we are setting/getting props in dataset state. it is not a supported operation.
[ https://issues.apache.org/jira/browse/GOBBLIN-892?focusedWorklogId=322076&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322076 ] ASF GitHub Bot logged work on GOBBLIN-892: -- Author: ASF GitHub Bot Created on: 02/Oct/19 17:24 Start Date: 02/Oct/19 17:24 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720 URL: https://github.com/apache/incubator-gobblin/pull/2747#issuecomment-537301592 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=h1) Report > Merging [#2747](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/edc2b3a5b4073d9a0e50abea609a27d082ddbb9d?src=pr&el=desc) will **increase** coverage by `0.04%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## + Coverage 45.19% 45.23% +0.04% - Complexity 8804 8814 +10 Files 1889 1889 Lines 7058170577 -4 Branches 7747 7747 + Hits 3189931928 +29 + Misses3572935699 -30 + Partials 2953 2950 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../org/apache/gobblin/runtime/SafeDatasetCommit.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU2FmZURhdGFzZXRDb21taXQuamF2YQ==) | `47.31% <50%> (+7.6%)` | `29 <0> (+4)` | :arrow_up: | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | | | [...main/java/org/apache/gobblin/runtime/JobState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iU3RhdGUuamF2YQ==) | `75.28% <0%> (-0.29%)` | `58% <0%> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `64.81% <0%> (+0.46%)` | `28% <0%> (ø)` | :arrow_down: | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `79.68% <0%> (+1.56%)` | `16% <0%> (+1%)` | :arrow_up: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `39.25% <0%> (+3.73%)` | `13% <0%> (+1%)` | :arrow_up: | | [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `66.43% <0%> (+4.79%)` | `25% <0%> (+3%)` | :arrow_up: | | ... and [1 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree-more) | |
[GitHub] [incubator-gobblin] codecov-io edited a comment on issue #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720
codecov-io edited a comment on issue #2747: [GOBBLIN-892] reverting back bad changes done in PR 2720 URL: https://github.com/apache/incubator-gobblin/pull/2747#issuecomment-537301592 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=h1) Report > Merging [#2747](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/edc2b3a5b4073d9a0e50abea609a27d082ddbb9d?src=pr&el=desc) will **increase** coverage by `0.04%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2747 +/- ## + Coverage 45.19% 45.23% +0.04% - Complexity 8804 8814 +10 Files 1889 1889 Lines 7058170577 -4 Branches 7747 7747 + Hits 3189931928 +29 + Misses3572935699 -30 + Partials 2953 2950 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../org/apache/gobblin/runtime/SafeDatasetCommit.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvU2FmZURhdGFzZXRDb21taXQuamF2YQ==) | `47.31% <50%> (+7.6%)` | `29 <0> (+4)` | :arrow_up: | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | | | [...main/java/org/apache/gobblin/runtime/JobState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvSm9iU3RhdGUuamF2YQ==) | `75.28% <0%> (-0.29%)` | `58% <0%> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `64.81% <0%> (+0.46%)` | `28% <0%> (ø)` | :arrow_down: | | [...e/gobblin/runtime/locks/ZookeeperBasedJobLock.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvbG9ja3MvWm9va2VlcGVyQmFzZWRKb2JMb2NrLmphdmE=) | `64.44% <0%> (+1.11%)` | `16% <0%> (+1%)` | :arrow_up: | | [...ain/java/org/apache/gobblin/runtime/TaskState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvVGFza1N0YXRlLmphdmE=) | `81.97% <0%> (+1.16%)` | `32% <0%> (ø)` | :arrow_down: | | [.../org/apache/gobblin/metrics/RootMetricContext.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9Sb290TWV0cmljQ29udGV4dC5qYXZh) | `79.68% <0%> (+1.56%)` | `16% <0%> (+1%)` | :arrow_up: | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `39.25% <0%> (+3.73%)` | `13% <0%> (+1%)` | :arrow_up: | | [...a/management/copy/publisher/CopyDataPublisher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcHVibGlzaGVyL0NvcHlEYXRhUHVibGlzaGVyLmphdmE=) | `66.43% <0%> (+4.79%)` | `25% <0%> (+3%)` | :arrow_up: | | ... and [1 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2747/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2747?src=pr&el=footer). Last update [edc2b3a..
[jira] [Work logged] (GOBBLIN-888) datePattern yyyy-MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
[
https://issues.apache.org/jira/browse/GOBBLIN-888?focusedWorklogId=322010&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322010
]
ASF GitHub Bot logged work on GOBBLIN-888:
--
Author: ASF GitHub Bot
Created on: 02/Oct/19 16:13
Start Date: 02/Oct/19 16:13
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2744: [GOBBLIN-888]
Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330639449
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
+this.isPatternHourly = !this.isPatternMinutely &&
isDatePatternHourly(datePattern);
this.currentTime = properties.containsKey(DATE_PATTERN_TIMEZONE_KEY) ?
LocalDateTime.now(
DateTimeZone.forID(DATE_PATTERN_TIMEZONE_KEY))
: LocalDateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
-//Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
-if (!this.isPatternHourly) {
- Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly format");
+if (!this.isPatternMinutely) {
+ //Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
+ if (!this.isPatternHourly) {
+Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly or minutely format");
Review comment:
Change Assert.assertTrue() -> Preconditions.checkArgument().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322010)
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
>
>
> Key: GOBBLIN-888
> URL: https://issues.apache.org/jira/browse/GOBBLIN-888
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Wei-Hsiang Lin
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-888) datePattern yyyy-MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
[
https://issues.apache.org/jira/browse/GOBBLIN-888?focusedWorklogId=322008&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322008
]
ASF GitHub Bot logged work on GOBBLIN-888:
--
Author: ASF GitHub Bot
Created on: 02/Oct/19 16:13
Start Date: 02/Oct/19 16:13
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2744: [GOBBLIN-888]
Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330639458
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
+this.isPatternHourly = !this.isPatternMinutely &&
isDatePatternHourly(datePattern);
this.currentTime = properties.containsKey(DATE_PATTERN_TIMEZONE_KEY) ?
LocalDateTime.now(
DateTimeZone.forID(DATE_PATTERN_TIMEZONE_KEY))
: LocalDateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
-//Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
-if (!this.isPatternHourly) {
- Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly format");
+if (!this.isPatternMinutely) {
+ //Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
+ if (!this.isPatternHourly) {
+Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly or minutely format");
+ } else {
+//hourly directories cannot have a "minutely" lookback pattern. But
minutely directories can accept lookback pattern with hours.
+Assert.assertTrue(isLookbackTimeStringHourly(this.lookbackTime),
"Expected hourly format for lookback time; found minutely format");
Review comment:
Assert.assertTrue() -> Preconditions.checkArgument().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322008)
Time Spent: 40m (was: 0.5h)
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
>
>
> Key: GOBBLIN-888
> URL: https://issues.apache.org/jira/browse/GOBBLIN-888
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Wei-Hsiang Lin
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-888) datePattern yyyy-MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
[
https://issues.apache.org/jira/browse/GOBBLIN-888?focusedWorklogId=322009&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322009
]
ASF GitHub Bot logged work on GOBBLIN-888:
--
Author: ASF GitHub Bot
Created on: 02/Oct/19 16:13
Start Date: 02/Oct/19 16:13
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2744: [GOBBLIN-888]
Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330639400
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
+this.isPatternHourly = !this.isPatternMinutely &&
isDatePatternHourly(datePattern);
this.currentTime = properties.containsKey(DATE_PATTERN_TIMEZONE_KEY) ?
LocalDateTime.now(
DateTimeZone.forID(DATE_PATTERN_TIMEZONE_KEY))
: LocalDateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
-//Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
-if (!this.isPatternHourly) {
- Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly format");
+if (!this.isPatternMinutely) {
Review comment:
Define this.isPatternDaily first and rewrite this block as: if
(this.isPatternDaily) {...} else if (this.isPatternHourly) {...} else {...}.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322009)
Time Spent: 40m (was: 0.5h)
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
>
>
> Key: GOBBLIN-888
> URL: https://issues.apache.org/jira/browse/GOBBLIN-888
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Wei-Hsiang Lin
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-888) datePattern yyyy-MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
[
https://issues.apache.org/jira/browse/GOBBLIN-888?focusedWorklogId=322012&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322012
]
ASF GitHub Bot logged work on GOBBLIN-888:
--
Author: ASF GitHub Bot
Created on: 02/Oct/19 16:13
Start Date: 02/Oct/19 16:13
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2744: [GOBBLIN-888]
Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330640174
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
Review comment:
Instead of having 2 booleans, we can use an Enum (DAILY, HOURLY, MINUTELY).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322012)
Time Spent: 1h (was: 50m)
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
>
>
> Key: GOBBLIN-888
> URL: https://issues.apache.org/jira/browse/GOBBLIN-888
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Wei-Hsiang Lin
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-888) datePattern yyyy-MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
[
https://issues.apache.org/jira/browse/GOBBLIN-888?focusedWorklogId=322011&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-322011
]
ASF GitHub Bot logged work on GOBBLIN-888:
--
Author: ASF GitHub Bot
Created on: 02/Oct/19 16:13
Start Date: 02/Oct/19 16:13
Worklog Time Spent: 10m
Work Description: sv2000 commented on pull request #2744: [GOBBLIN-888]
Make -MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330639687
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
+this.isPatternHourly = !this.isPatternMinutely &&
isDatePatternHourly(datePattern);
this.currentTime = properties.containsKey(DATE_PATTERN_TIMEZONE_KEY) ?
LocalDateTime.now(
DateTimeZone.forID(DATE_PATTERN_TIMEZONE_KEY))
: LocalDateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
-//Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
-if (!this.isPatternHourly) {
- Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly format");
+if (!this.isPatternMinutely) {
+ //Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
+ if (!this.isPatternHourly) {
+Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly or minutely format");
+ } else {
+//hourly directories cannot have a "minutely" lookback pattern. But
minutely directories can accept lookback pattern with hours.
+Assert.assertTrue(isLookbackTimeStringHourly(this.lookbackTime),
"Expected hourly format for lookback time; found minutely format");
+ }
}
}
public static class DateRangeIterator implements Iterator {
private LocalDateTime startDate;
private LocalDateTime endDate;
private boolean isDatePatternHourly;
+private boolean isDatePatternMinutely;
-public DateRangeIterator(LocalDateTime startDate, LocalDateTime endDate,
boolean isDatePatternHourly) {
+public DateRangeIterator(LocalDateTime startDate, LocalDateTime endDate,
boolean isDatePatternHourly, boolean isDatePatternMinutely) {
Review comment:
Can we change the signature of this method to pass an Enum instead of 2
booleans?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 322011)
Time Spent: 50m (was: 40m)
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
>
>
> Key: GOBBLIN-888
> URL: https://issues.apache.org/jira/browse/GOBBLIN-888
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Wei-Hsiang Lin
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> datePattern -MM-dd-HH-mm recognized by TimeAwareRecursiveCopyableDataset
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make yyyy-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make
-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330640174
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
Review comment:
Instead of having 2 booleans, we can use an Enum (DAILY, HOURLY, MINUTELY).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make yyyy-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make
-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330639458
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
+this.isPatternHourly = !this.isPatternMinutely &&
isDatePatternHourly(datePattern);
this.currentTime = properties.containsKey(DATE_PATTERN_TIMEZONE_KEY) ?
LocalDateTime.now(
DateTimeZone.forID(DATE_PATTERN_TIMEZONE_KEY))
: LocalDateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
-//Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
-if (!this.isPatternHourly) {
- Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly format");
+if (!this.isPatternMinutely) {
+ //Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
+ if (!this.isPatternHourly) {
+Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly or minutely format");
+ } else {
+//hourly directories cannot have a "minutely" lookback pattern. But
minutely directories can accept lookback pattern with hours.
+Assert.assertTrue(isLookbackTimeStringHourly(this.lookbackTime),
"Expected hourly format for lookback time; found minutely format");
Review comment:
Assert.assertTrue() -> Preconditions.checkArgument().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make yyyy-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make
-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330639449
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
+this.isPatternHourly = !this.isPatternMinutely &&
isDatePatternHourly(datePattern);
this.currentTime = properties.containsKey(DATE_PATTERN_TIMEZONE_KEY) ?
LocalDateTime.now(
DateTimeZone.forID(DATE_PATTERN_TIMEZONE_KEY))
: LocalDateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
-//Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
-if (!this.isPatternHourly) {
- Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly format");
+if (!this.isPatternMinutely) {
+ //Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
+ if (!this.isPatternHourly) {
+Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly or minutely format");
Review comment:
Change Assert.assertTrue() -> Preconditions.checkArgument().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make yyyy-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make
-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330639400
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
+this.isPatternHourly = !this.isPatternMinutely &&
isDatePatternHourly(datePattern);
this.currentTime = properties.containsKey(DATE_PATTERN_TIMEZONE_KEY) ?
LocalDateTime.now(
DateTimeZone.forID(DATE_PATTERN_TIMEZONE_KEY))
: LocalDateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
-//Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
-if (!this.isPatternHourly) {
- Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly format");
+if (!this.isPatternMinutely) {
Review comment:
Define this.isPatternDaily first and rewrite this block as: if
(this.isPatternDaily) {...} else if (this.isPatternHourly) {...} else {...}.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make yyyy-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
sv2000 commented on a change in pull request #2744: [GOBBLIN-888] Make
-MM-dd-HH-mm recognizable in TimeAwareRecursiveCopyableDataset
URL: https://github.com/apache/incubator-gobblin/pull/2744#discussion_r330639687
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/TimeAwareRecursiveCopyableDataset.java
##
@@ -50,34 +50,43 @@
private final String datePattern;
private final Period lookbackPeriod;
private final boolean isPatternHourly;
+ private final boolean isPatternMinutely;
private final LocalDateTime currentTime;
public TimeAwareRecursiveCopyableDataset(FileSystem fs, Path rootPath,
Properties properties, Path glob) {
super(fs, rootPath, properties, glob);
this.lookbackTime = properties.getProperty(LOOKBACK_TIME_KEY);
-PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").toFormatter();
+PeriodFormatter periodFormatter = new
PeriodFormatterBuilder().appendDays().appendSuffix("d").appendHours().appendSuffix("h").appendMinutes().appendSuffix("m").toFormatter();
this.lookbackPeriod = periodFormatter.parsePeriod(lookbackTime);
this.datePattern = properties.getProperty(DATE_PATTERN_KEY);
-this.isPatternHourly = isDatePatternHourly(datePattern);
+this.isPatternMinutely = isDatePatternMinutely(datePattern);
+this.isPatternHourly = !this.isPatternMinutely &&
isDatePatternHourly(datePattern);
this.currentTime = properties.containsKey(DATE_PATTERN_TIMEZONE_KEY) ?
LocalDateTime.now(
DateTimeZone.forID(DATE_PATTERN_TIMEZONE_KEY))
: LocalDateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
-//Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
-if (!this.isPatternHourly) {
- Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly format");
+if (!this.isPatternMinutely) {
+ //Daily directories cannot have a "hourly" lookback pattern. But hourly
directories can accept lookback pattern with days.
+ if (!this.isPatternHourly) {
+Assert.assertTrue(isLookbackTimeStringDaily(this.lookbackTime),
"Expected day format for lookback time; found hourly or minutely format");
+ } else {
+//hourly directories cannot have a "minutely" lookback pattern. But
minutely directories can accept lookback pattern with hours.
+Assert.assertTrue(isLookbackTimeStringHourly(this.lookbackTime),
"Expected hourly format for lookback time; found minutely format");
+ }
}
}
public static class DateRangeIterator implements Iterator {
private LocalDateTime startDate;
private LocalDateTime endDate;
private boolean isDatePatternHourly;
+private boolean isDatePatternMinutely;
-public DateRangeIterator(LocalDateTime startDate, LocalDateTime endDate,
boolean isDatePatternHourly) {
+public DateRangeIterator(LocalDateTime startDate, LocalDateTime endDate,
boolean isDatePatternHourly, boolean isDatePatternMinutely) {
Review comment:
Can we change the signature of this method to pass an Enum instead of 2
booleans?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
Gobblin podling report sign-off
Hi mentors, Please review and sign-off Gobblin's podling report: https://cwiki.apache.org/confluence/display/INCUBATOR/October2019#Gobblin Regards, Abhishek
