[jira] [Work logged] (GOBBLIN-1063) add some more info to logs
[ https://issues.apache.org/jira/browse/GOBBLIN-1063?focusedWorklogId=394649&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394649 ] ASF GitHub Bot logged work on GOBBLIN-1063: --- Author: ASF GitHub Bot Created on: 28/Feb/20 05:53 Start Date: 28/Feb/20 05:53 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2902: [GOBBLIN-1063] add log URL: https://github.com/apache/incubator-gobblin/pull/2902 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394649) Time Spent: 40m (was: 0.5h) > add some more info to logs > -- > > Key: GOBBLIN-1063 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1063 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Arjun Singh Bora >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2902: [GOBBLIN-1063] add log
asfgit closed pull request #2902: [GOBBLIN-1063] add log URL: https://github.com/apache/incubator-gobblin/pull/2902 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #2633: GOBBLIN-759: Added feature to support DistCP to copy files that were …
arjun4084346 commented on a change in pull request #2633: GOBBLIN-759: Added
feature to support DistCP to copy files that were …
URL: https://github.com/apache/incubator-gobblin/pull/2633#discussion_r385511940
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/ModifiedDateRangeBasedFileFilter.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.data.management.copy;
+
+import java.util.Properties;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.joda.time.DateTime;
+import org.joda.time.Period;
+
+
+/**
+ * A {@link CopyableFileFilter} that drops a {@link CopyableFile} if file
modification time not within the lookback
Review comment:
same. "in"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-759) DistCP files modified in last n days within a look back period
[
https://issues.apache.org/jira/browse/GOBBLIN-759?focusedWorklogId=394646&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394646
]
ASF GitHub Bot logged work on GOBBLIN-759:
--
Author: ASF GitHub Bot
Created on: 28/Feb/20 05:19
Start Date: 28/Feb/20 05:19
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on pull request #2633:
GOBBLIN-759: Added feature to support DistCP to copy files that were …
URL: https://github.com/apache/incubator-gobblin/pull/2633#discussion_r385511737
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/DateRangeBasedFileFilter.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.data.management.copy;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import java.util.Collection;
+import java.util.Iterator;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.hadoop.fs.FileSystem;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.Period;
+import org.joda.time.format.PeriodFormatter;
+import org.joda.time.format.PeriodFormatterBuilder;
+
+
+/**
+ * A {@link CopyableFileFilter} that drops a {@link CopyableFile} if file
modification time not within the lookback
+ * window
+ * sourceFs
+ */
+@Slf4j
+public class DateRangeBasedFileFilter implements CopyableFileFilter {
+
+ private Period minLookBackPeriod;
+ private Period maxLookBackPeriod;
+ private String timezone;
+ private DateTime currentTime;
+ private DateTime minLookBackTime;
+ private DateTime maxLookBackTime;
+
+ public static final String CONFIGURATION_KEY_PREFIX =
"gobblin.dataset.filter.";
+ public static final String DEFAULT_DATE_PATTERN_TIMEZONE =
ConfigurationKeys.PST_TIMEZONE_NAME;
+
+ public DateRangeBasedFileFilter(Period minLookback, Period maxLookback,
String timezone) {
+this.minLookBackPeriod = minLookback;
+this.maxLookBackPeriod = maxLookback;
+this.timezone = timezone;
+this.currentTime = !Strings.isNullOrEmpty(this.timezone) ?
DateTime.now(DateTimeZone.forID(this.timezone))
+: DateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
+this.minLookBackTime = this.currentTime.minus(this.minLookBackPeriod);
+this.maxLookBackTime = this.currentTime.minus(this.maxLookBackPeriod);
+ }
+
+ /**
+ * For every {@link CopyableFile} in copyableFiles checks if a
+ * {@link CopyableFile#getOrigin()#getPath()#getModificationTime()}
+ * + date between the min and max look back window on sourceFs
{@inheritDoc}
+ *
+ * @see CopyableFileFilter#filter(FileSystem,
+ * FileSystem, Collection)
+ */
+ @Override
+ public Collection filter(FileSystem sourceFs, FileSystem
targetFs,
+ Collection copyableFiles) {
+Iterator iterator = copyableFiles.iterator();
+
+ImmutableList.Builder filtered = ImmutableList.builder();
+
+while (iterator.hasNext()) {
+ CopyableFile file = iterator.next();
+ if
(isFileModifiedWithinLookBackPeriod(file.getOrigin().getModificationTime())) {
+filtered.add(file);
+ }
+}
+
+return filtered.build();
+ }
+
+ /**
+ *
+ * @param modTime file modification time in long.
+ * @return true if the file modification time within lookback
window;
+ * false if file modification time not within lookback
window.
+ *
+ */
+ private boolean isFileModifiedWithinLookBackPeriod(long modTime) {
+DateTime modifiedTime =
+!Strings.isNullOrEmpty(this.timezone) ? new DateTime(modTime,
DateTimeZone.forID(this.timezone))
+: new DateTime(modTime,
DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
Review comment:
I found this more readable
Strings.isNullOrEmpty(this.timezone) ? new DateTime(modTime,
DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE)) : new DateTime(modTime,
DateTimeZone.forID(this.timezone))
: ;
just saying :D
---
[jira] [Work logged] (GOBBLIN-759) DistCP files modified in last n days within a look back period
[
https://issues.apache.org/jira/browse/GOBBLIN-759?focusedWorklogId=394647&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394647
]
ASF GitHub Bot logged work on GOBBLIN-759:
--
Author: ASF GitHub Bot
Created on: 28/Feb/20 05:19
Start Date: 28/Feb/20 05:19
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on pull request #2633:
GOBBLIN-759: Added feature to support DistCP to copy files that were …
URL: https://github.com/apache/incubator-gobblin/pull/2633#discussion_r385511940
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/ModifiedDateRangeBasedFileFilter.java
##
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.data.management.copy;
+
+import java.util.Properties;
+
+import lombok.extern.slf4j.Slf4j;
+
+import org.joda.time.DateTime;
+import org.joda.time.Period;
+
+
+/**
+ * A {@link CopyableFileFilter} that drops a {@link CopyableFile} if file
modification time not within the lookback
Review comment:
same. "in"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394647)
Time Spent: 6h 40m (was: 6.5h)
> DistCP files modified in last n days within a look back period
> --
>
> Key: GOBBLIN-759
> URL: https://issues.apache.org/jira/browse/GOBBLIN-759
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Karthik Amarnath
>Priority: Major
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> *Feature Request:*
> # DistCP only the files modified in last n days within the look back window.
> # DistCP will copy only the files modified even when the source file which
> were NOT modified in last n days in the destination directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #2633: GOBBLIN-759: Added feature to support DistCP to copy files that were …
arjun4084346 commented on a change in pull request #2633: GOBBLIN-759: Added
feature to support DistCP to copy files that were …
URL: https://github.com/apache/incubator-gobblin/pull/2633#discussion_r385511737
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/DateRangeBasedFileFilter.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.data.management.copy;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import java.util.Collection;
+import java.util.Iterator;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.hadoop.fs.FileSystem;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.Period;
+import org.joda.time.format.PeriodFormatter;
+import org.joda.time.format.PeriodFormatterBuilder;
+
+
+/**
+ * A {@link CopyableFileFilter} that drops a {@link CopyableFile} if file
modification time not within the lookback
+ * window
+ * sourceFs
+ */
+@Slf4j
+public class DateRangeBasedFileFilter implements CopyableFileFilter {
+
+ private Period minLookBackPeriod;
+ private Period maxLookBackPeriod;
+ private String timezone;
+ private DateTime currentTime;
+ private DateTime minLookBackTime;
+ private DateTime maxLookBackTime;
+
+ public static final String CONFIGURATION_KEY_PREFIX =
"gobblin.dataset.filter.";
+ public static final String DEFAULT_DATE_PATTERN_TIMEZONE =
ConfigurationKeys.PST_TIMEZONE_NAME;
+
+ public DateRangeBasedFileFilter(Period minLookback, Period maxLookback,
String timezone) {
+this.minLookBackPeriod = minLookback;
+this.maxLookBackPeriod = maxLookback;
+this.timezone = timezone;
+this.currentTime = !Strings.isNullOrEmpty(this.timezone) ?
DateTime.now(DateTimeZone.forID(this.timezone))
+: DateTime.now(DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
+this.minLookBackTime = this.currentTime.minus(this.minLookBackPeriod);
+this.maxLookBackTime = this.currentTime.minus(this.maxLookBackPeriod);
+ }
+
+ /**
+ * For every {@link CopyableFile} in copyableFiles checks if a
+ * {@link CopyableFile#getOrigin()#getPath()#getModificationTime()}
+ * + date between the min and max look back window on sourceFs
{@inheritDoc}
+ *
+ * @see CopyableFileFilter#filter(FileSystem,
+ * FileSystem, Collection)
+ */
+ @Override
+ public Collection filter(FileSystem sourceFs, FileSystem
targetFs,
+ Collection copyableFiles) {
+Iterator iterator = copyableFiles.iterator();
+
+ImmutableList.Builder filtered = ImmutableList.builder();
+
+while (iterator.hasNext()) {
+ CopyableFile file = iterator.next();
+ if
(isFileModifiedWithinLookBackPeriod(file.getOrigin().getModificationTime())) {
+filtered.add(file);
+ }
+}
+
+return filtered.build();
+ }
+
+ /**
+ *
+ * @param modTime file modification time in long.
+ * @return true if the file modification time within lookback
window;
+ * false if file modification time not within lookback
window.
+ *
+ */
+ private boolean isFileModifiedWithinLookBackPeriod(long modTime) {
+DateTime modifiedTime =
+!Strings.isNullOrEmpty(this.timezone) ? new DateTime(modTime,
DateTimeZone.forID(this.timezone))
+: new DateTime(modTime,
DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE));
Review comment:
I found this more readable
Strings.isNullOrEmpty(this.timezone) ? new DateTime(modTime,
DateTimeZone.forID(DEFAULT_DATE_PATTERN_TIMEZONE)) : new DateTime(modTime,
DateTimeZone.forID(this.timezone))
: ;
just saying :D
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-759) DistCP files modified in last n days within a look back period
[
https://issues.apache.org/jira/browse/GOBBLIN-759?focusedWorklogId=394645&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394645
]
ASF GitHub Bot logged work on GOBBLIN-759:
--
Author: ASF GitHub Bot
Created on: 28/Feb/20 05:10
Start Date: 28/Feb/20 05:10
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on pull request #2633:
GOBBLIN-759: Added feature to support DistCP to copy files that were …
URL: https://github.com/apache/incubator-gobblin/pull/2633#discussion_r385509959
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/DateRangeBasedFileFilter.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.data.management.copy;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import java.util.Collection;
+import java.util.Iterator;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.hadoop.fs.FileSystem;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.Period;
+import org.joda.time.format.PeriodFormatter;
+import org.joda.time.format.PeriodFormatterBuilder;
+
+
+/**
+ * A {@link CopyableFileFilter} that drops a {@link CopyableFile} if file
modification time not within the lookback
Review comment:
time is not ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394645)
Time Spent: 6h 20m (was: 6h 10m)
> DistCP files modified in last n days within a look back period
> --
>
> Key: GOBBLIN-759
> URL: https://issues.apache.org/jira/browse/GOBBLIN-759
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Karthik Amarnath
>Priority: Major
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> *Feature Request:*
> # DistCP only the files modified in last n days within the look back window.
> # DistCP will copy only the files modified even when the source file which
> were NOT modified in last n days in the destination directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #2633: GOBBLIN-759: Added feature to support DistCP to copy files that were …
arjun4084346 commented on a change in pull request #2633: GOBBLIN-759: Added
feature to support DistCP to copy files that were …
URL: https://github.com/apache/incubator-gobblin/pull/2633#discussion_r385509959
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/DateRangeBasedFileFilter.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.data.management.copy;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import java.util.Collection;
+import java.util.Iterator;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.hadoop.fs.FileSystem;
+import org.joda.time.DateTime;
+import org.joda.time.DateTimeZone;
+import org.joda.time.Period;
+import org.joda.time.format.PeriodFormatter;
+import org.joda.time.format.PeriodFormatterBuilder;
+
+
+/**
+ * A {@link CopyableFileFilter} that drops a {@link CopyableFile} if file
modification time not within the lookback
Review comment:
time is not ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-gobblin] codecov-io commented on issue #2903: [GOBBLIN-1065] Fix SSL verification issue for macOS
codecov-io commented on issue #2903: [GOBBLIN-1065] Fix SSL verification issue for macOS URL: https://github.com/apache/incubator-gobblin/pull/2903#issuecomment-592281894 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=h1) Report > Merging [#2903](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/998c968cb84ff7a0f3951ae648cb9d2ed1c8e4ad?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2903 +/- ## - Coverage 45.86% 45.86% -0.01% Complexity 9169 9169 Files 1932 1932 Lines 7279572795 Branches 8022 8022 - Hits 3338933384 -5 - Misses3634936352 +3 - Partials 3057 3059 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2903/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2903/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-3.48%)` | `14% <0%> (-1%)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2903/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2903/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `65.06% <0%> (+0.43%)` | `29% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=footer). Last update [998c968...097194d](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1065) Fix Merge Script failure for SSL in MacOS
[ https://issues.apache.org/jira/browse/GOBBLIN-1065?focusedWorklogId=394611&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394611 ] ASF GitHub Bot logged work on GOBBLIN-1065: --- Author: ASF GitHub Bot Created on: 28/Feb/20 02:36 Start Date: 28/Feb/20 02:36 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2903: [GOBBLIN-1065] Fix SSL verification issue for macOS URL: https://github.com/apache/incubator-gobblin/pull/2903#issuecomment-592281894 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=h1) Report > Merging [#2903](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/998c968cb84ff7a0f3951ae648cb9d2ed1c8e4ad?src=pr&el=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2903 +/- ## - Coverage 45.86% 45.86% -0.01% Complexity 9169 9169 Files 1932 1932 Lines 7279572795 Branches 8022 8022 - Hits 3338933384 -5 - Misses3634936352 +3 - Partials 3057 3059 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2903/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2903/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-3.48%)` | `14% <0%> (-1%)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2903/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | | | [.../org/apache/gobblin/cluster/GobblinTaskRunner.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2903/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpblRhc2tSdW5uZXIuamF2YQ==) | `65.06% <0%> (+0.43%)` | `29% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=footer). Last update [998c968...097194d](https://codecov.io/gh/apache/incubator-gobblin/pull/2903?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394611) Time Spent: 0.5h (was: 20m) > Fix Merge Script failure for SSL in MacOS > - > > Key: GOBBLIN-1065 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1065 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Lei Sun >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (GOBBLIN-1054) Refactor HiveSource to make partition filter extensible
[ https://issues.apache.org/jira/browse/GOBBLIN-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Sun resolved GOBBLIN-1054. -- Fix Version/s: 0.15.0 Resolution: Fixed Issue resolved by pull request #2894 [https://github.com/apache/incubator-gobblin/pull/2894] > Refactor HiveSource to make partition filter extensible > --- > > Key: GOBBLIN-1054 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1054 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Fix For: 0.15.0 > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1054) Refactor HiveSource to make partition filter extensible
[ https://issues.apache.org/jira/browse/GOBBLIN-1054?focusedWorklogId=394604&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394604 ] ASF GitHub Bot logged work on GOBBLIN-1054: --- Author: ASF GitHub Bot Created on: 28/Feb/20 02:12 Start Date: 28/Feb/20 02:12 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible URL: https://github.com/apache/incubator-gobblin/pull/2894 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394604) Time Spent: 2h (was: 1h 50m) > Refactor HiveSource to make partition filter extensible > --- > > Key: GOBBLIN-1054 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1054 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
asfgit closed pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible URL: https://github.com/apache/incubator-gobblin/pull/2894 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] autumnust commented on issue #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
autumnust commented on issue #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible URL: https://github.com/apache/incubator-gobblin/pull/2894#issuecomment-592275598 +1 LGTM. Thanks for the contribution. @KumaravelKandasami This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1054) Refactor HiveSource to make partition filter extensible
[ https://issues.apache.org/jira/browse/GOBBLIN-1054?focusedWorklogId=394602&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394602 ] ASF GitHub Bot logged work on GOBBLIN-1054: --- Author: ASF GitHub Bot Created on: 28/Feb/20 02:10 Start Date: 28/Feb/20 02:10 Worklog Time Spent: 10m Work Description: autumnust commented on issue #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible URL: https://github.com/apache/incubator-gobblin/pull/2894#issuecomment-592275598 +1 LGTM. Thanks for the contribution. @KumaravelKandasami This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394602) Time Spent: 1h 50m (was: 1h 40m) > Refactor HiveSource to make partition filter extensible > --- > > Key: GOBBLIN-1054 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1054 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Lei Sun >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1054) Refactor HiveSource to make partition filter extensible
[
https://issues.apache.org/jira/browse/GOBBLIN-1054?focusedWorklogId=394601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394601
]
ASF GitHub Bot logged work on GOBBLIN-1054:
---
Author: ASF GitHub Bot
Created on: 28/Feb/20 02:08
Start Date: 28/Feb/20 02:08
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2894:
GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r385474038
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/filter/LookbackPartitionFilterGenerator.java
##
@@ -41,34 +40,44 @@
* must be such that lexycographical string and date ordering are compatible.
*
*/
+@Slf4j
public class LookbackPartitionFilterGenerator implements
PartitionFilterGenerator {
public static final String PARTITION_COLUMN =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.column";
public static final String LOOKBACK = HiveDatasetFinder.HIVE_DATASET_PREFIX
+ ".partition.filter.datetime.lookback";
public static final String DATETIME_FORMAT =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.format";
- private static final String ERROR_MESSAGE =
LookbackPartitionFilterGenerator.class.getName()
- + " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT});
-
- private final String partitionColumn;
- private final Period lookback;
- private final DateTimeFormatter formatter;
+ private final Properties prop;
public LookbackPartitionFilterGenerator(Properties properties) {
-Preconditions.checkArgument(properties.containsKey(PARTITION_COLUMN),
ERROR_MESSAGE);
-Preconditions.checkArgument(properties.containsKey(LOOKBACK),
ERROR_MESSAGE);
-Preconditions.checkArgument(properties.containsKey(DATETIME_FORMAT),
ERROR_MESSAGE);
-
-this.partitionColumn = properties.getProperty(PARTITION_COLUMN);
-this.lookback = Period.parse(properties.getProperty(LOOKBACK));
-this.formatter =
DateTimeFormat.forPattern(properties.getProperty(DATETIME_FORMAT));
+this.prop = (properties == null) ? System.getProperties(): properties;
}
@Override
public String getFilter(HiveDataset hiveDataset) {
-DateTime limitDate = (new DateTime()).minus(this.lookback);
+if (isValidConfig()) {
Review comment:
Sounds good to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394601)
Time Spent: 1h 40m (was: 1.5h)
> Refactor HiveSource to make partition filter extensible
> ---
>
> Key: GOBBLIN-1054
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1054
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust commented on a change in pull request #2894: GOBBLIN-1054: Refactor HiveSource to make partition filter extensible
autumnust commented on a change in pull request #2894: GOBBLIN-1054: Refactor
HiveSource to make partition filter extensible
URL: https://github.com/apache/incubator-gobblin/pull/2894#discussion_r385474038
##
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/filter/LookbackPartitionFilterGenerator.java
##
@@ -41,34 +40,44 @@
* must be such that lexycographical string and date ordering are compatible.
*
*/
+@Slf4j
public class LookbackPartitionFilterGenerator implements
PartitionFilterGenerator {
public static final String PARTITION_COLUMN =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.column";
public static final String LOOKBACK = HiveDatasetFinder.HIVE_DATASET_PREFIX
+ ".partition.filter.datetime.lookback";
public static final String DATETIME_FORMAT =
HiveDatasetFinder.HIVE_DATASET_PREFIX + ".partition.filter.datetime.format";
- private static final String ERROR_MESSAGE =
LookbackPartitionFilterGenerator.class.getName()
- + " requires the following properties " + Arrays.toString(new
String[]{PARTITION_COLUMN, LOOKBACK, DATETIME_FORMAT});
-
- private final String partitionColumn;
- private final Period lookback;
- private final DateTimeFormatter formatter;
+ private final Properties prop;
public LookbackPartitionFilterGenerator(Properties properties) {
-Preconditions.checkArgument(properties.containsKey(PARTITION_COLUMN),
ERROR_MESSAGE);
-Preconditions.checkArgument(properties.containsKey(LOOKBACK),
ERROR_MESSAGE);
-Preconditions.checkArgument(properties.containsKey(DATETIME_FORMAT),
ERROR_MESSAGE);
-
-this.partitionColumn = properties.getProperty(PARTITION_COLUMN);
-this.lookback = Period.parse(properties.getProperty(LOOKBACK));
-this.formatter =
DateTimeFormat.forPattern(properties.getProperty(DATETIME_FORMAT));
+this.prop = (properties == null) ? System.getProperties(): properties;
}
@Override
public String getFilter(HiveDataset hiveDataset) {
-DateTime limitDate = (new DateTime()).minus(this.lookback);
+if (isValidConfig()) {
Review comment:
Sounds good to me.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1065) Fix Merge Script failure for SSL in MacOS
[
https://issues.apache.org/jira/browse/GOBBLIN-1065?focusedWorklogId=394598&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394598
]
ASF GitHub Bot logged work on GOBBLIN-1065:
---
Author: ASF GitHub Bot
Created on: 28/Feb/20 02:04
Start Date: 28/Feb/20 02:04
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2903:
[GOBBLIN-1065] Fix SSL verification issue for macOS
URL: https://github.com/apache/incubator-gobblin/pull/2903
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-XXX
### Description
Symptoms:
```ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
(_ssl.c:1056)```
Solution found from:
https://stackoverflow.com/questions/35569042/ssl-certificate-verify-failed-with-python3
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394598)
Remaining Estimate: 0h
Time Spent: 10m
> Fix Merge Script failure for SSL in MacOS
> -
>
> Key: GOBBLIN-1065
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1065
> Project: Apache Gobblin
> Issue Type: Bug
>Reporter: Lei Sun
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust commented on issue #2903: [GOBBLIN-1065] Fix SSL verification issue for macOS
autumnust commented on issue #2903: [GOBBLIN-1065] Fix SSL verification issue for macOS URL: https://github.com/apache/incubator-gobblin/pull/2903#issuecomment-592274046 @htran1 Can you take a look ? Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1065) Fix Merge Script failure for SSL in MacOS
[ https://issues.apache.org/jira/browse/GOBBLIN-1065?focusedWorklogId=394599&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394599 ] ASF GitHub Bot logged work on GOBBLIN-1065: --- Author: ASF GitHub Bot Created on: 28/Feb/20 02:04 Start Date: 28/Feb/20 02:04 Worklog Time Spent: 10m Work Description: autumnust commented on issue #2903: [GOBBLIN-1065] Fix SSL verification issue for macOS URL: https://github.com/apache/incubator-gobblin/pull/2903#issuecomment-592274046 @htran1 Can you take a look ? Thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394599) Time Spent: 20m (was: 10m) > Fix Merge Script failure for SSL in MacOS > - > > Key: GOBBLIN-1065 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1065 > Project: Apache Gobblin > Issue Type: Bug >Reporter: Lei Sun >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust opened a new pull request #2903: [GOBBLIN-1065] Fix SSL verification issue for macOS
autumnust opened a new pull request #2903: [GOBBLIN-1065] Fix SSL verification
issue for macOS
URL: https://github.com/apache/incubator-gobblin/pull/2903
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-XXX
### Description
Symptoms:
```ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
(_ssl.c:1056)```
Solution found from:
https://stackoverflow.com/questions/35569042/ssl-certificate-verify-failed-with-python3
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (GOBBLIN-1065) Fix Merge Script failure for SSL in MacOS
Lei Sun created GOBBLIN-1065: Summary: Fix Merge Script failure for SSL in MacOS Key: GOBBLIN-1065 URL: https://issues.apache.org/jira/browse/GOBBLIN-1065 Project: Apache Gobblin Issue Type: Bug Reporter: Lei Sun -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (GOBBLIN-1064) Add writer's schema to workUnitState
Zihan Li created GOBBLIN-1064: - Summary: Add writer's schema to workUnitState Key: GOBBLIN-1064 URL: https://issues.apache.org/jira/browse/GOBBLIN-1064 Project: Apache Gobblin Issue Type: Task Reporter: Zihan Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1013) Gradle build fails on Windows on prepare_release_config
[
https://issues.apache.org/jira/browse/GOBBLIN-1013?focusedWorklogId=394589&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394589
]
ASF GitHub Bot logged work on GOBBLIN-1013:
---
Author: ASF GitHub Bot
Created on: 28/Feb/20 01:47
Start Date: 28/Feb/20 01:47
Worklog Time Spent: 10m
Work Description: asfgit commented on pull request #2859: [GOBBLIN-1013]
Fix prepare_release_config build step on Windows
URL: https://github.com/apache/incubator-gobblin/pull/2859
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394589)
Time Spent: 40m (was: 0.5h)
> Gradle build fails on Windows on prepare_release_config
> ---
>
> Key: GOBBLIN-1013
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1013
> Project: Apache Gobblin
> Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Alex Prokofiev
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {code:java}
> > Task :prepare_release_config FAILEDFAILURE: Build failed with an
> > exception.* What went wrong:
> Failed to capture snapshot of output files for task ':prepare_release_config'
> property 'destinationDir' during up-to-date check.
> > Failed to create MD5 hash for file content.
> {code}
> This seems to be a Gradle on Windows problem that has a workaround -
> [https://github.com/gradle/gradle/issues/4663]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2859: [GOBBLIN-1013] Fix prepare_release_config build step on Windows
asfgit closed pull request #2859: [GOBBLIN-1013] Fix prepare_release_config build step on Windows URL: https://github.com/apache/incubator-gobblin/pull/2859 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1053) Improve and update Gobblin readme
[ https://issues.apache.org/jira/browse/GOBBLIN-1053?focusedWorklogId=394588&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394588 ] ASF GitHub Bot logged work on GOBBLIN-1053: --- Author: ASF GitHub Bot Created on: 28/Feb/20 01:44 Start Date: 28/Feb/20 01:44 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2893: [GOBBLIN-1053] Update Readme with new badges and links URL: https://github.com/apache/incubator-gobblin/pull/2893 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394588) Time Spent: 0.5h (was: 20m) > Improve and update Gobblin readme > - > > Key: GOBBLIN-1053 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1053 > Project: Apache Gobblin > Issue Type: Task >Reporter: Alex Prokofiev >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2893: [GOBBLIN-1053] Update Readme with new badges and links
asfgit closed pull request #2893: [GOBBLIN-1053] Update Readme with new badges and links URL: https://github.com/apache/incubator-gobblin/pull/2893 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1013) Gradle build fails on Windows on prepare_release_config
[
https://issues.apache.org/jira/browse/GOBBLIN-1013?focusedWorklogId=394534&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394534
]
ASF GitHub Bot logged work on GOBBLIN-1013:
---
Author: ASF GitHub Bot
Created on: 27/Feb/20 23:51
Start Date: 27/Feb/20 23:51
Worklog Time Spent: 10m
Work Description: autumnust commented on issue #2859: [GOBBLIN-1013] Fix
prepare_release_config build step on Windows
URL:
https://github.com/apache/incubator-gobblin/pull/2859#issuecomment-592236391
> The build failed because some package could not be downloaded from
repo.maven.apache.org, possibly because of the rate limits. The problem is
unrelated to PR.
>
> However, I don't see how I can restart it.
An dirty way is to modify your last commit (message) and re-trigger it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394534)
Time Spent: 0.5h (was: 20m)
> Gradle build fails on Windows on prepare_release_config
> ---
>
> Key: GOBBLIN-1013
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1013
> Project: Apache Gobblin
> Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Alex Prokofiev
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {code:java}
> > Task :prepare_release_config FAILEDFAILURE: Build failed with an
> > exception.* What went wrong:
> Failed to capture snapshot of output files for task ':prepare_release_config'
> property 'destinationDir' during up-to-date check.
> > Failed to create MD5 hash for file content.
> {code}
> This seems to be a Gradle on Windows problem that has a workaround -
> [https://github.com/gradle/gradle/issues/4663]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] autumnust commented on issue #2859: [GOBBLIN-1013] Fix prepare_release_config build step on Windows
autumnust commented on issue #2859: [GOBBLIN-1013] Fix prepare_release_config build step on Windows URL: https://github.com/apache/incubator-gobblin/pull/2859#issuecomment-592236391 > The build failed because some package could not be downloaded from repo.maven.apache.org, possibly because of the rate limits. The problem is unrelated to PR. > > However, I don't see how I can restart it. An dirty way is to modify your last commit (message) and re-trigger it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1063) add some more info to logs
[ https://issues.apache.org/jira/browse/GOBBLIN-1063?focusedWorklogId=394529&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394529 ] ASF GitHub Bot logged work on GOBBLIN-1063: --- Author: ASF GitHub Bot Created on: 27/Feb/20 23:29 Start Date: 27/Feb/20 23:29 Worklog Time Spent: 10m Work Description: codecov-io commented on issue #2902: [GOBBLIN-1063] add log URL: https://github.com/apache/incubator-gobblin/pull/2902#issuecomment-592230407 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=h1) Report > Merging [#2902](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/166eb725c01c279068aaefa2c9af93c907bd9eff?src=pr&el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2902 +/- ## + Coverage 45.81% 45.85% +0.03% - Complexity 9148 9168 +20 Files 1929 1932 +3 Lines 7260272795 +193 Branches 7993 8022 +29 + Hits 3326433381 +117 - Misses3630336355 +52 - Partials 3035 3059 +24 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...nt/copy/writer/FileAwareInputStreamDataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvd3JpdGVyL0ZpbGVBd2FyZUlucHV0U3RyZWFtRGF0YVdyaXRlci5qYXZh) | `73.19% <0%> (ø)` | `36 <0> (ø)` | :arrow_down: | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-4.22%)` | `14% <0%> (-1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecConsumer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY0NvbnN1bWVyLmphdmE=) | `60% <0%> (-3.83%)` | `6% <0%> (+1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY1Byb2R1Y2VyLmphdmE=) | `72.54% <0%> (-3.07%)` | `9% <0%> (+1%)` | | | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...he/gobblin/writer/FineGrainedWatermarkTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0ZpbmVHcmFpbmVkV2F0ZXJtYXJrVHJhY2tlci5qYXZh) | `82.25% <0%> (-2.42%)` | `28% <0%> (-1%)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `68.87% <0%> (-1.68%)` | `10% <0%> (ø)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | | | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?sr
[GitHub] [incubator-gobblin] codecov-io commented on issue #2902: [GOBBLIN-1063] add log
codecov-io commented on issue #2902: [GOBBLIN-1063] add log URL: https://github.com/apache/incubator-gobblin/pull/2902#issuecomment-592230407 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=h1) Report > Merging [#2902](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gobblin/commit/166eb725c01c279068aaefa2c9af93c907bd9eff?src=pr&el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master#2902 +/- ## + Coverage 45.81% 45.85% +0.03% - Complexity 9148 9168 +20 Files 1929 1932 +3 Lines 7260272795 +193 Branches 7993 8022 +29 + Hits 3326433381 +117 - Misses3630336355 +52 - Partials 3035 3059 +24 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...nt/copy/writer/FileAwareInputStreamDataWriter.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvd3JpdGVyL0ZpbGVBd2FyZUlucHV0U3RyZWFtRGF0YVdyaXRlci5qYXZh) | `73.19% <0%> (ø)` | `36 <0> (ø)` | :arrow_down: | | [...lin/util/filesystem/FileSystemInstrumentation.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi11dGlsaXR5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3V0aWwvZmlsZXN5c3RlbS9GaWxlU3lzdGVtSW5zdHJ1bWVudGF0aW9uLmphdmE=) | `85.71% <0%> (-7.15%)` | `3% <0%> (ø)` | | | [...in/java/org/apache/gobblin/cluster/HelixUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvSGVsaXhVdGlscy5qYXZh) | `38.26% <0%> (-4.22%)` | `14% <0%> (-1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecConsumer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY0NvbnN1bWVyLmphdmE=) | `60% <0%> (-3.83%)` | `6% <0%> (+1%)` | | | [...org/apache/gobblin/runtime/api/FsSpecProducer.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0ZzU3BlY1Byb2R1Y2VyLmphdmE=) | `72.54% <0%> (-3.07%)` | `9% <0%> (+1%)` | | | [...gobblin/azkaban/AzkabanGobblinYarnAppLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tYXprYWJhbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZ29iYmxpbi9hemthYmFuL0F6a2FiYW5Hb2JibGluWWFybkFwcExhdW5jaGVyLmphdmE=) | `30.55% <0%> (-2.78%)` | `2% <0%> (ø)` | | | [...he/gobblin/writer/FineGrainedWatermarkTracker.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1jb3JlLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vd3JpdGVyL0ZpbmVHcmFpbmVkV2F0ZXJtYXJrVHJhY2tlci5qYXZh) | `82.25% <0%> (-2.42%)` | `28% <0%> (-1%)` | | | [...anagement/copy/replication/ConfigBasedDataset.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1kYXRhLW1hbmFnZW1lbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vZGF0YS9tYW5hZ2VtZW50L2NvcHkvcmVwbGljYXRpb24vQ29uZmlnQmFzZWREYXRhc2V0LmphdmE=) | `68.87% <0%> (-1.68%)` | `10% <0%> (ø)` | | | [.../apache/gobblin/runtime/api/JobExecutionState.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1ydW50aW1lL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3J1bnRpbWUvYXBpL0pvYkV4ZWN1dGlvblN0YXRlLmphdmE=) | `79.43% <0%> (-0.94%)` | `24% <0%> (ø)` | | | [...pache/gobblin/cluster/GobblinHelixJobLauncher.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree#diff-Z29iYmxpbi1jbHVzdGVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NsdXN0ZXIvR29iYmxpbkhlbGl4Sm9iTGF1bmNoZXIuamF2YQ==) | `81.28% <0%> (-0.72%)` | `27% <0%> (ø)` | | | ... and [25 more](https://codecov.io/gh/apache/incubator-gobblin/pull/2902/diff?src=pr&el=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2902?src=pr&el=continue
[jira] [Work logged] (GOBBLIN-1063) add some more info to logs
[ https://issues.apache.org/jira/browse/GOBBLIN-1063?focusedWorklogId=394519&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394519 ] ASF GitHub Bot logged work on GOBBLIN-1063: --- Author: ASF GitHub Bot Created on: 27/Feb/20 23:04 Start Date: 27/Feb/20 23:04 Worklog Time Spent: 10m Work Description: yukuai518 commented on issue #2902: [GOBBLIN-1063] add log URL: https://github.com/apache/incubator-gobblin/pull/2902#issuecomment-59642 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394519) Time Spent: 20m (was: 10m) > add some more info to logs > -- > > Key: GOBBLIN-1063 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1063 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Arjun Singh Bora >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] yukuai518 commented on issue #2902: [GOBBLIN-1063] add log
yukuai518 commented on issue #2902: [GOBBLIN-1063] add log URL: https://github.com/apache/incubator-gobblin/pull/2902#issuecomment-59642 +1 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Work logged] (GOBBLIN-1063) add some more info to logs
[
https://issues.apache.org/jira/browse/GOBBLIN-1063?focusedWorklogId=394514&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394514
]
ASF GitHub Bot logged work on GOBBLIN-1063:
---
Author: ASF GitHub Bot
Created on: 27/Feb/20 22:56
Start Date: 27/Feb/20 22:56
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on pull request #2902:
[GOBBLIN-1063] add log
URL: https://github.com/apache/incubator-gobblin/pull/2902
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1063
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
add some more info to logs
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
trivial changes
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394514)
Remaining Estimate: 0h
Time Spent: 10m
> add some more info to logs
> --
>
> Key: GOBBLIN-1063
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1063
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (GOBBLIN-1063) add some more info to logs
Arjun Singh Bora created GOBBLIN-1063: - Summary: add some more info to logs Key: GOBBLIN-1063 URL: https://issues.apache.org/jira/browse/GOBBLIN-1063 Project: Apache Gobblin Issue Type: Improvement Reporter: Arjun Singh Bora -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 opened a new pull request #2902: [GOBBLIN-1063] add log
arjun4084346 opened a new pull request #2902: [GOBBLIN-1063] add log
URL: https://github.com/apache/incubator-gobblin/pull/2902
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [x] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1063
### Description
- [x] Here are some details about my PR, including screenshots (if
applicable):
add some more info to logs
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
trivial changes
### Commits
- [x] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1050) Verify requester when updating/deleting FlowConfig
[ https://issues.apache.org/jira/browse/GOBBLIN-1050?focusedWorklogId=394508&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394508 ] ASF GitHub Bot logged work on GOBBLIN-1050: --- Author: ASF GitHub Bot Created on: 27/Feb/20 22:24 Start Date: 27/Feb/20 22:24 Worklog Time Spent: 10m Work Description: arjun4084346 commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig URL: https://github.com/apache/incubator-gobblin/pull/2890#issuecomment-592208783 +1 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394508) Time Spent: 1h 20m (was: 1h 10m) > Verify requester when updating/deleting FlowConfig > -- > > Key: GOBBLIN-1050 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1050 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig
arjun4084346 commented on issue #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig URL: https://github.com/apache/incubator-gobblin/pull/2890#issuecomment-592208783 +1 LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-gobblin] jack-moseley commented on a change in pull request #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig
jack-moseley commented on a change in pull request #2890: [GOBBLIN-1050] Verify
requester when updating/deleting FlowConfig
URL: https://github.com/apache/incubator-gobblin/pull/2890#discussion_r385403224
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigsResource.java
##
@@ -124,12 +124,38 @@ public UpdateResponse update(ComplexResourceKey key, FlowCo
*/
@Override
public UpdateResponse delete(ComplexResourceKey key) {
+checkRequester(get(key), this.requesterService.findRequesters(this));
String flowGroup = key.getKey().getFlowGroup();
String flowName = key.getKey().getFlowName();
FlowId flowId = new FlowId().setFlowGroup(flowGroup).setFlowName(flowName);
return this.flowConfigsResourceHandler.deleteFlowConfig(flowId,
getHeaders());
}
+ /**
+ * Check that all {@link ServiceRequester}s in this request are contained
within the original service requester list
+ * when the flow was submitted and throw a {@link FlowConfigLoggedException}
if they are not.
+ *
+ * @param originalFlowConfig original flow config to find original requester
+ * @param requesterList list of requesters for this request
+ */
+ public static void checkRequester(FlowConfig originalFlowConfig,
List requesterList) {
Review comment:
I think I still prefer throwing exception in the method to avoid having to
copy the throwing exception code to five places. I updated the javadoc to make
the exceptions thrown more clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1050) Verify requester when updating/deleting FlowConfig
[
https://issues.apache.org/jira/browse/GOBBLIN-1050?focusedWorklogId=394501&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394501
]
ASF GitHub Bot logged work on GOBBLIN-1050:
---
Author: ASF GitHub Bot
Created on: 27/Feb/20 22:15
Start Date: 27/Feb/20 22:15
Worklog Time Spent: 10m
Work Description: jack-moseley commented on pull request #2890:
[GOBBLIN-1050] Verify requester when updating/deleting FlowConfig
URL: https://github.com/apache/incubator-gobblin/pull/2890#discussion_r385403224
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigsResource.java
##
@@ -124,12 +124,38 @@ public UpdateResponse update(ComplexResourceKey key, FlowCo
*/
@Override
public UpdateResponse delete(ComplexResourceKey key) {
+checkRequester(get(key), this.requesterService.findRequesters(this));
String flowGroup = key.getKey().getFlowGroup();
String flowName = key.getKey().getFlowName();
FlowId flowId = new FlowId().setFlowGroup(flowGroup).setFlowName(flowName);
return this.flowConfigsResourceHandler.deleteFlowConfig(flowId,
getHeaders());
}
+ /**
+ * Check that all {@link ServiceRequester}s in this request are contained
within the original service requester list
+ * when the flow was submitted and throw a {@link FlowConfigLoggedException}
if they are not.
+ *
+ * @param originalFlowConfig original flow config to find original requester
+ * @param requesterList list of requesters for this request
+ */
+ public static void checkRequester(FlowConfig originalFlowConfig,
List requesterList) {
Review comment:
I think I still prefer throwing exception in the method to avoid having to
copy the throwing exception code to five places. I updated the javadoc to make
the exceptions thrown more clear.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394501)
Time Spent: 1h 10m (was: 1h)
> Verify requester when updating/deleting FlowConfig
> --
>
> Key: GOBBLIN-1050
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1050
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1050) Verify requester when updating/deleting FlowConfig
[
https://issues.apache.org/jira/browse/GOBBLIN-1050?focusedWorklogId=394490&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394490
]
ASF GitHub Bot logged work on GOBBLIN-1050:
---
Author: ASF GitHub Bot
Created on: 27/Feb/20 22:03
Start Date: 27/Feb/20 22:03
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on pull request #2890:
[GOBBLIN-1050] Verify requester when updating/deleting FlowConfig
URL: https://github.com/apache/incubator-gobblin/pull/2890#discussion_r385397444
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigsResource.java
##
@@ -124,12 +124,38 @@ public UpdateResponse update(ComplexResourceKey key, FlowCo
*/
@Override
public UpdateResponse delete(ComplexResourceKey key) {
+checkRequester(get(key), this.requesterService.findRequesters(this));
String flowGroup = key.getKey().getFlowGroup();
String flowName = key.getKey().getFlowName();
FlowId flowId = new FlowId().setFlowGroup(flowGroup).setFlowName(flowName);
return this.flowConfigsResourceHandler.deleteFlowConfig(flowId,
getHeaders());
}
+ /**
+ * Check that all {@link ServiceRequester}s in this request are contained
within the original service requester list
+ * when the flow was submitted and throw a {@link FlowConfigLoggedException}
if they are not.
+ *
+ * @param originalFlowConfig original flow config to find original requester
+ * @param requesterList list of requesters for this request
+ */
+ public static void checkRequester(FlowConfig originalFlowConfig,
List requesterList) {
Review comment:
I think this method should return a boolean.
Or at least javadoc needs to have information on what it throws and what
that means.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 394490)
Time Spent: 1h (was: 50m)
> Verify requester when updating/deleting FlowConfig
> --
>
> Key: GOBBLIN-1050
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1050
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [incubator-gobblin] arjun4084346 commented on a change in pull request #2890: [GOBBLIN-1050] Verify requester when updating/deleting FlowConfig
arjun4084346 commented on a change in pull request #2890: [GOBBLIN-1050] Verify
requester when updating/deleting FlowConfig
URL: https://github.com/apache/incubator-gobblin/pull/2890#discussion_r385397444
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigsResource.java
##
@@ -124,12 +124,38 @@ public UpdateResponse update(ComplexResourceKey key, FlowCo
*/
@Override
public UpdateResponse delete(ComplexResourceKey key) {
+checkRequester(get(key), this.requesterService.findRequesters(this));
String flowGroup = key.getKey().getFlowGroup();
String flowName = key.getKey().getFlowName();
FlowId flowId = new FlowId().setFlowGroup(flowGroup).setFlowName(flowName);
return this.flowConfigsResourceHandler.deleteFlowConfig(flowId,
getHeaders());
}
+ /**
+ * Check that all {@link ServiceRequester}s in this request are contained
within the original service requester list
+ * when the flow was submitted and throw a {@link FlowConfigLoggedException}
if they are not.
+ *
+ * @param originalFlowConfig original flow config to find original requester
+ * @param requesterList list of requesters for this request
+ */
+ public static void checkRequester(FlowConfig originalFlowConfig,
List requesterList) {
Review comment:
I think this method should return a boolean.
Or at least javadoc needs to have information on what it throws and what
that means.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Work logged] (GOBBLIN-1062) Add log when loading dags from state store
[ https://issues.apache.org/jira/browse/GOBBLIN-1062?focusedWorklogId=394385&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-394385 ] ASF GitHub Bot logged work on GOBBLIN-1062: --- Author: ASF GitHub Bot Created on: 27/Feb/20 18:35 Start Date: 27/Feb/20 18:35 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #2901: [GOBBLIN-1062] Add log when loading dags from state store URL: https://github.com/apache/incubator-gobblin/pull/2901 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 394385) Time Spent: 50m (was: 40m) > Add log when loading dags from state store > -- > > Key: GOBBLIN-1062 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1062 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [incubator-gobblin] asfgit closed pull request #2901: [GOBBLIN-1062] Add log when loading dags from state store
asfgit closed pull request #2901: [GOBBLIN-1062] Add log when loading dags from state store URL: https://github.com/apache/incubator-gobblin/pull/2901 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
