[jira] [Updated] (HADOOP-16728) SFTPFileSystem connection pooling is not handling the pool properly
[ https://issues.apache.org/jira/browse/HADOOP-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Perroud updated HADOOP-16728: Description: This issue is trying to address few issues in the SFTP filesystem, and especially in SFTPConnectionPool: * all channels for a given connectionInfo are removed from the pool instead of only the selected channel. * As a result, if you have 2 threads sharing the pool, the first one acquiring a channels removes all the others, leading to inefficient SSH reconnections. was: In SFTPConnectionPool::getFromPool, all channels for a given connectionInfo are removed from the pool instead of only the selected channel. As a result, if you have 2 threads sharing the pool, the first one acquiring a channels removes all the others, leading to inefficient SSH reconnections. > SFTPFileSystem connection pooling is not handling the pool properly > --- > > Key: HADOOP-16728 > URL: https://issues.apache.org/jira/browse/HADOOP-16728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: Benoit Perroud >Priority: Major > > This issue is trying to address few issues in the SFTP filesystem, and > especially in SFTPConnectionPool: > * all channels for a given connectionInfo are removed from the pool instead > of only the selected channel. > * > As a result, if you have 2 threads sharing the pool, the first one acquiring > a channels removes all the others, leading to inefficient SSH reconnections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16728) SFTPFileSystem connection pooling is not handling the pool properly
[ https://issues.apache.org/jira/browse/HADOOP-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Perroud updated HADOOP-16728: Summary: SFTPFileSystem connection pooling is not handling the pool properly (was: SFTPConnectionPool is wrongly removing channels from the pool) > SFTPFileSystem connection pooling is not handling the pool properly > --- > > Key: HADOOP-16728 > URL: https://issues.apache.org/jira/browse/HADOOP-16728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: Benoit Perroud >Priority: Major > > In SFTPConnectionPool::getFromPool, all channels for a given connectionInfo > are removed from the pool instead of only the selected channel. > As a result, if you have 2 threads sharing the pool, the first one acquiring > a channels removes all the others, leading to inefficient SSH reconnections. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] c-w commented on issue #452: HADOOP-16005: Add XAttr support to WASB and ABFS
c-w commented on issue #452: HADOOP-16005: Add XAttr support to WASB and ABFS URL: https://github.com/apache/hadoop/pull/452#issuecomment-558924209 Thanks for the help, @DadanielZ. I was able to run the tests by changing the ABFS endpoint to dfs as opposed to blob as you suggested. All test now pass for the branch except for one test which also fails on trunk: [comparison](https://user-images.githubusercontent.com/1086421/69693891-efad1480-10a4-11ea-923c-7a2c9d497e3b.png), [branch log](https://github.com/apache/hadoop/files/3895120/test.log.txt), [trunk log](https://github.com/apache/hadoop/files/3895121/test.log.txt). Let me know if this is sufficient for the purposes of this pull request or if there's further testing/verification you'd like me to perform. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
[ https://issues.apache.org/jira/browse/HADOOP-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar resolved HADOOP-16685. --- Fix Version/s: 3.3.0 Resolution: Fixed > FileSystem#listStatusIterator does not check if given path exists > - > > Key: HADOOP-16685 > URL: https://issues.apache.org/jira/browse/HADOOP-16685 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Fix For: 3.3.0 > > > The Javadocs of FileSystem#listStatusIterator(final Path p) state that it > "@throws FileNotFoundException if p does not exist". However, > that does not seem to be the case. The method simply creates a > DirListingIterator which doesn't do an existence check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16983034#comment-16983034
]
Hadoop QA commented on HADOOP-16005:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
57s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/452 |
| JIRA Issue | HADOOP-16005 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9e270c2aaaf6 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ef950b0 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/console |
| versions | git=2.7.4 maven=3.3.9 fin
[GitHub] [hadoop] hadoop-yetus commented on issue #452: HADOOP-16005: Add XAttr support to WASB and ABFS
hadoop-yetus commented on issue #452: HADOOP-16005: Add XAttr support to WASB and ABFS URL: https://github.com/apache/hadoop/pull/452#issuecomment-558875953 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 3m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 23m 59s | trunk passed | | +1 :green_heart: | compile | 0m 30s | trunk passed | | +1 :green_heart: | checkstyle | 0m 23s | trunk passed | | +1 :green_heart: | mvnsite | 0m 33s | trunk passed | | +1 :green_heart: | shadedclient | 15m 47s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 36s | trunk passed | | +0 :ok: | spotbugs | 0m 57s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 55s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 30s | the patch passed | | +1 :green_heart: | compile | 0m 22s | the patch passed | | +1 :green_heart: | javac | 0m 22s | the patch passed | | +1 :green_heart: | checkstyle | 0m 21s | the patch passed | | +1 :green_heart: | mvnsite | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 15m 7s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 23s | the patch passed | | +1 :green_heart: | findbugs | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 18s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. | | | | 67m 37s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/452 | | JIRA Issue | HADOOP-16005 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9e270c2aaaf6 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ef950b0 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/testReport/ | | Max. process+thread count | 305 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-452/2/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] DadanielZ commented on issue #452: HADOOP-16005: Add XAttr support to WASB and ABFS
DadanielZ commented on issue #452: HADOOP-16005: Add XAttr support to WASB and ABFS URL: https://github.com/apache/hadoop/pull/452#issuecomment-558866258 @c-w the configuration for ABFS is not correct, the account name and related suffix should be "dfs.core.windows.net", not "blob.core.windows.net". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] c-w commented on issue #452: HADOOP-16005: Add XAttr support to WASB and ABFS
c-w commented on issue #452: HADOOP-16005: Add XAttr support to WASB and ABFS URL: https://github.com/apache/hadoop/pull/452#issuecomment-558863173 Apologies for the delay on this pull request. I resolved the merge conflicts and pushed an update. All the unit testes pass, but when running the integration tests with [my azure-auth-keys.xml file](https://user-images.githubusercontent.com/1086421/69680906-641e8e00-107a-11ea-9323-0c5276c71aa8.png) against StorageV2 accounts, I'm getting some failures. However, I'm also getting these failures when running against trunk so I doubt that the errors are related to this pull request. I ran the tests by executing `sudo mvn clean install -Pdist -Dtar -Ptest-patch -pl ./hadoop-tools/hadoop-azure | tee test.log.txt` in the development environment Docker container running on WSL: see [test.log.txt](https://github.com/apache/hadoop/files/3894410/test.log.txt) and a sample [detailed test failure](https://user-images.githubusercontent.com/1086421/69681049-cecfc980-107a-11ea-9234-cf21cd8dc00f.png). As a new contributor to this project, I'd appreciate any guidance on how to resolve the errors and properly integration test my change. Any ideas @DadanielZ? Thanks in advance for your help! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sapant-msft commented on issue #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads.
sapant-msft commented on issue #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads. URL: https://github.com/apache/hadoop/pull/1708#issuecomment-558851272 Hi @steveloughran , @DadanielZ , Thank you for your comments, I have tried to address all of them with my latest commits. Please kindly review again and let me know if you have any suggestions/ concerns. Thanks, Saurabh This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sapant-msft commented on a change in pull request #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads.
sapant-msft commented on a change in pull request #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads. URL: https://github.com/apache/hadoop/pull/1708#discussion_r351018848 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java ## @@ -51,6 +51,7 @@ public static final String FS_AZURE_ALWAYS_USE_HTTPS = "fs.azure.always.use.https"; public static final String FS_AZURE_ATOMIC_RENAME_KEY = "fs.azure.atomic.rename.key"; public static final String FS_AZURE_READ_AHEAD_QUEUE_DEPTH = "fs.azure.readaheadqueue.depth"; + public static final String FS_AZURE_ALWAYS_READ_AHEAD = "fs.azure.always.readahead"; Review comment: Fixed, thanks for pointing out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sapant-msft commented on a change in pull request #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads.
sapant-msft commented on a change in pull request #1708: HADOOP-16696: Always
read ahead config, to use read ahead even for non sequential reads.
URL: https://github.com/apache/hadoop/pull/1708#discussion_r351018883
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAbfsReadWriteAndSeekReadAheadEnabled.java
##
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs;
+
+import java.util.Arrays;
+import java.util.Random;
+
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+
+import static
org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.DEFAULT_READ_BUFFER_SIZE;
+import static
org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.MAX_BUFFER_SIZE;
+import static
org.apache.hadoop.fs.azurebfs.constants.FileSystemConfigurations.MIN_BUFFER_SIZE;
+
+/**
+ * Test read, write and seek.
+ * Uses package-private methods in AbfsConfiguration, which is why it is in
+ * this package.
+ */
+@RunWith(Parameterized.class)
+public class ITestAbfsReadWriteAndSeekReadAheadEnabled extends
AbstractAbfsScaleTest {
+ private static final Path TEST_PATH = new Path("/testfile");
+
+ @Parameterized.Parameters(name = "Size={0}")
+ public static Iterable sizes() {
+return Arrays.asList(new Object[][]{{MIN_BUFFER_SIZE},
+{DEFAULT_READ_BUFFER_SIZE},
+{MAX_BUFFER_SIZE}});
+ }
+
+ private final int size;
+
+ public ITestAbfsReadWriteAndSeekReadAheadEnabled(final int size) throws
Exception {
+this.size = size;
+ }
+
+ @Test
+ public void testReadAndWriteWithDifferentBufferSizesAndSeek() throws
Exception {
+testReadWriteAndSeek(size);
+ }
+
+ private void testReadWriteAndSeek(int bufferSize) throws Exception {
+final AzureBlobFileSystem fs = getFileSystem();
+final AbfsConfiguration abfsConfiguration =
fs.getAbfsStore().getAbfsConfiguration();
+
+abfsConfiguration.setWriteBufferSize(bufferSize);
+abfsConfiguration.setReadBufferSize(bufferSize);
+// enabling AlwaysReadAhead
+abfsConfiguration.setAlwaysReadAhead(true);
+
+final byte[] b = new byte[2 * bufferSize];
+new Random().nextBytes(b);
+try (FSDataOutputStream stream = fs.create(TEST_PATH)) {
+ stream.write(b);
+}
+
+final byte[] readBufferForReadAhead = new byte[2 * bufferSize];
+int resultForReadAhead;
+try (FSDataInputStream inputStreamForReadAhead = fs.open(TEST_PATH)) {
+ inputStreamForReadAhead.seek(bufferSize);
+ resultForReadAhead =
inputStreamForReadAhead.read(readBufferForReadAhead, bufferSize, bufferSize);
+ assertNotEquals(-1, resultForReadAhead);
Review comment:
Fixed, thanks for pointing out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sapant-msft commented on a change in pull request #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads.
sapant-msft commented on a change in pull request #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads. URL: https://github.com/apache/hadoop/pull/1708#discussion_r351018807 ## File path: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsInputStream.java ## @@ -30,6 +30,7 @@ import org.apache.hadoop.fs.FileSystem.Statistics; import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException; import org.apache.hadoop.fs.azurebfs.contracts.exceptions.AzureBlobFileSystemException; +import org.apache.hadoop.fs.azurebfs.AbfsConfiguration; Review comment: Fixed, thanks for pointing out. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16696) Adding an option to Always use Read Ahead, even for non sequential reads
[
https://issues.apache.org/jira/browse/HADOOP-16696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982980#comment-16982980
]
Hadoop QA commented on HADOOP-16696:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
51s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1708/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1708 |
| JIRA Issue | HADOOP-16696 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 829e792c0573 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ef950b0 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1708/9/testReport/ |
| Max. process+thread count | 428 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1708/9/console |
| versions | git=2.7.4 maven=3.3.9
[GitHub] [hadoop] hadoop-yetus commented on issue #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads.
hadoop-yetus commented on issue #1708: HADOOP-16696: Always read ahead config, to use read ahead even for non sequential reads. URL: https://github.com/apache/hadoop/pull/1708#issuecomment-558849871 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 18m 3s | trunk passed | | +1 :green_heart: | compile | 0m 31s | trunk passed | | +1 :green_heart: | checkstyle | 0m 23s | trunk passed | | +1 :green_heart: | mvnsite | 0m 33s | trunk passed | | +1 :green_heart: | shadedclient | 13m 33s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 27s | trunk passed | | +0 :ok: | spotbugs | 0m 51s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 50s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 29s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed | | +1 :green_heart: | javac | 0m 23s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 12s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 24s | the patch passed | | +1 :green_heart: | findbugs | 0m 52s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 22s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | | | | 54m 18s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1708/9/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1708 | | JIRA Issue | HADOOP-16696 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 829e792c0573 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / ef950b0 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1708/9/testReport/ | | Max. process+thread count | 428 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1708/9/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16730) ABFS: Support for Shared Access Signatures (SAS)
Thomas Marqardt created HADOOP-16730: Summary: ABFS: Support for Shared Access Signatures (SAS) Key: HADOOP-16730 URL: https://issues.apache.org/jira/browse/HADOOP-16730 Project: Hadoop Common Issue Type: New Feature Components: fs/azure Affects Versions: 3.2.1 Reporter: Thomas Marqardt Assignee: Sneha Vijayarajan ABFS supports OAuth and Shared Key but currently lacks support for [Shared Access Signatures (SAS)|[https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview]]. SAS is a great way to constrain access to a low-privilege ABFS client. The ABFS client does not need to possess persistent credentials for accessing storage but instead can request temporary, constrained access tokens from a trusted endpoint. This endpoint can authenticate the caller, make an authorization decision and return a constrained SAS token. The token may have an expiration, it may be scoped to a specific file or directory, and it may grant an action or set of actions such as read, write, list, or delete. Azure Storage also has a new identity based SAS scheme in preview named Delegation SAS. These new Delegation SAS have these advantages over Service SAS: 1) Delegation SAS provide authentication as well as authorization. The user identity associated with each request will appear in the logs when logging is enabled for the account. 2) Instead of using storage account keys to sign tokens, Delegation SAS relies on keys assigned to each user. These keys are called user delegation keys. If a storage account key is leaked, an attacker would have full access to the storage account. If a user delegation key is leaked, an attacker would only have access to resources that user has access to within the Blob service–for example, the user might only have read access to a specific container. This feature will add support for the ABFS driver to authenticate against a trusted endpoint. The endpoint will return a SAS which the ABFS driver will use to access Azure storage. The SAS may be a container or directory SAS to be used for all subsequent operations, and thus cached for the lifetime of the filesystem. Or it may be a SAS to be used for the current filesystem operation, in this case, the ABFS driver will request a SAS for each operation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982783#comment-16982783
]
Clemens Wolff edited comment on HADOOP-16005 at 11/26/19 7:24 PM:
--
[~virajith] [~ste...@apache.org] [~DanielZhou] Apologies for the delayed
response. I had to deprioritize this pull request for a while but am now
working again on this. I'm updating my pull request and fixing the merge
conflicts. Expect an update on Github soon.
was (Author: c-w):
[~virajith] [~ste...@apache.org] Apologies for the delayed response. I had to
deprioritize this pull request for a while but am now working again on this.
I'm updating my pull request and fixing the merge conflicts. Expect an update
on Github soon.
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16685) FileSystem#listStatusIterator does not check if given path exists
[ https://issues.apache.org/jira/browse/HADOOP-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982784#comment-16982784 ] Hudson commented on HADOOP-16685: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17696 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17696/]) HADOOP-16685: FileSystem#listStatusIterator does not check if given path (weichiu: rev 3161813482868e42befb618d6f5687d8ffed0e5c) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractGetFileStatusTest.java > FileSystem#listStatusIterator does not check if given path exists > - > > Key: HADOOP-16685 > URL: https://issues.apache.org/jira/browse/HADOOP-16685 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > > The Javadocs of FileSystem#listStatusIterator(final Path p) state that it > "@throws FileNotFoundException if p does not exist". However, > that does not seem to be the case. The method simply creates a > DirListingIterator which doesn't do an existence check. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16005) NativeAzureFileSystem does not support setXAttr
[
https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982783#comment-16982783
]
Clemens Wolff commented on HADOOP-16005:
[~virajith] [~ste...@apache.org] Apologies for the delayed response. I had to
deprioritize this pull request for a while but am now working again on this.
I'm updating my pull request and fixing the merge conflicts. Expect an update
on Github soon.
> NativeAzureFileSystem does not support setXAttr
> ---
>
> Key: HADOOP-16005
> URL: https://issues.apache.org/jira/browse/HADOOP-16005
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
>Reporter: Clemens Wolff
>Assignee: Clemens Wolff
>Priority: Major
>
> When interacting with Azure Blob Storage via the Hadoop FileSystem client,
> it's currently (as of
> [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b])
> not possible to set custom metadata attributes.
> Here is a snippet that demonstrates the missing behavior (throws an
> UnsupportedOperationException):
> {code:java}
> val blobAccount = "SET ME"
> val blobKey = "SET ME"
> val blobContainer = "SET ME"
> val blobFile = "SET ME"
> import org.apache.hadoop.conf.Configuration
> import org.apache.hadoop.fs.{FileSystem, Path}
> val conf = new Configuration()
> conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem")
> conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey)
> val path = new
> Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile")
> val fs = FileSystem.get(path, conf)
> fs.setXAttr(path, "somekey", "somevalue".getBytes)
> {code}
> Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem
> inherits the default setXAttr from FileSystem which throws the
> UnsupportedOperationException.
> The underlying Azure Blob Storage service does support custom metadata
> ([service
> docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata])
> as does the azure-storage SDK that's being used by NativeAzureFileSystem
> ([SDK
> docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]).
> Is there another way that I should be setting custom metadata on Azure Blob
> Storage files? Is there a specific reason why setXAttr hasn't been
> implemented on NativeAzureFileSystem? If not, I can take a shot at
> implementing it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists
jojochuang merged pull request #1695: HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists URL: https://github.com/apache/hadoop/pull/1695 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16346) Stabilize S3A OpenSSL support
[ https://issues.apache.org/jira/browse/HADOOP-16346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982724#comment-16982724 ] Steve Loughran commented on HADOOP-16346: - this is good > Stabilize S3A OpenSSL support > - > > Key: HADOOP-16346 > URL: https://issues.apache.org/jira/browse/HADOOP-16346 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Sahil Takiar >Priority: Blocker > > HADOOP-16050 switched S3A to trying to use OpenSSL. We need to make sure this > is stable, that people know it exists and aren't left wondering why things > which did work have now stopped. Which, given I know who will end up with > those support calls, is not something I want. > * Set the default back to the original JDK version. > * Document how to change this so you don't need to use an IDE to work out > what other values are allowed > * core-default.xml to include the default value and the text listing the > other options. > + anything else -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16346) Stabilize S3A OpenSSL support
[ https://issues.apache.org/jira/browse/HADOOP-16346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982667#comment-16982667 ] Sahil Takiar commented on HADOOP-16346: --- Good news. It looks like post Wildfly + AWS SDK upgrade IMPALA-8577 has been resolved. I can no longer reproduce the issue in Impala. Originally, I was able to re-produce it with just run one run of TPC-DS on an Impala ASAN build. After three runs, everything looks normal. Going to do some more Impala testing still, and looking into running some tests for MR + Spark as well. > Stabilize S3A OpenSSL support > - > > Key: HADOOP-16346 > URL: https://issues.apache.org/jira/browse/HADOOP-16346 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Sahil Takiar >Priority: Blocker > > HADOOP-16050 switched S3A to trying to use OpenSSL. We need to make sure this > is stable, that people know it exists and aren't left wondering why things > which did work have now stopped. Which, given I know who will end up with > those support calls, is not something I want. > * Set the default back to the original JDK version. > * Document how to change this so you don't need to use an IDE to work out > what other values are allowed > * core-default.xml to include the default value and the text listing the > other options. > + anything else -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml
hadoop-yetus commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730#issuecomment-558716216 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 33s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 21m 25s | trunk passed | | +1 :green_heart: | compile | 0m 15s | trunk passed | | +1 :green_heart: | mvnsite | 0m 18s | trunk passed | | +1 :green_heart: | shadedclient | 36m 10s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 17s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 11s | the patch passed | | +1 :green_heart: | compile | 0m 11s | the patch passed | | +1 :green_heart: | javac | 0m 11s | the patch passed | | +1 :green_heart: | mvnsite | 0m 14s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | +1 :green_heart: | shadedclient | 14m 55s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 14s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 13s | hadoop-project in the patch passed. | | +1 :green_heart: | asflicense | 0m 27s | The patch does not generate ASF License warnings. | | | | 56m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1730/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1730 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml | | uname | Linux 8b399f5e047c 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 52e9ee3 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1730/1/testReport/ | | Max. process+thread count | 308 (vs. ulimit of 5500) | | modules | C: hadoop-project U: hadoop-project | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1730/1/console | | versions | git=2.7.4 maven=3.3.9 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16709) S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota resolved HADOOP-16709. - Resolution: Fixed > S3Guard: Make authoritative mode exclusive for metadata - don't check for > expiry for authoritative paths > > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16709) S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982623#comment-16982623 ] Hudson commented on HADOOP-16709: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17694 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17694/]) HADOOP-16709. S3Guard: Make authoritative mode exclusive for metadata - (github: rev ea25f4de236611d388e14a710ebe5d6872c421b6) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/s3guard.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardOutOfBandOperations.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardTtl.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java > S3Guard: Make authoritative mode exclusive for metadata - don't check for > expiry for authoritative paths > > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16709) S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982609#comment-16982609 ] Gabor Bota commented on HADOOP-16709: - Got +1 on PR #1721 from [~ste...@apache.org]. Committing. > S3Guard: Make authoritative mode exclusive for metadata - don't check for > expiry for authoritative paths > > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16709) [S3Guard] Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16709: Summary: [S3Guard] Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths (was: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths) > [S3Guard] Make authoritative mode exclusive for metadata - don't check for > expiry for authoritative paths > - > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16709) S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16709: Summary: S3Guard: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths (was: [S3Guard] Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths) > S3Guard: Make authoritative mode exclusive for metadata - don't check for > expiry for authoritative paths > > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg merged pull request #1721: HADOOP-16709. Consider having the ability to turn off TTL in S3Guard …
bgaborg merged pull request #1721: HADOOP-16709. Consider having the ability to turn off TTL in S3Guard … URL: https://github.com/apache/hadoop/pull/1721 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16709) Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths
[ https://issues.apache.org/jira/browse/HADOOP-16709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Bota updated HADOOP-16709: Summary: Make authoritative mode exclusive for metadata - don't check for expiry for authoritative paths (was: Consider having the ability to turn off TTL in S3Guard + Authoritative mode) > Make authoritative mode exclusive for metadata - don't check for expiry for > authoritative paths > --- > > Key: HADOOP-16709 > URL: https://issues.apache.org/jira/browse/HADOOP-16709 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Rajesh Balamohan >Assignee: Gabor Bota >Priority: Minor > > Authoritative mode has TTL which is set to 15 minutes by default. However, > there are cases when we know for sure that the data wouldn't be > changed/updated. > In certain cases, AppMaster ends up spending good amount of time in getSplits > due to TTL expiry. It would be great to have an option to disable TTL (or > specify as -1 when TTL shouldn't be checked). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg removed a comment on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml
bgaborg removed a comment on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730#issuecomment-558682244 LGTM. Any historical reason why we had this as versions and not variables? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bgaborg commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml
bgaborg commented on issue #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730#issuecomment-558682244 LGTM. Any historical reason why we had this as versions and not variables? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tamaashu opened a new pull request #1730: HADOOP-16729. Extract version numbers to head of pom.xml
tamaashu opened a new pull request #1730: HADOOP-16729. Extract version numbers to head of pom.xml URL: https://github.com/apache/hadoop/pull/1730 To be able to easily replace third-party dependency version numbers it would be useful to collect their version numbers at the top of the pom.xml. For many third-parties (e.g. slf4j, jetty, etc.) this is already done, but I would need the same for others. The change doesn't have any effect on the code or the build, no version numbers would be changed. ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16729) Extract version numbers to head of pom.xml
Tamas Penzes created HADOOP-16729: - Summary: Extract version numbers to head of pom.xml Key: HADOOP-16729 URL: https://issues.apache.org/jira/browse/HADOOP-16729 Project: Hadoop Common Issue Type: Improvement Components: build Reporter: Tamas Penzes Assignee: Tamas Penzes To be able to easily replace third-party dependency version numbers it would be useful to collect their version numbers at the top of the pom.xml. For many third-parties (e.g. slf4j, jetty, etc.) this is already done, but I would need the same for others. The change doesn't have any effect on the code or the build, no version numbers would be changed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on issue #1721: HADOOP-16709. Consider having the ability to turn off TTL in S3Guard …
steveloughran commented on issue #1721: HADOOP-16709. Consider having the ability to turn off TTL in S3Guard … URL: https://github.com/apache/hadoop/pull/1721#issuecomment-558662339 LGTM, +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1729: HADOOP-16539. ABFS: Add missing query parameter for getPathStatus
hadoop-yetus commented on issue #1729: HADOOP-16539. ABFS: Add missing query parameter for getPathStatus URL: https://github.com/apache/hadoop/pull/1729#issuecomment-558644499 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 26m 19s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 19m 15s | trunk passed | | +1 :green_heart: | compile | 0m 29s | trunk passed | | +1 :green_heart: | checkstyle | 0m 21s | trunk passed | | +1 :green_heart: | mvnsite | 0m 30s | trunk passed | | +1 :green_heart: | shadedclient | 12m 55s | branch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 26s | trunk passed | | +0 :ok: | spotbugs | 0m 53s | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 :green_heart: | findbugs | 0m 51s | trunk passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 0m 28s | the patch passed | | +1 :green_heart: | compile | 0m 23s | the patch passed | | +1 :green_heart: | javac | 0m 23s | the patch passed | | +1 :green_heart: | checkstyle | 0m 16s | the patch passed | | +1 :green_heart: | mvnsite | 0m 27s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | shadedclient | 13m 17s | patch has no errors when building and testing our client artifacts. | | +1 :green_heart: | javadoc | 0m 22s | the patch passed | | +1 :green_heart: | findbugs | 0m 53s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 14s | hadoop-azure in the patch passed. | | +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. | | | | 80m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1729/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1729 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9c42cce2373f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 448ffb1 | | Default Java | 1.8.0_222 | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1729/1/testReport/ | | Max. process+thread count | 411 (vs. ulimit of 5500) | | modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1729/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] adamantal opened a new pull request #1729: HADOOP-16539. ABFS: Add missing query parameter for getPathStatus
adamantal opened a new pull request #1729: HADOOP-16539. ABFS: Add missing query parameter for getPathStatus URL: https://github.com/apache/hadoop/pull/1729 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16539) ABFS: Add missing query parameter for getPathStatus
[ https://issues.apache.org/jira/browse/HADOOP-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982370#comment-16982370 ] Steve Loughran commented on HADOOP-16539: - # github PR please # and remember the test policy: declare the abfs endpoint you ran all the tests against > ABFS: Add missing query parameter for getPathStatus > --- > > Key: HADOOP-16539 > URL: https://issues.apache.org/jira/browse/HADOOP-16539 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16539.001.patch > > > When calling > [getPathStatus|https://github.com/apache/hadoop/blob/e220dac15cc9972ebdd54ea9c82f288f234fca51/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L356], > query parameter "action=getStatus" is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16539) ABFS: Add missing query parameter for getPathStatus
[
https://issues.apache.org/jira/browse/HADOOP-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982303#comment-16982303
]
Hadoop QA commented on HADOOP-16539:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 33m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
18s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:104ccca9169 |
| JIRA Issue | HADOOP-16539 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12986754/HADOOP-16539.001.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 034435fa4905 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6c20512 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16681/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/16681/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> ABFS: Add missing query parameter for g
[jira] [Updated] (HADOOP-16539) ABFS: Add missing query parameter for getPathStatus
[ https://issues.apache.org/jira/browse/HADOOP-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16539: Status: Patch Available (was: Open) > ABFS: Add missing query parameter for getPathStatus > --- > > Key: HADOOP-16539 > URL: https://issues.apache.org/jira/browse/HADOOP-16539 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16539.001.patch > > > When calling > [getPathStatus|https://github.com/apache/hadoop/blob/e220dac15cc9972ebdd54ea9c82f288f234fca51/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L356], > query parameter "action=getStatus" is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16539) ABFS: Add missing query parameter for getPathStatus
[ https://issues.apache.org/jira/browse/HADOOP-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal updated HADOOP-16539: Attachment: HADOOP-16539.001.patch > ABFS: Add missing query parameter for getPathStatus > --- > > Key: HADOOP-16539 > URL: https://issues.apache.org/jira/browse/HADOOP-16539 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Adam Antal >Priority: Major > Attachments: HADOOP-16539.001.patch > > > When calling > [getPathStatus|https://github.com/apache/hadoop/blob/e220dac15cc9972ebdd54ea9c82f288f234fca51/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L356], > query parameter "action=getStatus" is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16539) ABFS: Add missing query parameter for getPathStatus
[ https://issues.apache.org/jira/browse/HADOOP-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Antal reassigned HADOOP-16539: --- Assignee: Adam Antal > ABFS: Add missing query parameter for getPathStatus > --- > > Key: HADOOP-16539 > URL: https://issues.apache.org/jira/browse/HADOOP-16539 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Affects Versions: 3.2.0 >Reporter: Da Zhou >Assignee: Adam Antal >Priority: Major > > When calling > [getPathStatus|https://github.com/apache/hadoop/blob/e220dac15cc9972ebdd54ea9c82f288f234fca51/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsClient.java#L356], > query parameter "action=getStatus" is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snvijaya edited a comment on issue #1712: HADOOP-16699: Adding verbose TRACE logging
snvijaya edited a comment on issue #1712: HADOOP-16699: Adding verbose TRACE logging URL: https://github.com/apache/hadoop/pull/1712#issuecomment-557502027 @steveloughran - As mentioned in my comments above, please let me know your opinion if the log verbosity is undesired even though its at log level TRACE and is mostly into the initialize flow where its hard to debug any hung scenario. Test results with the latest changes: On an account without Namespace enabled: [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 395, Failures: 0, Errors: 0, Skipped: 211 [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 126 On an account with Namespace enabled: [INFO] Tests run: 52, Failures: 0, Errors: 0, Skipped: 0 [WARNING] Tests run: 395, Failures: 0, Errors: 0, Skipped: 25 [WARNING] Tests run: 192, Failures: 0, Errors: 0, Skipped: 126 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
