[jira] [Resolved] (HADOOP-18592) Sasl connection failure should log remote address
[ https://issues.apache.org/jira/browse/HADOOP-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-18592. Fix Version/s: 3.4.0 3.3.9 Hadoop Flags: Reviewed Resolution: Fixed Committed to branch-3.3 and trunk branches. Thank you for your contribution, [~vjasani] . Thank you all for review. > Sasl connection failure should log remote address > - > > Key: HADOOP-18592 > URL: https://issues.apache.org/jira/browse/HADOOP-18592 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > If Sasl connection fails with some generic error, we miss logging remote > server that the client was trying to connect to. > Sample log: > {code:java} > 2023-01-12 00:22:28,148 WARN [20%2C1673404849949,1] ipc.Client - Exception > encountered while connecting to the server > java.io.IOException: Connection reset by peer > at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > at sun.nio.ch.IOUtil.read(IOUtil.java:197) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > at > org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) > at > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) > at java.io.FilterInputStream.read(FilterInputStream.java:133) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read(BufferedInputStream.java:265) > at java.io.DataInputStream.readInt(DataInputStream.java:387) > at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367) > at > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623) > at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414) > ... > ... {code} > We should log the remote server address. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17728) Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp
[ https://issues.apache.org/jira/browse/HADOOP-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-17728: > Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp > - > > Key: HADOOP-17728 > URL: https://issues.apache.org/jira/browse/HADOOP-17728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: yikf >Assignee: yikf >Priority: Minor > Labels: pull-request-available, reverted > Time Spent: 5h 10m > Remaining Estimate: 0h > > Cleaner thread will be blocked if we remove reference from ReferenceQueue > unless the `queue.enqueue` called. > > As shown below, We call ReferenceQueue.remove() now while cleanUp, Call > chain as follow: > *StatisticsDataReferenceCleaner#queue.remove() -> > ReferenceQueue.remove(0) -> lock.wait(0)* > But, lock.notifyAll is called when queue.enqueue only, so Cleaner thread > will be blocked. > > ThreadDump: > {code:java} > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7f7afc088800 > nid=0x2119 in Object.wait() [0x7f7b0023] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xc00c2f58> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:502) > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > - locked <0xc00c2f58> (a java.lang.ref.Reference$Lock) > at > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153){code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17728) Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp
[ https://issues.apache.org/jira/browse/HADOOP-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-17728. Fix Version/s: 3.2.3 3.4.0 3.3.1 Hadoop Flags: Reviewed Resolution: Fixed Committed to 3.1+ Thanks for your reporting and contribution [~kaifeiYi]. Thanks for your review [~ste...@apache.org] > Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp > - > > Key: HADOOP-17728 > URL: https://issues.apache.org/jira/browse/HADOOP-17728 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.2.1 >Reporter: yikf >Assignee: yikf >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.1, 3.4.0, 3.2.3 > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Cleaner thread will be blocked if we remove reference from ReferenceQueue > unless the `queue.enqueue` called. > > As shown below, We call ReferenceQueue.remove() now while cleanUp, Call > chain as follow: > *StatisticsDataReferenceCleaner#queue.remove() -> > ReferenceQueue.remove(0) -> lock.wait(0)* > But, lock.notifyAll is called when queue.enqueue only, so Cleaner thread > will be blocked. > > ThreadDump: > {code:java} > "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7f7afc088800 > nid=0x2119 in Object.wait() [0x7f7b0023] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xc00c2f58> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:502) > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > - locked <0xc00c2f58> (a java.lang.ref.Reference$Lock) > at > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153){code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16355) ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store
[ https://issues.apache.org/jira/browse/HADOOP-16355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-16355. Resolution: Abandoned As documented by [[HADOOP-17480]], AWS S3 is consistent and that S3Guard is not needed. > ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store > -- > > Key: HADOOP-16355 > URL: https://issues.apache.org/jira/browse/HADOOP-16355 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Mingliang Liu >Priority: Major > > When S3Guard was proposed, there are a couple of valid reasons to choose > DynamoDB as its default backend store: 0) seamless integration as part of AWS > ecosystem e.g. client library 1) it's a managed web service which is zero > operational cost, highly available and infinitely scalable 2) it's performant > with single digit latency 3) it's proven by Netflix's S3mper (not actively > maintained) and EMRFS (closed source and usage). As it's pluggable, it's > possible to implement {{MetadataStore}} with other backend store without > changing semantics, besides null and in-memory local ones. > Here we propose {{ZookeeperMetadataStore}} which uses Zookeeper as S3Guard > backend store. Its main motivation is to provide a new MetadataStore option > which: > # can be easily integrated as Zookeeper is heavily used in Hadoop community > # affordable performance as both client and Zookeeper ensemble are usually > "local" in a Hadoop cluster (ZK/HBase/Hive etc) > # removes DynamoDB dependency > Obviously all use cases will not prefer this to default DynamoDB store. For > e.g. ZK might not scale well if there are dozens of S3 buckets and each has > millions of objects. Our use case is targeting HBase to store HFiles on S3 > instead of HDFS. A total solution for HBase on S3 must be HBOSS (see > HBASE-22149) for recovering atomicity of metadata operations like rename, and > S3Guard for consistent enumeration and access to object store bucket > metadata. We would like to use Zookeeper as backend store for both. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17252) Website to link to latest Hadoop wiki
Mingliang Liu created HADOOP-17252: -- Summary: Website to link to latest Hadoop wiki Key: HADOOP-17252 URL: https://issues.apache.org/jira/browse/HADOOP-17252 Project: Hadoop Common Issue Type: Bug Reporter: Mingliang Liu Currently the website links to the [old wiki|https://wiki.apache.org/hadoop]. Shall we update that to the latest one: https://cwiki.apache.org/confluence/display/HADOOP2/Home -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17159) Make UGI support forceful relogin from keytab ignoring the last login time
[ https://issues.apache.org/jira/browse/HADOOP-17159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-17159. Fix Version/s: 2.10.1 Hadoop Flags: Reviewed Resolution: Fixed Committed to 2.10.1 and 3.1.5+ see "Fix Version/s". Thank you for your contribution, [~sandeep.guggilam] > Make UGI support forceful relogin from keytab ignoring the last login time > -- > > Key: HADOOP-17159 > URL: https://issues.apache.org/jira/browse/HADOOP-17159 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.10.0, 3.3.0, 3.2.1, 3.1.3 >Reporter: Sandeep Guggilam >Assignee: Sandeep Guggilam >Priority: Major > Fix For: 3.2.2, 2.10.1, 3.3.1, 3.4.0, 3.1.5 > > Time Spent: 40m > Remaining Estimate: 0h > > Currently we have a relogin() method in UGI which attempts to login if there > is no login attempted in the last 10 minutes or configured amount of time > We should also have provision for doing a forceful relogin irrespective of > the time window that the client can choose to use it if needed . Consider the > below scenario: > # SASL Server is reimaged and new keytabs are fetched with refreshing the > password > # SASL client connection to the server would fail when it tries with the > cached service ticket > # We should try to logout to clear the service tickets in cache and then try > to login back in such scenarios. But since the current relogin() doesn't > guarantee a login, it could cause an issue > # A forceful relogin in this case would help after logout > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17184) Add --mvn-custom-repos parameter to yetus calls
Mingliang Liu created HADOOP-17184: -- Summary: Add --mvn-custom-repos parameter to yetus calls Key: HADOOP-17184 URL: https://issues.apache.org/jira/browse/HADOOP-17184 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Mingliang Liu In my request PR [#2188|https://github.com/apache/hadoop/pull/2188], I see the QA build fails with unrelated errors. {code} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:2.5.1:install (default-install) on project hadoop-project: Failed to install metadata org.apache.hadoop:hadoop-project:3.4.0-SNAPSHOT/maven-metadata.xml: Could not parse metadata /home/jenkins/.m2/repository/org/apache/hadoop/hadoop-project/3.4.0-SNAPSHOT/maven-metadata-local.xml: in epilog non whitespace content is not allowed but got n (position: END_TAG seen ...\nn... @21:2) -> [Help 1] {code} As reported by HBASE-22474 and HBASE-22801, PreCommit validation from yetus uses a shared .m2 repository. By adding {{--mvn-custom-repos}} and {{--jenkins}} paramters, yetus will use a custom .m2 directory for executions for PR validations. This is a change to mimic that for Hadoop project. CC: [~aajisaka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17052) NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort
[ https://issues.apache.org/jira/browse/HADOOP-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-17052. Fix Version/s: 3.4.0 3.3.1 2.10.1 3.2.2 3.1.4 2.9.3 Hadoop Flags: Reviewed Resolution: Fixed Committed to all fixed versions branches. Thanks [~dhegde] for the reporting and providing a fix; thanks [~hemanthboyina] for review. > NetUtils.connect() throws unchecked exception (UnresolvedAddressException) > causing clients to abort > --- > > Key: HADOOP-17052 > URL: https://issues.apache.org/jira/browse/HADOOP-17052 > Project: Hadoop Common > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.10.0, 2.9.2, 3.2.1, 3.1.3 >Reporter: Dhiraj Hegde >Assignee: Dhiraj Hegde >Priority: Major > Fix For: 2.9.3, 3.1.4, 3.2.2, 2.10.1, 3.3.1, 3.4.0 > > Attachments: read_failure.log, write_failure1.log, write_failure2.log > > > Hadoop components are increasingly being deployed on VMs and containers. One > aspect of this environment is that DNS is dynamic. Hostname records get > modified (or deleted/recreated) as a container in Kubernetes (or even VM) is > being created/recreated. In such dynamic environments, the initial DNS > resolution request might return resolution failure briefly as DNS client > doesn't always get the latest records. This has been observed in Kubernetes > in particular. In such cases NetUtils.connect() appears to throw > java.nio.channels.UnresolvedAddressException. In much of Hadoop code (like > DFSInputStream and DFSOutputStream), the code is designed to retry > IOException. However, since UnresolvedAddressException is not child of > IOException, no retry happens and the code aborts immediately. It is much > better if NetUtils.connect() throws java.net.UnknownHostException as that is > derived from IOException and the code will treat this as a retry-able error. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17013) this bug is bla bla bla
[ https://issues.apache.org/jira/browse/HADOOP-17013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-17013. Fix Version/s: (was: 0.24.0) Target Version/s: (was: 3.2.1) Resolution: Invalid This is not fun [~islam.saied]. Please stop doing this. > this bug is bla bla bla > --- > > Key: HADOOP-17013 > URL: https://issues.apache.org/jira/browse/HADOOP-17013 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 3.2.1 >Reporter: islam >Priority: Major > Labels: bulk-closed > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16853) ITestS3GuardOutOfBandOperations failing on versioned S3 buckets
[ https://issues.apache.org/jira/browse/HADOOP-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-16853. Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to {{trunk}}. Thanks [~ste...@apache.org] for reporting and fixing. Thanks [~gabor.bota] for review. > ITestS3GuardOutOfBandOperations failing on versioned S3 buckets > --- > > Key: HADOOP-16853 > URL: https://issues.apache.org/jira/browse/HADOOP-16853 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.0 > > > org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations.testListingDelete[auth=true] > failing because the deleted file can still be read when the s3guard entry has > the versionId. > Proposed: if the FS is versioned and the file status has versionID then we > switch to tests which assert the file is readable, rather than tests which > assert it isn't there -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16827) TestHarFileSystem.testInheritedMethodsImplemented broken
[ https://issues.apache.org/jira/browse/HADOOP-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-16827. Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed > TestHarFileSystem.testInheritedMethodsImplemented broken > > > Key: HADOOP-16827 > URL: https://issues.apache.org/jira/browse/HADOOP-16827 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, test >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > > caused by HADOOP-16759. > I'm surprised this didn't surface earlier -or embarrassed that if it did, > somehow I missed it. Will fix. > > will also review the checksum FS to make sure it's gone through there too. > > Given I was using the IDE to refactor, it should have all been automatic -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16759) Filesystem openFile() builder to take a FileStatus param
[ https://issues.apache.org/jira/browse/HADOOP-16759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-16759. Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to {{trunk}} branch. I see major conflicts when backporting to 3.2. [~ste...@apache.org] Do you plan to bulk backport some time later those patches which apply, or you upload a new patch for branch-3.2/3.1? Thanks! > Filesystem openFile() builder to take a FileStatus param > > > Key: HADOOP-16759 > URL: https://issues.apache.org/jira/browse/HADOOP-16759 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 3.3.0 > > > Let us pass in a file status to openFile() so that S3A & ABFS will skip > their own HEAD requests just to see if a file is there, a normal file and get > its length + etag, > {code} > CompletableFuture streamF = fs.openFile(stat.getPath()) > .withFileStatus(stat).build() > {code} > code opening files off a listing of everything in a directory can eliminate a > lot of requests here. > Also: change the specification of openFile's completable future to say > "returned stream may only raise FNFE, access restrictions on the first read" > That is: it's not just potentially an async open, it's possibly lazy > evaluated entirely. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16757) Increase timeout unit test rule for MetadataStoreTestBase
[ https://issues.apache.org/jira/browse/HADOOP-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-16757. Fix Version/s: 3.3.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to {{trunk}} branch. Not sure about backporting practice here, and left a message in the uber JIRA. Thanks [~ste...@apache.org] for reviewing! > Increase timeout unit test rule for MetadataStoreTestBase > - > > Key: HADOOP-16757 > URL: https://issues.apache.org/jira/browse/HADOOP-16757 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Mingliang Liu >Assignee: Mingliang Liu >Priority: Major > Fix For: 3.3.0 > > > Not sure if this is a good proposal, but I saw a few cases where some > integration test methods in {{ITestDynamoDBMetadataStore}} timed out simply. > Specially, the one keeps failing me today is > {{testAncestorOverwriteConflict}}. I increase the timeout to 200s and it > works for me happily. Am I using VPN and a slow home network, I'm afraid so. > The time out rule, as inherited from base class {{HadoopTestBase}}, is 100s > by default. Though that 100s time out default value can be overridden in base > class via system property {{test.default.timeout}}, that's system wide > affecting all other tests. Changing that time out value for one test is no > better than overriding in this test {{ITestDynamoDBMetadataStore}} > explicitly. I think doubling it to 200s would not be very crazy considering > we are testing against a remote web service, create and destroy tables > sometimes. > Thoughts? > [EDIT]: I meant 100s when I said 10s. Sorry for confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16758) Refine testing.md to tell user better how to use auth-keys.xml
Mingliang Liu created HADOOP-16758: -- Summary: Refine testing.md to tell user better how to use auth-keys.xml Key: HADOOP-16758 URL: https://issues.apache.org/jira/browse/HADOOP-16758 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu I have a small patch to refine \{{testing.md}} about `auth-keys.xml`. I think this might be applicable to others. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16757) Increase timeout unit test rule for ITestDynamoDBMetadataStore
Mingliang Liu created HADOOP-16757: -- Summary: Increase timeout unit test rule for ITestDynamoDBMetadataStore Key: HADOOP-16757 URL: https://issues.apache.org/jira/browse/HADOOP-16757 Project: Hadoop Common Issue Type: Sub-task Reporter: Mingliang Liu Not sure if this is a good proposal, but I saw a few cases where some integration test methods in {{ITestDynamoDBMetadataStore}} timed out simply. Specially, the one keeps failing me today is {{testAncestorOverwriteConflict}}. I increase the timeout to 20s and it works for me happily. Am I using VPN and a slow home network, I'm afraid so. The time out rule, as inherited from base class {{HadoopTestBase}}, is 10s by default. Though that 10s time out default value can be overridden in base class via system property {{test.default.timeout}}, that's system wide affecting all other tests. Changing that time out value for one test is no better than overriding in this test {{ITestDynamoDBMetadataStore}} explicitly. I think doubling it to 20s would not be very crazy considering we are testing against a remote web service, create and destroy tables sometimes. {code} @Rule public Timeout timeout = new Timeout(20 * 1000); {code} Thoughts? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16735) Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN
Mingliang Liu created HADOOP-16735: -- Summary: Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN Key: HADOOP-16735 URL: https://issues.apache.org/jira/browse/HADOOP-16735 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu In the great doc {{hadoop-aws/tools/hadoop-aws/index.html}}, user can find that authenticating via the AWS Environment Variables supports session token. However, the config description in core-default.xml does not make it clear. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16732) S3Guard to support encrypted DynamoDB table
Mingliang Liu created HADOOP-16732: -- Summary: S3Guard to support encrypted DynamoDB table Key: HADOOP-16732 URL: https://issues.apache.org/jira/browse/HADOOP-16732 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Mingliang Liu S3Guard is not yet supporting [encrypted DynamoDB table|https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/encryption.tutorial.html]. We can provide an option to enable encrypted DynamoDB table so data at rest could be encrypted. S3Guard data in DynamoDB usually is not sensitive since it's the S3 namespace mirroring, but some times even this is a concern. By default it's not enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16722) S3GuardTool to support FilterFileSystem
Mingliang Liu created HADOOP-16722: -- Summary: S3GuardTool to support FilterFileSystem Key: HADOOP-16722 URL: https://issues.apache.org/jira/browse/HADOOP-16722 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Mingliang Liu Currently S3GuardTool is operating against a S3AFileSystem implementation. There are cases where the {{fs.hboss.fs.s3a.impl}} is a FilterFileSystem wrapping another implementation. For example, [[HBASE-23314]] made {{HBaseObjectStoreSemantics}} a {{FilterFileSystem}} from {{FileSystem}}. S3GuardTool could use {{FilterFileSystem::getRawFileSystem}} method to get the wrapped S3AFileSystem implementation. Without this support, a simple S3GuardTool against HBOSS will get confusing error like "s3a://mybucket is not a S3A file system". -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16355) ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store
Mingliang Liu created HADOOP-16355: -- Summary: ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store Key: HADOOP-16355 URL: https://issues.apache.org/jira/browse/HADOOP-16355 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Mingliang Liu When S3Guard was proposed, there are a couple of valid reasons to choose DynamoDB as its default backend store: 0) seamless integration as part of AWS ecosystem e.g. client library 1) it's a managed web service which is zero operational cost, highly available and infinitely scalable 2) it's performant with single digit latency 3) it's proven by Netflix's S3mper (not actively maintained) and EMRFS (closed source and usage). As it's pluggable, it's possible to implement {{MetadataStore}} with other backend store without changing semantics, besides null and in-memory local ones. Here we propose {{ZookeeperMetadataStore}} which uses Zookeeper as S3Guard backend store. Its main motivation is to provide a new MetadataStore option which: # can be easily integrated as Zookeeper is heavily used in Hadoop community # affordable performance as both client and Zookeeper ensemble are usually "local" in a Hadoop cluster (ZK/HBase/Hive etc) # removes DynamoDB dependency Obviously all use cases will not prefer this to default DynamoDB store. For e.g. ZK might not scale well if there are dozens of S3 buckets and each has millions of objects. Our use case is targeting HBase to store HFiles on S3 instead of HDFS. A total solution for HBase on S3 must be HBOSS (see HBASE-22149) for recovering atomicity of metadata operations like rename, and S3Guard for consistent enumeration and access to object store bucket metadata. We would like to use Zookeeper as backend store for both. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14601) Azure: Reuse ObjectMapper
Mingliang Liu created HADOOP-14601: -- Summary: Azure: Reuse ObjectMapper Key: HADOOP-14601 URL: https://issues.apache.org/jira/browse/HADOOP-14601 Project: Hadoop Common Issue Type: Sub-task Components: fa/azure Reporter: Mingliang Liu Assignee: Mingliang Liu Currently there are a few places in {{hadoop-azure}} module that uses creates {{ObjectMapper}} for each request/call. We should re-use the object mapper for performance purpose. The general caveat is about thread safety; I think the change will be safe though. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14594) ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions
Mingliang Liu created HADOOP-14594: -- Summary: ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions Key: HADOOP-14594 URL: https://issues.apache.org/jira/browse/HADOOP-14594 Project: Hadoop Common Issue Type: Bug Reporter: Mingliang Liu Assignee: Mingliang Liu Per discussion [HADOOP-14255] and [HADOOP-13222], we can delete the TODO comment in tests for metric assertions. See the attached patch for more details. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14536) Update azure-storage sdk to version 5.2.0
Mingliang Liu created HADOOP-14536: -- Summary: Update azure-storage sdk to version 5.2.0 Key: HADOOP-14536 URL: https://issues.apache.org/jira/browse/HADOOP-14536 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.0.0-alpha3 Reporter: Mingliang Liu Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients. We are currently using version 4.2.0 of the SDK. Azure Storage Clients changes between 4.2 and 5.2: * Fixed Exists() calls on Shares and Directories to now populate metadata. This was already being done for Files. * Changed blob constants to support up to 256 MB on put blob for block blobs. The default value for put blob threshold has also been updated to half of the maximum, or 128 MB currently. * Fixed a bug that prevented setting content MD5 to true when creating a new file. * Fixed a bug where access conditions, options, and operation context were not being passed when calling openWriteExisting() on a page blob or a file. * Fixed a bug where an exception was being thrown on a range get of a blob or file when the options disableContentMD5Validation is set to false and useTransactionalContentMD5 is set to true and there is no overall MD5. * Fixed a bug where retries were happening immediately if a socket exception was thrown. * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. This check has been moved to create() and uploadProperties() in CloudFileShare. * Prefix support for listing files and directories. * Added support for setting public access when creating a blob container * The public access setting on a blob container is now a container property returned from downloadProperties. * Add Message now modifies the PopReceipt, Id, NextVisibleTime, InsertionTime, and ExpirationTime properties of its CloudQueueMessage parameter. * Populate content MD5 for range gets on Blobs and Files. * Added support in Page Blob for incremental copy. * Added large BlockBlob upload support. Blocks can now support sizes up to 100 MB. * Added a new, memory-optimized upload strategy for the upload* APIs. This algorithm only applies for blocks greater than 4MB and when storeBlobContentMD5 and Client-Side Encryption are disabled. * getQualifiedUri() has been deprecated for Blobs. Please use getSnapshotQualifiedUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token. * getQualifiedStorageUri() has been deprecated for Blobs. Please use getSnapshotQualifiedStorageUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token. * Fixed a bug where copying from a blob that included a SAS token and a snapshot ommitted the SAS token. * Fixed a bug in client-side encryption for tables that was preventing the Java client from decrypting entities encrypted with the .NET client, and vice versa. * Added support for server-side encryption. * Added support for getBlobReferenceFromServer methods on CloudBlobContainer to support retrieving a blob without knowing its type. * Fixed a bug in the retry policies where 300 status codes were being retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14516) Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients
[ https://issues.apache.org/jira/browse/HADOOP-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-14516. Resolution: Duplicate Closing as duplicates. Please see [HADOOP-14490] and comment there. Thanks, > Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft > Azure Storage Clients > --- > > Key: HADOOP-14516 > URL: https://issues.apache.org/jira/browse/HADOOP-14516 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov > > Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.2: > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14500) Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails
Mingliang Liu created HADOOP-14500: -- Summary: Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails Key: HADOOP-14500 URL: https://issues.apache.org/jira/browse/HADOOP-14500 Project: Hadoop Common Issue Type: Bug Components: fs/azure, test Reporter: Mingliang Liu The following test fails: {code} TestFileSystemOperationExceptionHandling.testSingleThreadBlockBlobSeekScenario Expected exception: java.io.FileNotFoundExceptionTestFileSystemOperationsExceptionHandlingMultiThreaded.testMultiThreadBlockBlobSeekScenario Expected exception: java.io.FileNotFoundException {code} I did early analysis and found [HADOOP-14478] maybe the reason. I think we can fix the test itself here. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14498) HADOOP_OPTIONAL_TOOLS not parsed correctly
Mingliang Liu created HADOOP-14498: -- Summary: HADOOP_OPTIONAL_TOOLS not parsed correctly Key: HADOOP-14498 URL: https://issues.apache.org/jira/browse/HADOOP-14498 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0-alpha1 Reporter: Mingliang Liu # This will make hadoop-azure not show up in the hadoop classpath, though both hadoop-aws and hadoop-azure-datalake are in the classpath.{code:title=hadoop-env.sh} export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws,hadoop-azure-datalake" {code} # And if we put only hadoop-azure and hadoop-aws, both of them are shown in the classpath. {code:title=hadoop-env.sh} export HADOOP_OPTIONAL_TOOLS="hadoop-azure,hadoop-aws" {code} This makes me guess that, while parsing the {{HADOOP_OPTIONAL_TOOLS}}, we make some assumptions that hadoop tool modules have a single "-" in names, and the _hadoop-azure-datalake_ overrides the _hadoop-azure_. Or any other assumptions about the {{${project.artifactId\}}}? Ping [~aw]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14491) Azure has messed doc structure
Mingliang Liu created HADOOP-14491: -- Summary: Azure has messed doc structure Key: HADOOP-14491 URL: https://issues.apache.org/jira/browse/HADOOP-14491 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs/azure Reporter: Mingliang Liu Assignee: Mingliang Liu # The _WASB Secure mode and configuration_ and _Authorization Support in WASB_ sections are missing in the navigation # _Authorization Support in WASB_ should be header level 3 instead of level 2 # Some of the code format is not specified # Sample code indent not unified. Let's use the auto-generated navigation instead of manually updating it, just as other documents. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14490) Upgrade azure-storage sdk version
Mingliang Liu created HADOOP-14490: -- Summary: Upgrade azure-storage sdk version Key: HADOOP-14490 URL: https://issues.apache.org/jira/browse/HADOOP-14490 Project: Hadoop Common Issue Type: Improvement Reporter: Mingliang Liu As required by [HADOOP-14478], we're expecting the {{BlobInputStream}} to support advanced {{readFully()}} by taking hints of mark. This can only be done by means of sdk version bump. cc: [~rajesh.balamohan]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14472) Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently
Mingliang Liu created HADOOP-14472: -- Summary: Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently Key: HADOOP-14472 URL: https://issues.apache.org/jira/browse/HADOOP-14472 Project: Hadoop Common Issue Type: Bug Components: fs/azure Reporter: Mingliang Liu Reported by [HADOOP-14461] {code} testManySmallWritesWithHFlush(org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite) Time elapsed: 1.051 sec <<< FAILURE! java.lang.AssertionError: hflush duration of 13, less than minimum expected of 20 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite.writeAndReadOneFile(TestReadAndSeekPageBlobAfterWrite.java:286) at org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite.testManySmallWritesWithHFlush(TestReadAndSeekPageBlobAfterWrite.java:247) {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14460) Azure: fs.azure.account.key.youraccount.blob.core.windows.net -> fs.azure.account.key.youraccount
Mingliang Liu created HADOOP-14460: -- Summary: Azure: fs.azure.account.key.youraccount.blob.core.windows.net -> fs.azure.account.key.youraccount Key: HADOOP-14460 URL: https://issues.apache.org/jira/browse/HADOOP-14460 Project: Hadoop Common Issue Type: Bug Components: documentation, fs/azure Reporter: Mingliang Liu Assignee: Mingliang Liu In {{SimpleKeyProvider}}, we have following code for getting the key: {code} protected static final String KEY_ACCOUNT_KEY_PREFIX = "fs.azure.account.key."; ... protected String getStorageAccountKeyName(String accountName) { return KEY_ACCOUNT_KEY_PREFIX + accountName; } {code} While in documentation {{index.md}}, we have: {code} fs.azure.account.key.youraccount.blob.core.windows.net YOUR ACCESS KEY {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14458) TestAliyunOSSFileSystemContract missing imports/
Mingliang Liu created HADOOP-14458: -- Summary: TestAliyunOSSFileSystemContract missing imports/ Key: HADOOP-14458 URL: https://issues.apache.org/jira/browse/HADOOP-14458 Project: Hadoop Common Issue Type: Bug Components: fs/oss, test Reporter: Mingliang Liu Assignee: Mingliang Liu {code} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile (default-testCompile) on project hadoop-aliyun: Compilation failure: Compilation failure: [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[71,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[90,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[91,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[92,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[93,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[95,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[96,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[98,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[99,5] cannot find symbol [ERROR] symbol: method assertTrue(java.lang.String,boolean) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[115,7] cannot find symbol [ERROR] symbol: method fail(java.lang.String) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[129,7] cannot find symbol [ERROR] symbol: method fail(java.lang.String) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[143,7] cannot find symbol [ERROR] symbol: method fail(java.lang.String) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContract.java:[163,7] cannot find symbol [ERROR] symbol: method fail(java.lang.String) [ERROR] location: class org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract [ERROR] /Users/mliu/Workspace/hadoop/hadoop-tools/hadoop-aliyun/src/test/java/org/apache/hadoop/fs/aliyun/oss/TestAliyunOSSFileSystemContrac
[jira] [Created] (HADOOP-14438) Make ADLS doc of setting up client key up to date
Mingliang Liu created HADOOP-14438: -- Summary: Make ADLS doc of setting up client key up to date Key: HADOOP-14438 URL: https://issues.apache.org/jira/browse/HADOOP-14438 Project: Hadoop Common Issue Type: Sub-task Components: fs/adl Reporter: Mingliang Liu In the doc {{hadoop-tools/hadoop-azure-datalake/src/site/markdown/index.md}}, we have such a statement: {code:title=Note down the properties you will need to auth} ... - Resource: Always https://management.core.windows.net/ , for all customers {code} Is the {{Resource}} useful here? It seems not necessary to me. {code:title=Adding the service principal to your ADL Account} - ... - Select Users under Settings ... {code} According to the portal, it should be "Access control (IAM)" under "Settings" -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14363) Inconsistent default block location in FileSystem javadoc
Mingliang Liu created HADOOP-14363: -- Summary: Inconsistent default block location in FileSystem javadoc Key: HADOOP-14363 URL: https://issues.apache.org/jira/browse/HADOOP-14363 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Mingliang Liu Assignee: Chen Liang Priority: Trivial {code:title=FileSystem::getFileBlockLocations()} .. * * The default implementation returns an array containing one element: * * BlockLocation( { "localhost:50010" }, { "localhost" }, 0, file.getLen()) * > * * @param file FilesStatus to get data from * @param start offset into the given file * @param len length for which to get locations for * @throws IOException IO failure */ public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) throws IOException { ... String[] name = {"localhost:9866"}; String[] host = {"localhost"}; return new BlockLocation[] { new BlockLocation(name, host, 0, file.getLen()) }; } {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14349) Rename ADLS CONTRACT_ENABLE_KEY
Mingliang Liu created HADOOP-14349: -- Summary: Rename ADLS CONTRACT_ENABLE_KEY Key: HADOOP-14349 URL: https://issues.apache.org/jira/browse/HADOOP-14349 Project: Hadoop Common Issue Type: Sub-task Components: fs/adl Affects Versions: 2.8.0 Reporter: Mingliang Liu dfs.adl.test.contract.enable -> fs.adl.test.contract.enable -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14345) S3Guard: S3GuardTool to support provisioning metadata store
Mingliang Liu created HADOOP-14345: -- Summary: S3Guard: S3GuardTool to support provisioning metadata store Key: HADOOP-14345 URL: https://issues.apache.org/jira/browse/HADOOP-14345 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Mingliang Liu Priority: Minor I don't know if this is considered a requested feature from S3Guard code, as the user can always provision the DDB tables via the cloud portal. Implementing should be straightforward. The {{DynamoDBMetadataStore}} has provision method to use. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14323) ITestS3GuardListConsistency failure w/ Local, authoritative metadata store
[ https://issues.apache.org/jira/browse/HADOOP-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-14323: > ITestS3GuardListConsistency failure w/ Local, authoritative metadata store > -- > > Key: HADOOP-14323 > URL: https://issues.apache.org/jira/browse/HADOOP-14323 > Project: Hadoop Common > Issue Type: Sub-task > Components: s3 >Affects Versions: HADOOP-13345 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Minor > Fix For: HADOOP-13345 > > Attachments: HADOOP-14323-HADOOP-13345.001.patch, > HADOOP-14323-HADOOP-13345.002.patch > > > When doing some testing for HADOOP-14266 I noticed this test failure: > {noformat} > java.lang.NullPointerException: null > at > org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:317) > {noformat} > I was running with LocalMetadataStore and > {{fs.s3a.metadatastore.authoritative}} set to true. I haven't been testing > this mode recently so not sure if this case ever worked. Lower priority but > we should fix it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14165) Add S3Guard.dirListingUnion in S3AFileSystem#listFiles, listLocatedStatus
[ https://issues.apache.org/jira/browse/HADOOP-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-14165. Resolution: Duplicate Resolving as duplicate to [HADOOP-13926] and [HADOOP-14266]. > Add S3Guard.dirListingUnion in S3AFileSystem#listFiles, listLocatedStatus > - > > Key: HADOOP-14165 > URL: https://issues.apache.org/jira/browse/HADOOP-14165 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Rajesh Balamohan >Priority: Minor > > {{S3Guard::dirListingUnion}} merges information from backing store and DDB to > create consistent view. This needs to be added in > {{S3AFileSystem::listFiles}} and {{S3AFileSystem::listLocatedStatus}} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14215) DynamoDB client should waitForActive on existing tables
[ https://issues.apache.org/jira/browse/HADOOP-14215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-14215. Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HADOOP-13345 Committed to feature branch. Thanks for your contribution [~mackrorysd]. Thanks [~fabbri] for review and [~rajesh.balamohan] for offline discussion. > DynamoDB client should waitForActive on existing tables > --- > > Key: HADOOP-14215 > URL: https://issues.apache.org/jira/browse/HADOOP-14215 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Sean Mackrory >Assignee: Sean Mackrory >Priority: Critical > Fix For: HADOOP-13345 > > Attachments: HADOOP-14215-HADOOP-13345.000.patch, > HADOOP-14215-HADOOP-13345.001.patch, HADOOP-14215-HADOOP-13345.002.patch > > > I saw a case where 2 separate applications tried to use the same > non-pre-existing table with table.create = true at about the same time. One > failed with a ResourceInUse exception. If a table does not exist, we attempt > to create it and then wait for it to enter the active state. If another jumps > in in the middle of that, the table may exist, thus bypassing our call to > waitForActive(), and then try to use the table immediately. > While we're at it, let's also make sure that the race condition where a table > might get created between checking if it exists and attempting to create it > is handled gracefully. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14282) S3Guard: DynamoDBMetadata::prune() should self interrupt correctly
Mingliang Liu created HADOOP-14282: -- Summary: S3Guard: DynamoDBMetadata::prune() should self interrupt correctly Key: HADOOP-14282 URL: https://issues.apache.org/jira/browse/HADOOP-14282 Project: Hadoop Common Issue Type: Sub-task Reporter: Mingliang Liu Assignee: Mingliang Liu -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14266) S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
Mingliang Liu created HADOOP-14266: -- Summary: S3Guard: S3AFileSystem::listFiles() to employ MetadataStore Key: HADOOP-14266 URL: https://issues.apache.org/jira/browse/HADOOP-14266 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Mingliang Liu Similar to [HADOOP-13926], this is to track the effort of employing MetadataStore in {{S3AFileSystem::listFiles()}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14255) S3A to delete unnecessary fake directory objects in mkdirs()
Mingliang Liu created HADOOP-14255: -- Summary: S3A to delete unnecessary fake directory objects in mkdirs() Key: HADOOP-14255 URL: https://issues.apache.org/jira/browse/HADOOP-14255 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu In S3AFileSystem, as an optimization, we delete unnecessary fake directory objects if that directory contains at least one (nested) file. That is done in closing stream of newly created file. However, if the directory becomes non-empty after we just create an empty subdirectory, we do not delete its fake directory object though that fake directory object becomes "unnecessary". So in {{S3AFileSystem::mkdirs()}}, we have a pending TODO: {quote} // TODO: If we have created an empty file at /foo/bar and we then call // mkdirs for /foo/bar/baz/roo what happens to the empty file /foo/bar/? private boolean innerMkdirs(Path p, FsPermission permission) {quote} This JIRA is to fix the TODO: provide consistent behavior for a fake directory object between its nested subdirectory and nested file by deleting it. See related discussion in [HADOOP-14236]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14248) Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade
Mingliang Liu created HADOOP-14248: -- Summary: Retire SharedInstanceProfileCredentialsProvider after AWS SDK upgrade Key: HADOOP-14248 URL: https://issues.apache.org/jira/browse/HADOOP-14248 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.0.0-alpha3 Reporter: Mingliang Liu Assignee: Mingliang Liu This is from the discussion in [HADOOP-13050]. So [HADOOP-13727] added the SharedInstanceProfileCredentialsProvider, which effectively reduces high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. That patch, in order to prevent the throttling problem, defined new class {{SharedInstanceProfileCredentialsProvider}} as a subclass of {{InstanceProfileCredentialsProvider}}, which enforces creation of only a single instance. Per [HADOOP-13050], we upgraded the AWS Java SDK. Since then, the {{InstanceProfileCredentialsProvider}} in SDK code internally enforces a singleton. That confirms that our effort in [HADOOP-13727] makes 100% sense. Meanwhile, {{SharedInstanceProfileCredentialsProvider}} can retire gracefully in trunk branch. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14247) FileContextMainOperationsBaseTest should clean up test root path
Mingliang Liu created HADOOP-14247: -- Summary: FileContextMainOperationsBaseTest should clean up test root path Key: HADOOP-14247 URL: https://issues.apache.org/jira/browse/HADOOP-14247 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14236) S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store
Mingliang Liu created HADOOP-14236: -- Summary: S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store Key: HADOOP-14236 URL: https://issues.apache.org/jira/browse/HADOOP-14236 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu After running integration test {{ITestS3AFileSystemContract}}, I found the following items are not cleaned up in DynamoDB: {code} parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExisting/dir, child=subdir parent=/mliu-s3guard/user/mliu/s3afilesystemcontract/testRenameDirectoryAsExistingNew/newdir/subdir, child=file2 {code} At first I thought it’s similar to [HADOOP-14226] or [HADOOP-14227], and we need to be careful when cleaning up test data. Then I found it’s a bug in the code of integrating S3Guard with S3AFileSystem: for rename we miss sub-directory items to put (dest) and delete (src). The reason is that in S3A, we delete those fake directory objects if they are not necessary, e.g. non-empty. So when we list the objects to rename, the object summaries will only return _file_ objects. This has two consequences after rename: # there will be left items for src path in metadata store - left-overs will confuse {{get(Path)}} which should return null # we are not persisting the whole subtree for dest path to metadata store - this will break the DynamoDBMetadataStore invariant: _if a path exists, all its ancestors will also exist in the table_. Existing tests are not complaining about this though. If this is a real bug, let’s address it here. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14227) S3Guard: ITestS3AConcurrentOps is not cleaning up test data
Mingliang Liu created HADOOP-14227: -- Summary: S3Guard: ITestS3AConcurrentOps is not cleaning up test data Key: HADOOP-14227 URL: https://issues.apache.org/jira/browse/HADOOP-14227 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu After running {{ITestS3AConcurrentOps}}, the test data is not cleanup in DynamoDB. There are two reasons: # The {{ITestS3AConcurrentOps::teardown()}} method is not calling super teardown() method to clean up the default test directory. # The {{auxFs}} is not S3Guard aware even though the {{fs}} to test is. That's because the {{auxFs}} is creating a new Configuration object without patching in S3Guard options (via {{maybeEnableS3Guard(conf);}}). This JIRA is to clean up the data after test. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14226) S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
Mingliang Liu created HADOOP-14226: -- Summary: S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data Key: HADOOP-14226 URL: https://issues.apache.org/jira/browse/HADOOP-14226 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Mingliang Liu Assignee: Mingliang Liu After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not cleaned up. There is a call to {{clearMetadataStore(ms, count);}} in the finally clause though. The reason is that, the internally called method {{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an item for the parent dest path: {code} parent=/fake-bucket, child=moved-here, is_dir=true {code} In DynamoDBMetadataStore implementation, we assume that _if a path exists, all its ancestors will also exist in the table_. We need to pre-create dest path to maintain this invariant so that test data can be cleaned up successfully. I think there may be other tests with the same problem. Let's identify/address them separately. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14214) DomainSocketWatcher::add()/delete() should not self interrupt while looping await()
Mingliang Liu created HADOOP-14214: -- Summary: DomainSocketWatcher::add()/delete() should not self interrupt while looping await() Key: HADOOP-14214 URL: https://issues.apache.org/jira/browse/HADOOP-14214 Project: Hadoop Common Issue Type: Bug Reporter: Mingliang Liu Assignee: Mingliang Liu Our hive team found a TPCDS job whose queries running on LLAP seem to be getting stuck. Dozens of threads were waiting for the {{DfsClientShmManager::lock}}, as following jstack: {code} Thread 251 (IO-Elevator-Thread-5): State: WAITING Blocked count: 3871 Wtaited count: 4565 Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16ead198 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitUninterruptibly(AbstractQueuedSynchronizer.java:1976) org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:255) org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434) org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017) org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476) org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784) org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718) org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422) org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333) org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181) org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118) org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478) org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441) org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121) org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111) org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.readStripeFooter(RecordReaderUtils.java:166) org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata.(OrcStripeMetadata.java:64) org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.readStripesMetadata(OrcEncodedDataReader.java:622) {code} The thread that is expected to signal those threads is calling {{DomainSocketWatcher::add()}} method, but it gets stuck there dealing with InterruptedException infinitely. The jstack is like: {code} Thread 44417 (TezTR-257387_2840_12_10_52_0): State: RUNNABLE Blocked count: 3 Wtaited count: 5 Stack: java.lang.Throwable.fillInStackTrace(Native Method) java.lang.Throwable.fillInStackTrace(Throwable.java:783) java.lang.Throwable.(Throwable.java:250) java.lang.Exception.(Exception.java:54) java.lang.InterruptedException.(InterruptedException.java:57) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2034) org.apache.hadoop.net.unix.DomainSocketWatcher.add(DomainSocketWatcher.java:325) org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager$EndpointShmManager.allocSlot(DfsClientShmManager.java:266) org.apache.hadoop.hdfs.shortcircuit.DfsClientShmManager.allocSlot(DfsClientShmManager.java:434) org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.allocShmSlot(ShortCircuitCache.java:1017) org.apache.hadoop.hdfs.BlockReaderFactory.createShortCircuitReplicaInfo(BlockReaderFactory.java:476) org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.create(ShortCircuitCache.java:784) org.apache.hadoop.hdfs.shortcircuit.ShortCircuitCache.fetchOrCreate(ShortCircuitCache.java:718) org.apache.hadoop.hdfs.BlockReaderFactory.getBlockReaderLocal(BlockReaderFactory.java:422) org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:333) org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1181) org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:1118) org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1478) org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1441) org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121) {code} The whole job makes no progress because of this. The thread in {{DomainSocketWatcher::add()}} is expected to eventually break the while loop where it waits for the newly added entry being deleted by another thread. However, if this thread is ever interrupted, chances are that it will hold the lock forever so {{if(!toAdd.contains(e
[jira] [Created] (HADOOP-14194) Alyun OSS should not use empty endpoint as default
Mingliang Liu created HADOOP-14194: -- Summary: Alyun OSS should not use empty endpoint as default Key: HADOOP-14194 URL: https://issues.apache.org/jira/browse/HADOOP-14194 Project: Hadoop Common Issue Type: Bug Components: fs/oss Reporter: Mingliang Liu Assignee: Xiaobing Zhou In {{AliyunOSSFileSystemStore::initialize()}}, it retrieves the endPoint and using empty string as a default value. {code} String endPoint = conf.getTrimmed(ENDPOINT_KEY, ""); {code} The plain value without validation is passed to OSSClient. If the endPoint is not provided (empty string) or the endPoint is not valid, users will get exception from Aliyun OSS sdk with raw exception message like: {code} java.lang.IllegalArgumentException: java.net.URISyntaxException: Expected authority at index 8: https:// at com.aliyun.oss.OSSClient.toURI(OSSClient.java:359) at com.aliyun.oss.OSSClient.setEndpoint(OSSClient.java:313) at com.aliyun.oss.OSSClient.(OSSClient.java:297) at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.initialize(AliyunOSSFileSystemStore.java:134) at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:272) at org.apache.hadoop.fs.aliyun.oss.AliyunOSSTestUtils.createTestFileSystem(AliyunOSSTestUtils.java:63) at org.apache.hadoop.fs.aliyun.oss.TestAliyunOSSFileSystemContract.setUp(TestAliyunOSSFileSystemContract.java:47) at junit.framework.TestCase.runBare(TestCase.java:139) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) Caused by: java.net.URISyntaxException: Expected authority at index 8: https:// at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.failExpecting(URI.java:2854) at java.net.URI$Parser.parseHierarchical(URI.java:3102) at java.net.URI$Parser.parse(URI.java:3053) at java.net.URI.(URI.java:588) at com.aliyun.oss.OSSClient.toURI(OSSClient.java:357) {code} Let's check endPoint is not null or empty, catch the IllegalArgumentException and log it, wrapping the exception with clearer message stating the misconfiguration in endpoint or credentials. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14192) Aliyun OSS FileSystem contract test should implement getTestBaseDir()
Mingliang Liu created HADOOP-14192: -- Summary: Aliyun OSS FileSystem contract test should implement getTestBaseDir() Key: HADOOP-14192 URL: https://issues.apache.org/jira/browse/HADOOP-14192 Project: Hadoop Common Issue Type: Bug Components: fs/oss Reporter: Mingliang Liu Assignee: Mingliang Liu [HADOOP-14170] is the recent effort of improving the file system contract tests {{FileSystemContractBaseTest}}, which make {{path()}} method final and add a new method {{getTestBaseDir()}} for subclasses to implement. Aliyun OSS should override that as it uses unique directory (naming with fork id) for supporting parallel tests. Plus, the current {{testWorkingDirectory}} needs not override per changes in {{FileSystemContractBaseTest}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13959) S3guard: replace dynamo.describe() call in init with more efficient query
[ https://issues.apache.org/jira/browse/HADOOP-13959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-13959. Resolution: Fixed As [HADOOP-13985] is committed, I think we can resolve this one. Feel free to re-open if necessary. > S3guard: replace dynamo.describe() call in init with more efficient query > - > > Key: HADOOP-13959 > URL: https://issues.apache.org/jira/browse/HADOOP-13959 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Mingliang Liu >Priority: Minor > > HADOOP-13908 adds initialization when a table isn't created, using the > {{describe()}} call. > AWS document this as inefficient, and throttle it. We should be able to get > away with a simple table lookup as the probe -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14180) FileSystem contract tests to replace JUnit 3 with 4
Mingliang Liu created HADOOP-14180: -- Summary: FileSystem contract tests to replace JUnit 3 with 4 Key: HADOOP-14180 URL: https://issues.apache.org/jira/browse/HADOOP-14180 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Mingliang Liu This is from discussion in [HADOOP-14170], as Steve commented: {quote} ...it's time to move this to JUnit 4, annotate all tests with @test, and make the test cases skip if they don't have the test FS defined. JUnit 3 doesn't support Assume, so when I do test runs without the s3n or s3 fs specced, I get lots of errors I just ignore. ...Move to Junit 4, and, in our own code, find everywhere we've subclassed a method to make the test a no-op, and insert an Assume.assumeTrue(false) in there so they skip properly. {quote} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14170) FileSystemContractBaseTest is not cleaning up test directory clearly
Mingliang Liu created HADOOP-14170: -- Summary: FileSystemContractBaseTest is not cleaning up test directory clearly Key: HADOOP-14170 URL: https://issues.apache.org/jira/browse/HADOOP-14170 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Mingliang Liu Assignee: Mingliang Liu In {{FileSystemContractBaseTest::tearDown()}} method, it cleans up the {{path("/test")}} directory, which will be qualified as {{/test}} (against root instead of working directory because it's absolute): {code} @Override protected void tearDown() throws Exception { try { if (fs != null) { fs.delete(path("/test"), true); } } catch (IOException e) { LOG.error("Error deleting /test: " + e, e); } } {code} But in the test, it uses {{path("test")}} sometimes, which will be made qualified against the working directory (e.g. {{/user/bob/test}}). This makes some tests fail intermittently, e.g. {{ITestS3AFileSystemContract}}. Also see the discussion in [HADOOP-13934]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14153) ADL module has messed doc structure
Mingliang Liu created HADOOP-14153: -- Summary: ADL module has messed doc structure Key: HADOOP-14153 URL: https://issues.apache.org/jira/browse/HADOOP-14153 Project: Hadoop Common Issue Type: Bug Components: fs/adl Reporter: Mingliang Liu Assignee: Mingliang Liu -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-14129) ITestS3ACredentialsInURL sometimes fails
[ https://issues.apache.org/jira/browse/HADOOP-14129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-14129: > ITestS3ACredentialsInURL sometimes fails > > > Key: HADOOP-14129 > URL: https://issues.apache.org/jira/browse/HADOOP-14129 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Fix For: HADOOP-13345 > > Attachments: HADOOP-14129-HADOOP-13345.001.patch, > HADOOP-14129-HADOOP-13345.002.patch > > > This test sometimes fails. I believe it's expected that DynamoDB doesn't have > access to the credentials if they're embedded in the URL instead of the > configuration (and IMO that's fine - since the functionality hasn't been in > previous releases and since we want to discourage this practice especially > now that there are better alternatives). Weirdly, I only sometimes get this > failure on the HADOOP-13345 branch. But if the problem turns out to be what I > think it is, a simple Assume should fix it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14135) Remove URI parameter in AWSCredentialProvider constructors
Mingliang Liu created HADOOP-14135: -- Summary: Remove URI parameter in AWSCredentialProvider constructors Key: HADOOP-14135 URL: https://issues.apache.org/jira/browse/HADOOP-14135 Project: Hadoop Common Issue Type: Sub-task Reporter: Mingliang Liu Assignee: Mingliang Liu This was from comment in [HADOOP-13252]. It looks like the URI parameter is not needed for our AWSCredentialProvider constructors. This was useful because we relied on URI parameter for retrieving user:pass. Now in binding URIs, we have {code} 279 S3xLoginHelper.Login creds = getAWSAccessKeys(binding, conf); 280 credentials.add(new BasicAWSCredentialsProvider( 281 creds.getUser(), creds.getPassword())); {code} This way, we only need configuration object (if necessary) for all AWSCredentialProvider implementations. The benefit is that, if we create AWSCredentialProvider list for DynamoDB, we don't have to pass down the associated file system URI. This might be useful to S3Guard tools. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14130) Simplify DynamoDBClientFactory for creating Amazon DynamoDB clients
Mingliang Liu created HADOOP-14130: -- Summary: Simplify DynamoDBClientFactory for creating Amazon DynamoDB clients Key: HADOOP-14130 URL: https://issues.apache.org/jira/browse/HADOOP-14130 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu So, we are using deprecated {{AmazonDynamoDBClient}} class to create a DynamoDB client instead of the recommended builder. We had discussion in [HADOOP-13345] for preferring region to endpoints for user to specify the DynamoDB region (if associated S3 region is unknown or different). We have reported inconsistent behavior if endpoint and S3 region are different in [HADOOP-14027]. We also noticed that {{DynamoDBMetadataStore}} may sometimes logs nonsense region. And in [HADOOP-13252], we also have feelings that file system URI is not needed to create a {{AWSCredentialProvider}}. Resultantly we don't need to pass down file system URI for creating a DynamoDB client. So this JIRA is to change this, best effort. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13994) explicitly declare the commons-lang3 dependency as 3.4
[ https://issues.apache.org/jira/browse/HADOOP-13994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-13994. Resolution: Fixed > explicitly declare the commons-lang3 dependency as 3.4 > -- > > Key: HADOOP-13994 > URL: https://issues.apache.org/jira/browse/HADOOP-13994 > Project: Hadoop Common > Issue Type: Bug > Components: build, fs/azure, fs/s3 >Affects Versions: 3.0.0-alpha2, HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: HADOOP-13345 > > Attachments: HADOOP-13994-HADOOP-13345-001.patch, > HADOOP-13994-HADOOP-13445-001.patch > > > Other people aren't seeing this (yet?), but unless you explicitly exclude v > 3.4 of commons-lang3 from the azure build (which HADOOP-13660 does), then the > dependency declaration of commons-lang3 v 3.3.2 is creating a resolution > conflict. That's a dependency only needed for the local dynamodb & tests. > I propose to fix this in s3guard by explicitly declaring the version used in > the tests to be that of the azure-storage one, excluding that you get for > free. It doesn't impact anything shipped in production, but puts the hadoop > build in control of what versions of commons-lang are coming in everywhere by > way of the commons-config version declared in hadoop-common -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13994) explicitly declare the commons-lang3 dependency as 3.4
[ https://issues.apache.org/jira/browse/HADOOP-13994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-13994: > explicitly declare the commons-lang3 dependency as 3.4 > -- > > Key: HADOOP-13994 > URL: https://issues.apache.org/jira/browse/HADOOP-13994 > Project: Hadoop Common > Issue Type: Bug > Components: build, fs/azure, fs/s3 >Affects Versions: 3.0.0-alpha2, HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: HADOOP-13345 > > Attachments: HADOOP-13994-HADOOP-13345-001.patch, > HADOOP-13994-HADOOP-13445-001.patch > > > Other people aren't seeing this (yet?), but unless you explicitly exclude v > 3.4 of commons-lang3 from the azure build (which HADOOP-13660 does), then the > dependency declaration of commons-lang3 v 3.3.2 is creating a resolution > conflict. That's a dependency only needed for the local dynamodb & tests. > I propose to fix this in s3guard by explicitly declaring the version used in > the tests to be that of the azure-storage one, excluding that you get for > free. It doesn't impact anything shipped in production, but puts the hadoop > build in control of what versions of commons-lang are coming in everywhere by > way of the commons-config version declared in hadoop-common -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14091) AbstractFileSystem implementaion for 'wasbs' scheme
[ https://issues.apache.org/jira/browse/HADOOP-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-14091. Resolution: Fixed Sure, but let's track in our private channel. This is for community effort. Thanks > AbstractFileSystem implementaion for 'wasbs' scheme > --- > > Key: HADOOP-14091 > URL: https://issues.apache.org/jira/browse/HADOOP-14091 > Project: Hadoop Common > Issue Type: Task > Components: fs/azure > Environment: humboldt >Reporter: Varada Hemeswari >Assignee: Varada Hemeswari > Labels: SECURE, WASB > Fix For: 2.8.0, 3.0.0-alpha3 > > Attachments: HADOOP-14091.001.patch, HADOOP-14091.002.patch > > > Currently org.apache.hadoop.fs.azure.Wasb provides AbstractFileSystem > implementation for 'wasb' scheme. > This task refers to providing AbstractFileSystem implementation for 'wasbs' > scheme -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14107) ITestS3GuardListConsistency fails intermittently
Mingliang Liu created HADOOP-14107: -- Summary: ITestS3GuardListConsistency fails intermittently Key: HADOOP-14107 URL: https://issues.apache.org/jira/browse/HADOOP-14107 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Mingliang Liu {code} mvn -Dit.test='ITestS3GuardListConsistency' -Dtest=none -Dscale -Ds3guard -Ddynamo -q clean verify --- T E S T S --- Running org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.544 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency testListStatusWriteBack(org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency) Time elapsed: 3.147 sec <<< FAILURE! java.lang.AssertionError: Unexpected number of results from metastore. Metastore should only know about /XYZ: DirListingMetadata{path=s3a://mliu-s3guard/test/ListStatusWriteBack, listMap={s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/XYZ; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}, s3a://mliu-s3guard/test/ListStatusWriteBack/123=PathMetadata{fileStatus=S3AFileStatus{path=s3a://mliu-s3guard/test/ListStatusWriteBack/123; isDirectory=true; modification_time=0; access_time=0; owner=mliu; group=mliu; permission=rwxrwxrwx; isSymlink=false} isEmptyDirectory=true}}, isAuthoritative=false} at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.assertTrue(Assert.java:41) at org.apache.hadoop.fs.s3a.ITestS3GuardListConsistency.testListStatusWriteBack(ITestS3GuardListConsistency.java:127) {code} See discussion on the parent JIRA [HADOOP-13345]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14106) Fix deprecated FileSystem inefficient calls in unit test
Mingliang Liu created HADOOP-14106: -- Summary: Fix deprecated FileSystem inefficient calls in unit test Key: HADOOP-14106 URL: https://issues.apache.org/jira/browse/HADOOP-14106 Project: Hadoop Common Issue Type: Improvement Reporter: Mingliang Liu [HADOOP-13321] deprecates FileSystem APIs that promote inefficient call patterns. There are existing code patterns in tests and this JIRA is to address them. {code} [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractFSContractTestBase.java:[372,10] [deprecation] isDirectory(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[249,14] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[251,16] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[264,14] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[266,16] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[280,14] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[282,16] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[288,14] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[290,16] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[306,14] [deprecation] isDirectory(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[308,16] [deprecation] isDirectory(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[313,14] [deprecation] isDirectory(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[315,16] [deprecation] isDirectory(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[340,14] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[342,16] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[351,14] [deprecation] isDirectory(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[353,16] [deprecation] isDirectory(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[400,14] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[759,14] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java:[761,16] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java:[124,18] [deprecation] isFile(Path) in FileSystem has been deprecated [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContract
[jira] [Created] (HADOOP-14102) Relax error message assertion in S3A test ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey
Mingliang Liu created HADOOP-14102: -- Summary: Relax error message assertion in S3A test ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey Key: HADOOP-14102 URL: https://issues.apache.org/jira/browse/HADOOP-14102 Project: Hadoop Common Issue Type: Test Components: fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Trivial [HADOOP-13075] added support for SSE-KMS and SSE-C in s3a filesystem, along with integration test {{ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey}}. For {{AccessDeniedException}} test case, it assumes the error message contains string _Forbidden (Service: Amazon S3; Status Code: 403;_, which is true in the current AWS java sdk and current S3AFileSystem code path. When enabling S3Guard (see feature JIRA [HADOOP-13345]), the code path that fails in {{S3AFileSystem}} changes for that test. Specifically, the request w/o S3Guard was calling {{getFileStatus()}} and fails with access denied exception containing _Forbidden_ keyword; while the request w/ S3Guard is able to call {{getFileStatus()}} successfully and then fails later for read operations with access denied exception containing _Access Denied_ keyword. AWS sdk does not provide the exactly same error message for different AccessDeniedExceptions. In the meantime, the AWS sdk may evolve (as we have been upgrading the sdk version recently in a timely manner) and the error message may be different in the future. This is to relax exception message assertion in test. Thanks [~fabbri] for discussion. See [HADOOP-13075]. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14079) Fix breaking link in s3guard.md
Mingliang Liu created HADOOP-14079: -- Summary: Fix breaking link in s3guard.md Key: HADOOP-14079 URL: https://issues.apache.org/jira/browse/HADOOP-14079 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Trivial See the initial patch. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13960) Initialize DynamoDBMetadataStore without associated S3AFileSystem
Mingliang Liu created HADOOP-13960: -- Summary: Initialize DynamoDBMetadataStore without associated S3AFileSystem Key: HADOOP-13960 URL: https://issues.apache.org/jira/browse/HADOOP-13960 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Reporter: Mingliang Liu Assignee: Mingliang Liu Per the discussion in [HADOOP-13650], it's helpful to initialize a DynamoDBMetadataStore object without S3AFileSystem. In the current code, we can achieve this via {{DynamoDBMetadataStore#initialize(Configuration)}}. However, users still have to provide the associated S3AFileSystem URI in the configuration, by means of either setting the {{fs.defaultFS}} in configuration file or {{-fs s3://bucket}} command line parameter. Setting the default FileSystem in configuration seems not necessary as the command line is to manipulate metadata store, e.g. command line tools on an existing HDFS cluster. This JIRA is to track the effort of initializing a DynamoDBMetadataStore without associating any S3 buckets, so that S3AFileSystem is not needed. Users have to specify in configuration the DynamoDB endpoints (for region) and table name along with credentials, AWS client settings. See [~eddyxu] and [~liuml07]'s comments in [HADOOP-13650] for more details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13946) Document how HDFS updates timestamps in the FS spec; compare with object stores
[ https://issues.apache.org/jira/browse/HADOOP-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-13946: > Document how HDFS updates timestamps in the FS spec; compare with object > stores > --- > > Key: HADOOP-13946 > URL: https://issues.apache.org/jira/browse/HADOOP-13946 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation, fs >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13946-001.patch > > > SPARK-17159 shows that the behavior of when HDFS updates timestamps isn't > well documented. Document these in the FS spec. > I'm not going to add tests for this, as it is so very dependent on FS > implementations, as in "POSIX filesystems may behave differently from HDFS". > If someone knows what happens there, their contribution is welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-13938) Address javadoc errors in Azure WASB.
[ https://issues.apache.org/jira/browse/HADOOP-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-13938. Resolution: Duplicate > Address javadoc errors in Azure WASB. > - > > Key: HADOOP-13938 > URL: https://issues.apache.org/jira/browse/HADOOP-13938 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth > > As observed in HADOOP-13863, there are few javadocs error that are thrown > while building WASB in QA builds. This JIRA is created to track the fix for > these javadoc errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13937) Mock bucket locations in MockS3ClientFactory
Mingliang Liu created HADOOP-13937: -- Summary: Mock bucket locations in MockS3ClientFactory Key: HADOOP-13937 URL: https://issues.apache.org/jira/browse/HADOOP-13937 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Minor Currently the MockS3ClientFactory}} does not mock the bucket locations. One effect is that {{TestDynamoDBMetadataStore}} will have null region for the bucket. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13908) Existing tables may not be initialized correctly in DynamoDBMetadataStore
Mingliang Liu created HADOOP-13908: -- Summary: Existing tables may not be initialized correctly in DynamoDBMetadataStore Key: HADOOP-13908 URL: https://issues.apache.org/jira/browse/HADOOP-13908 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: HADOOP-13345 Reporter: Mingliang Liu Assignee: Mingliang Liu This was based on discussion in [HADOOP-13455]. Though we should not create table unless the config {{fs.s3a.s3guard.ddb.table.create}} is set true, we still have to get the existing table in {{DynamoDBMetadataStore#initialize()}} and wait for its becoming active, before any table/item operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13813) TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer is failing
Mingliang Liu created HADOOP-13813: -- Summary: TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer is failing Key: HADOOP-13813 URL: https://issues.apache.org/jira/browse/HADOOP-13813 Project: Hadoop Common Issue Type: Bug Reporter: Mingliang Liu Assignee: Mingliang Liu [HADOOP-13720] added more info to the msgs printed in {{AbstractDelegationTokenSecretManager}} for better supportability, which is good. Unfortunately the unit test {{TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer}} that asserts the message string was not updated accordingly. The unit test is failing in both {{trunk}} and {{branch-2}} branches, see example builds [1|https://issues.apache.org/jira/browse/HDFS-11129?focusedCommentId=15657488&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15657488], [2|https://issues.apache.org/jira/browse/HDFS-11130?focusedCommentId=15658086&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15658086], and [3|https://issues.apache.org/jira/browse/HDFS-7?focusedCommentId=15656939&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15656939]. This JIRA is to track the effort of fixing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13697) LogLevel#main throws exception if no arguments provided
Mingliang Liu created HADOOP-13697: -- Summary: LogLevel#main throws exception if no arguments provided Key: HADOOP-13697 URL: https://issues.apache.org/jira/browse/HADOOP-13697 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.9.0 Reporter: Mingliang Liu Assignee: Mingliang Liu {code} root@b9ab37566005:/# hadoop daemonlog Usage: General options are: [-getlevel [-protocol (http|https)] [-setlevel[-protocol (http|https)] Exception in thread "main" org.apache.hadoop.HadoopIllegalArgumentException: No arguments specified at org.apache.hadoop.log.LogLevel$CLI.parseArguments(LogLevel.java:138) at org.apache.hadoop.log.LogLevel$CLI.run(LogLevel.java:106) at org.apache.hadoop.log.LogLevel.main(LogLevel.java:70) {code} I think we can catch the exception in the main method, and dump a log error message instead of throw the stack which may frustrate users. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-11513) Artifact errors with Maven build
[ https://issues.apache.org/jira/browse/HADOOP-11513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-11513. Resolution: Cannot Reproduce Feel free to re-open if it happens again. Thanks. > Artifact errors with Maven build > > > Key: HADOOP-11513 > URL: https://issues.apache.org/jira/browse/HADOOP-11513 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 2.7.0 >Reporter: Arpit Agarwal > > I recently started getting the following errors with _mvn -q clean compile > install_ on Linux and OS X. > {code} > [ERROR] Artifact: org.xerial.snappy:snappy-java:jar:1.0.4.1 has no file. > [ERROR] Artifact: xerces:xercesImpl:jar:2.9.1 has no file. > [ERROR] Artifact: xml-apis:xml-apis:jar:1.3.04 has no file. > [ERROR] Artifact: xmlenc:xmlenc:jar:0.52 has no file. > [ERROR] Artifact: org.xerial.snappy:snappy-java:jar:1.0.4.1 has no file. > [ERROR] Artifact: xerces:xercesImpl:jar:2.9.1 has no file. > [ERROR] Artifact: xml-apis:xml-apis:jar:1.3.04 has no file. > [ERROR] Artifact: xmlenc:xmlenc:jar:0.52 has no file. > {code} > _mvn --version_ on Linux reports: > {code} > Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; > 2014-12-14T09:29:23-08:00) > Maven home: /home/vagrant/usr/share/maven > Java version: 1.7.0_65, vendor: Oracle Corporation > Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "3.13.0-24-generic", arch: "amd64", family: "unix" > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13644) Replace config key literal strings with config key names
Mingliang Liu created HADOOP-13644: -- Summary: Replace config key literal strings with config key names Key: HADOOP-13644 URL: https://issues.apache.org/jira/browse/HADOOP-13644 Project: Hadoop Common Issue Type: Bug Reporter: Mingliang Liu Assignee: Chen Liang Priority: Minor There are some places that use config key literal strings instead of config key names, e.g. {code:title=IOUtils.java} copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true); {code} We should replace places like this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-12642) Update documentation to cover fs.s3.buffer.dir enhancements
[ https://issues.apache.org/jira/browse/HADOOP-12642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-12642. Resolution: Implemented Resolving this issue as [HADOOP-12982] covered this. Feel free to re-open if not. > Update documentation to cover fs.s3.buffer.dir enhancements > --- > > Key: HADOOP-12642 > URL: https://issues.apache.org/jira/browse/HADOOP-12642 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: 2.6.0 >Reporter: Jason Archip >Priority: Minor > > Could you please update the documentation at > https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html > to include the options for the updated fs.s3.buffer.dir > Please let me know if there is a different place to put my request > Thanks, > Jason Archip -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13621) s3:// should have been fully cut off from trunk
Mingliang Liu created HADOOP-13621: -- Summary: s3:// should have been fully cut off from trunk Key: HADOOP-13621 URL: https://issues.apache.org/jira/browse/HADOOP-13621 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 3.0.0-alpha1 Reporter: Mingliang Liu Assignee: Mingliang Liu [HADOOP-12709] should have cut off S3 fully from the {{trunk}} branch for Hadoop 3. However, during the long rebasing period, there are some omissions left in documentation aka {{hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md}}. This jira is to address this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13512) ReloadingX509TrustManager should keep reloading in case of exception
Mingliang Liu created HADOOP-13512: -- Summary: ReloadingX509TrustManager should keep reloading in case of exception Key: HADOOP-13512 URL: https://issues.apache.org/jira/browse/HADOOP-13512 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu {{org.apache.hadoop.security.ssl.TestReloadingX509TrustManager}} checks the key store file's last modified time to decide whether to reload. This is to avoid unnecessary reload if the key store file is not changed. To do this, it maintains an internal state {{lastLoaded}} whenever it tries to reload a file. It also updates the {{lastLoaded}} variable in case of exception so failing reload will not be retried until the key store file's last modified time changes again. Chances are that the reload happens when the key store file is being written. The reload fails (probably with EOFException) and won't load until key store files's last modified time changes. After a short period, the key store file is closed after update. However, the last modified time may not be updated as if it's in the same precision period (e.g. 1 second). In this case, the updated key store file is never reloaded. A simple fix is to update the {{lastLoaded}} only when the reload succeeds. {{ReloadingX509TrustManager}} will keep reloading in case of exception. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-13470) GenericTestUtils$LogCapturer is flaky
[ https://issues.apache.org/jira/browse/HADOOP-13470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu reopened HADOOP-13470: > GenericTestUtils$LogCapturer is flaky > - > > Key: HADOOP-13470 > URL: https://issues.apache.org/jira/browse/HADOOP-13470 > Project: Hadoop Common > Issue Type: Bug > Components: test, util >Affects Versions: 2.8.0 >Reporter: Mingliang Liu >Assignee: Mingliang Liu > Labels: reviewed > Fix For: 2.8.0 > > Attachments: HADOOP-13470.000.patch, HADOOP-13470.001.patch > > > {{GenericTestUtils$LogCapturer}} is useful for assertions against service > logs. However it should be fixed in following aspects: > # In the constructor, it uses the stdout appender's layout. > {code} > Layout layout = Logger.getRootLogger().getAppender("stdout").getLayout(); > {code} > However, the stdout appender may be named "console" or alike which makes the > constructor throw NPE. Actually the layout does not matter and we can use a > default pattern layout that only captures application logs. > # {{stopCapturing()}} method is not working. The major reason is that the > {{appender}} internal variable is never assigned and thus removing it to stop > capturing makes no sense. > # It does not support {{org.slf4j.Logger}} which is preferred to log4j in > many modules. > # There is no unit test for it. > This jira is to address these. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13470) GenericTestUtils$LogCapturer is flaky
Mingliang Liu created HADOOP-13470: -- Summary: GenericTestUtils$LogCapturer is flaky Key: HADOOP-13470 URL: https://issues.apache.org/jira/browse/HADOOP-13470 Project: Hadoop Common Issue Type: Bug Components: test, util Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu {{GenericTestUtils$LogCapturer}} is useful for assertions against service logs. However it should be fixed in following aspects: # In the constructor, it uses the stdout appender's layout. {code} Layout layout = Logger.getRootLogger().getAppender("stdout").getLayout(); {code} However, the stdout appender may be named "console" or alike which makes the constructor throw NPE. Actually the layout does not matter and we can use a default pattern layout that only captures application logs. # {{stopCapturing()}} method is not working. The major reason is that the {{appender}} internal variable is never assigned and thus removing it to stop capturing makes no sense. # It does not support {{org.slf4j.Logger}} which is preferred to log4j in many modules. # There is no unit test for it. This jira is to address these. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats
Mingliang Liu created HADOOP-13435: -- Summary: Add thread local mechanism for aggregating file system storage stats Key: HADOOP-13435 URL: https://issues.apache.org/jira/browse/HADOOP-13435 Project: Hadoop Common Issue Type: Sub-task Components: fs Reporter: Mingliang Liu Assignee: Mingliang Liu As discussed in [HADOOP-13032], this is to add thread local mechanism for aggregating file system storage stats. This class will also be used in [HADOOP-13031], which is to separate the distance-oriented rack-aware read bytes logic from {{FileSystemStorageStatistics}} to new DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the {{FileSystemStorageStatistics}} can live without the to-be-removed {{FileSystem$Statistics}} implementation. A unit test should also be added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13368) DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation
Mingliang Liu created HADOOP-13368: -- Summary: DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation Key: HADOOP-13368 URL: https://issues.apache.org/jira/browse/HADOOP-13368 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu To lookup, {{DFSOpsCountStatistics$OpType#fromSymbol}} and {{s3a.Statistic#fromSymbol}} iterates all the enums to get the entry by its symbol. Usages of {{fromSymbol()}} include {{isTracked()}} and {{getLong()}}. As there are dozens of enum entries, it merits to make these two similar operations O(1) complexity. This point is especially true if downstream app probes a dozen of stats in an outer loop (see [TEZ-3331]). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13305) Define common statistics names across schemes
Mingliang Liu created HADOOP-13305: -- Summary: Define common statistics names across schemes Key: HADOOP-13305 URL: https://issues.apache.org/jira/browse/HADOOP-13305 Project: Hadoop Common Issue Type: Sub-task Components: fs Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu The {{StorageStatistics}} provides a pretty general interface, i.e. {{getLong(name)}} and {{getLongStatistics()}}. There is no shared or standard names for the storage statistics and thus the getLong(name) is up to the implementation of storage statistics. The problems: # For the common statistics, downstream applications expect the same statistics name across different storage statistics and/or file system schemes. Chances are they have to use {{DFSOpsCountStorageStatistics#getLong(“getStatus”)}} and {{S3A.Statistics#getLong(“get_status”)}} for retrieving the getStatus operation stat. # Moreover, probing per-operation stats is hard if there is no standard/shared common names. It makes a lot of sense for different schemes to issue the per-operation stats of the same name. Meanwhile, every FS will have its own internal things to count, which can't be centrally defined or managed. But there are some common which would be easier managed if they all had the same name. Another motivation is that having a common set of names here will encourage uniform instrumentation of all filesystems; it will also make it easier to analyze the output of runs, were the stats to be published to a "performance log" similar to the audit log. See Steve's work for S3 (e.g. [HADOOP-13171]) This jira is track the effort of defining common StorageStatistics entry names. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13291) DFSOpsCountStatistics/S3AStorageStatistics should implement isTracked() correctly
Mingliang Liu created HADOOP-13291: -- Summary: DFSOpsCountStatistics/S3AStorageStatistics should implement isTracked() correctly Key: HADOOP-13291 URL: https://issues.apache.org/jira/browse/HADOOP-13291 Project: Hadoop Common Issue Type: Sub-task Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Minor {{StorageStatistics#isTracked()}} is a simple and useful API for probing a stat with key name. Currently {{DFSOpsCountStatistics}} implements this method wrongly. {{S3AStorageStatistics}} borrowed the same idea and also has the same error. This jira is to make simple things right. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13288) Guard null stats key in FileSystemStorageStatistics
Mingliang Liu created HADOOP-13288: -- Summary: Guard null stats key in FileSystemStorageStatistics Key: HADOOP-13288 URL: https://issues.apache.org/jira/browse/HADOOP-13288 Project: Hadoop Common Issue Type: Sub-task Reporter: Mingliang Liu Assignee: Mingliang Liu Currently in {{FileSystemStorageStatistics}} we simply returns data from {{FileSystem#Statistics}}. However there is no null key check, which leads to NPE problems to downstream applications. For example, we got a NPE when passing a null key to {{FileSystemStorageStatistics#getLong()}}, exception stack as following: {quote} NullPointerException at org.apache.hadoop.fs.FileSystemStorageStatistics.fetch(FileSystemStorageStatistics.java:80) at org.apache.hadoop.fs.FileSystemStorageStatistics.getLong(FileSystemStorageStatistics.java:108) at org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:60) at org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118) at org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {quote} This jira is to add null stat key check to {{FileSystemStorageStatistics}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13284) Remove the rack-aware read stats in FileSystemStorageStatistics from branch-2
Mingliang Liu created HADOOP-13284: -- Summary: Remove the rack-aware read stats in FileSystemStorageStatistics from branch-2 Key: HADOOP-13284 URL: https://issues.apache.org/jira/browse/HADOOP-13284 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu As [HDFS-9579] was not committed to {{branch-2.8}}, {{FileSystemStorageStatistics#KEYS}} should not include those rack aware read stats brought by [HDFS-9579], including {{bytesReadLocalHost, bytesReadDistanceOfOneOrTwo, bytesReadDistanceOfThreeOrFour, bytesReadDistanceOfFiveOrLarger}}. Or else, the long iterator will throw NPE when traversing. See detailed exception stack as following (it happens when Tez uses the new FileSystemStorageStatistics). {code} 2016-06-15 15:56:59,242 [DEBUG] [TezChild] |impl.TezProcessorContextImpl|: Cleared TezProcessorContextImpl related information 2016-06-15 15:56:59,243 [WARN] [main] |task.TezTaskRunner2|: Exception from RunnerCallable java.lang.NullPointerException at org.apache.hadoop.fs.FileSystemStorageStatistics$LongStatisticIterator.next(FileSystemStorageStatistics.java:74) at org.apache.hadoop.fs.FileSystemStorageStatistics$LongStatisticIterator.next(FileSystemStorageStatistics.java:51) at org.apache.tez.runtime.metrics.FileSystemStatisticsUpdater2.updateCounters(FileSystemStatisticsUpdater2.java:51) at org.apache.tez.runtime.metrics.TaskCounterUpdater.updateCounters(TaskCounterUpdater.java:118) at org.apache.tez.runtime.RuntimeTask.setFrameworkCounters(RuntimeTask.java:172) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:100) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-06-15 15:56:59,245 [DEBUG] [main] |task.TaskReporter|: Sending heartbeat to AM, request={ containerId=container_1466028486194_0005_01_02, requestId=10, startIndex=0, preRoutedStartIndex=1, maxEventsToGet=500, taskAttemptId=attempt_1466028486194_0005_1_00_00_0, eventCount=4 } {code} Thanks [~hitesh] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13283) Support reset operation for new global storage statistics and per FS storage stats
Mingliang Liu created HADOOP-13283: -- Summary: Support reset operation for new global storage statistics and per FS storage stats Key: HADOOP-13283 URL: https://issues.apache.org/jira/browse/HADOOP-13283 Project: Hadoop Common Issue Type: Sub-task Reporter: Mingliang Liu Assignee: Mingliang Liu Applications may reuse the file system object across jobs and its storage statistics should be reset. Specially the {{FileSystem.Statistics}} supports reset and [HADOOP-13032] needs to keep that use case valid. This jira is for supporting reset operations for storage statistics. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13280) FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps
Mingliang Liu created HADOOP-13280: -- Summary: FileSystemStorageStatistics#getLong(“readOps“) should return readOps + largeReadOps Key: HADOOP-13280 URL: https://issues.apache.org/jira/browse/HADOOP-13280 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu Currently {{FileSystemStorageStatistics}} instance simply returns data from {{FileSystem$Statistics}}. As to {{readOps}}, the {{FileSystem$Statistics#getReadOps()}} returns {{readOps + largeReadOps}}. We should make the {{FileSystemStorageStatistics#getLong(“readOps“)}} return the sum as well. Moreover, there is no unit tests for {{FileSystemStorageStatistics}} and this JIRA will also address this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13239) Deprecate s3:// in branch-2
Mingliang Liu created HADOOP-13239: -- Summary: Deprecate s3:// in branch-2 Key: HADOOP-13239 URL: https://issues.apache.org/jira/browse/HADOOP-13239 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.8.0 Reporter: Mingliang Liu Assignee: Mingliang Liu The fact that s3:// was broken in Hadoop 2.7 *and nobody noticed until now* shows that it's not being used. while invaluable at the time, s3n and especially s3a render it obsolete except for reading existing data. I propose # Mark Java source as {{@deprecated}} # Warn the first time in a JVM that an S3 instance is created, "deprecated -will be removed in future releases" # in Hadoop trunk we really cut it. Maybe have an attic project (external?) which holds it for anyone who still wants it. Or: retain the code but remove the {{fs.s3.impl}} config option, so you have to explicitly add it for use. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13032) Refactor FileSystem$Statistics as a separate source file
Mingliang Liu created HADOOP-13032: -- Summary: Refactor FileSystem$Statistics as a separate source file Key: HADOOP-13032 URL: https://issues.apache.org/jira/browse/HADOOP-13032 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Minor This jira is to track the effort of moving the {{Statistics}} class out of {{FileSystem}} for: # shorter source code ({{FileSystem.java}} is gonna have 3780 LoC after [HDFS-10175]) # simpler class structure (the {{Statistics}} is not very straightforward as it's optimized for performance in [HDFS-5276] and evloving rapidly recently, see [HDFS-9579], [HDFS-10175] and [MAPREDUCE-6660]). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-13031) Refactor the code that maintains rack-aware counters in FileSystem$Statistics
Mingliang Liu created HADOOP-13031: -- Summary: Refactor the code that maintains rack-aware counters in FileSystem$Statistics Key: HADOOP-13031 URL: https://issues.apache.org/jira/browse/HADOOP-13031 Project: Hadoop Common Issue Type: Improvement Reporter: Mingliang Liu Priority: Minor According to discussion in [HDFS-10175], using a composite (e.g. enum map, array) data structure to manage the distance->bytesRead mapping will probably make the code simpler. # {{StatisticsData}} will be a bit shorter by delegating the operations to the composite data structure. # The {{incrementBytesReadByDistance(int distance, long newBytes)}} and {{getBytesReadByDistance(int distance)}} which switch-case all hard-code variables, may be simplified as we can set/get the {{bytesRead}} by distance directly from map/array. This jira is to track the discussion and effort of refactoring the code that maintains rack-aware counters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12849) TestSymlinkLocalFSFileSystem fails intermittently
Mingliang Liu created HADOOP-12849: -- Summary: TestSymlinkLocalFSFileSystem fails intermittently Key: HADOOP-12849 URL: https://issues.apache.org/jira/browse/HADOOP-12849 Project: Hadoop Common Issue Type: Bug Components: test Reporter: Mingliang Liu *Error Message* expected:<1456523612000> but was:<1456523613000> *Stacktrace* java.lang.AssertionError: expected:<1456523612000> but was:<1456523613000> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.fs.SymlinkBaseTest.testSetTimesDanglingLink(SymlinkBaseTest.java:1410) at org.apache.hadoop.fs.TestSymlinkLocalFS.testSetTimesDanglingLink(TestSymlinkLocalFS.java:239) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) It happens in recent builds: * https://builds.apache.org/job/PreCommit-HADOOP-Build/8732/testReport/org.apache.hadoop.fs/TestSymlinkLocalFSFileSystem/testSetTimesDanglingLink/ * https://builds.apache.org/job/PreCommit-HADOOP-Build/8721/testReport/org.apache.hadoop.fs/TestSymlinkLocalFSFileSystem/testSetTimesSymlinkToFile/ * https://builds.apache.org/job/PreCommit-HADOOP-Build/8590/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_72.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12514) Make
Mingliang Liu created HADOOP-12514: -- Summary: Make Key: HADOOP-12514 URL: https://issues.apache.org/jira/browse/HADOOP-12514 Project: Hadoop Common Issue Type: Task Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Minor This is a follow up of [HADOOP-12472]. It makes sense to make the following static fields package private instead of protected, as they are for test purpose (protected keyword makes more sense to sub-classes). - protected static String E_NULL_THROWABLE = "Null Throwable"; - protected static String E_NULL_THROWABLE_STRING = "Null Throwable.toString() value"; - protected static String E_UNEXPECTED_EXCEPTION = "but got unexpected exception"; Meanwhile, we may need to make them final. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12474) Fix data race when allocating server port in MiniKMS
Mingliang Liu created HADOOP-12474: -- Summary: Fix data race when allocating server port in MiniKMS Key: HADOOP-12474 URL: https://issues.apache.org/jira/browse/HADOOP-12474 Project: Hadoop Common Issue Type: Bug Reporter: Mingliang Liu Assignee: Mingliang Liu In {{MiniKMS}}, there is potential data race when create jetty server in {{createJettyServer}}. It looks like the code searches for a free port and then starts jetty, but maybe there's enough of a race condition between port location and jetty.start to cause intermittent failures. {code} ServerSocket ss = new ServerSocket((inPort < 0) ? 0 : inPort, 50, localhost); int port = ss.getLocalPort(); Server server = new Server(0); if (!ssl) { server.getConnectors()[0].setHost(host); server.getConnectors()[0].setPort(port); } else { ... c.setPort(port); {code} We've seen test failures saying {{java.net.BindException: Address already in use}}, e.g. [https://builds.apache.org/job/PreCommit-HDFS-Build/12942/testReport/] As in [HADOOP-12417], we should always bind port 0 which gives us an ephemeral port, instead of searching a free port before starting jetty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12473) distcp's ignoring failures should be mutually exclusive with the atomic option
Mingliang Liu created HADOOP-12473: -- Summary: distcp's ignoring failures should be mutually exclusive with the atomic option Key: HADOOP-12473 URL: https://issues.apache.org/jira/browse/HADOOP-12473 Project: Hadoop Common Issue Type: Bug Components: tools/distcp Affects Versions: 2.7.1 Reporter: Mingliang Liu Assignee: Mingliang Liu Priority: Critical Fix For: 2.8.0 {{RetriableFileCopyCommand.CopyReadException}} is double-wrapped via # via {{RetriableCommand::execute}} # via {{CopyMapper#copyFileWithRetry}} before {{CopyMapper::handleFailure}} tests {code} if (ignoreFailures && exception.getCause() instanceof RetriableFileCopyCommand.CopyReadException {code} which is always false. Orthogonally, ignoring failures should be mutually exclusive with the atomic option otherwise an incomplete dir is eligible for commit defeating the purpose. -- This message was sent by Atlassian JIRA (v6.3.4#6332)