[jira] [Created] (HADOOP-11486) org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken.testHttpUGI fails.
Spandan Dutta created HADOOP-11486: -- Summary: org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken.testHttpUGI fails. Key: HADOOP-11486 URL: https://issues.apache.org/jira/browse/HADOOP-11486 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Spandan Dutta The jenkins job which was setup gave the following stack trace. java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376) at java.net.ServerSocket.bind(ServerSocket.java:376) at java.net.ServerSocket.(ServerSocket.java:237) at org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80) at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73) at org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283) at org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.Server.doStart(Server.java:235) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken.testHttpUGI(TestWebDelegationToken.java:934) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10643) Add NativeS3Fs that delgates calls from FileContext apis to native s3 fs implementation
[
https://issues.apache.org/jira/browse/HADOOP-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280942#comment-14280942
]
Sumit Kumar commented on HADOOP-10643:
--
[~ste...@apache.org] I'm assuming your concern is on this portion of the patch
(as part of AbstractFileSystem.java):
{noformat}
-// A file system implementation that requires authority must always
-// specify default port
-if (defaultPort < 0 && authorityNeeded) {
- throw new HadoopIllegalArgumentException(
- "FileSystem implementation error - default port " + defaultPort
- + " is not valid");
-}
{noformat}
If so, s3's urls have a specific requirement that they don't contain any port
(so defaultPort becomes -1 in this case) and they don't have any authority in
the url as well. Does this work?
> Add NativeS3Fs that delgates calls from FileContext apis to native s3 fs
> implementation
> ---
>
> Key: HADOOP-10643
> URL: https://issues.apache.org/jira/browse/HADOOP-10643
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: 2.4.0
>Reporter: Sumit Kumar
>Assignee: Sumit Kumar
> Attachments: HADOOP-10643.patch
>
>
> The new set of file system related apis (FileContext/AbstractFileSystem)
> already support local filesytem, hdfs, viewfs) however they don't support
> s3n. This patch is to add that support using configurations like
> fs.AbstractFileSystem.s3n.impl = org.apache.hadoop.fs.s3native.NativeS3Fs
> This patch however doesn't provide a new implementation, instead relies on
> DelegateToFileSystem abstract class to delegate all calls from FileContext
> apis for s3n to the NativeS3FileSystem implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11364) [Java 8] Over usage of virtual memory
[ https://issues.apache.org/jira/browse/HADOOP-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280923#comment-14280923 ] Mohammad Kamrul Islam commented on HADOOP-11364: Sorry [~jira.shegalov] for the late reply. The failure was coming from distcp command which uses 1GB as mapreduce.map.memory.mb. I think distcp is map-only job. But in other cases, we used higher memory.mb (2GB) and got the similar exception with max 4.2 GB VM. > [Java 8] Over usage of virtual memory > - > > Key: HADOOP-11364 > URL: https://issues.apache.org/jira/browse/HADOOP-11364 > Project: Hadoop Common > Issue Type: Bug >Reporter: Mohammad Kamrul Islam >Assignee: Mohammad Kamrul Islam > > In our Hadoop 2 + Java8 effort , we found few jobs are being Killed by Hadoop > due to excessive virtual memory allocation. Although the physical memory > usage is low. > The most common error message is "Container [pid=??,containerID=container_??] > is running beyond virtual memory limits. Current usage: 365.1 MB of 1 GB > physical memory used; 3.2 GB of 2.1 GB virtual memory used. Killing > container." > We see this problem for MR job as well as in spark driver/executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11479) hdfs crypto -createZone fails to impersonate the real user in a kerberised environment
[ https://issues.apache.org/jira/browse/HADOOP-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280870#comment-14280870 ] Charles Lamb commented on HADOOP-11479: --- [~ranadip], I am pretty sure that the READ exception is being thrown when the NN is doing getMetadata. Here's the explanation: There are KMS operation ACLs (hadoop.kms...) and KMS key ACLs (default.key.acl...). The KMS key ACLs are more coarse-grained (MANAGEMENT, GENERATE_EEK, DECRYPT_EEK, READ, ALL) than the KMS operation ACLs (which cover each public KMS method call individually). So, by default, the HDFS user has READ permission on all keys (default.key.acl.READ=*). This gives that user access to the getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata and getCurrentKey methods. But, then, to lock down HDFS user access to key material, you also need to add the HDFS user to the blacklist for the following KMS operation ACLs: CREATE, DELETE, ROLLOVER, GET, GET_KEYS, SET_KEY_MATERIAL, DECRYPT_EEK (This is what setting the KMS Blacklist property in CM does: it is a shortcut to setting these seven KMS operation ACLs) There is also a specific KMS operation ACL for GET_METADATA, but you don't want to set that in this case. > hdfs crypto -createZone fails to impersonate the real user in a kerberised > environment > -- > > Key: HADOOP-11479 > URL: https://issues.apache.org/jira/browse/HADOOP-11479 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 > Environment: CentOS >Reporter: Ranadip > Attachments: KMS-test-acl.txt > > > The problem occurs when KMS key level acl is created for the key before the > encryption zone is created. The command tried to create the encryption zone > using "hdfs" user's identity and not the real user's identity. > Steps: > In a kerberised environment: > 1. Create key level ACL in KMS for a new key. > 2. Create encryption key now. (Goes through fine) > 3. Create encryption zone. (Fails) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11171) Enable using a proxy server to connect to S3a.
[ https://issues.apache.org/jira/browse/HADOOP-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-11171: --- Attachment: HADOOP-11171-10.patch By default failed connections are retried 10 times. Thi spatch reduces this to 2 for the proxy tests causing the runtime to decrease from 5 mins to 8 secs. The huge difference in runtime is caused by the exponential backoff strategy used by the underlying aws-sdk. > Enable using a proxy server to connect to S3a. > -- > > Key: HADOOP-11171 > URL: https://issues.apache.org/jira/browse/HADOOP-11171 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.4.0 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Attachments: HADOOP-11171-10.patch, HADOOP-11171-2.patch, > HADOOP-11171-3.patch, HADOOP-11171-4.patch, HADOOP-11171-5.patch, > HADOOP-11171-6.patch, HADOOP-11171-7.patch, HADOOP-11171-8.patch, > HADOOP-11171-9.patch, HADOOP-11171.patch > > > This exposes the AWS SDK config for a proxy (host and port) to s3a through > config settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11460) Deprecrate shell vars
[
https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280825#comment-14280825
]
Hadoop QA commented on HADOOP-11460:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692800/HADOOP-11460-01.patch
against trunk revision 000ca83.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.ha.TestZKFailoverControllerStress
org.apache.hadoop.hdfs.server.blockmanagement.TestDatanodeManager
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5418//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5418//console
This message is automatically generated.
> Deprecrate shell vars
> -
>
> Key: HADOOP-11460
> URL: https://issues.apache.org/jira/browse/HADOOP-11460
> Project: Hadoop Common
> Issue Type: New Feature
> Components: scripts
>Affects Versions: 3.0.0
>Reporter: Allen Wittenauer
> Labels: scripts, shell
> Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch
>
>
> It is a very common shell pattern in 3.x to effectively replace sub-project
> specific vars with generics. We should have a function that does this
> replacement and provides a warning to the end user that the old shell var is
> deprecated. Additionally, we should use this shell function to deprecate the
> shell vars that are holdovers already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-9248) Allow configuration of Amazon S3 Endpoint
[ https://issues.apache.org/jira/browse/HADOOP-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280707#comment-14280707 ] Jean-Martin Archer commented on HADOOP-9248: As far as I am concerned, the s3a implementation makes this feature request unnecessary for our use cases. > Allow configuration of Amazon S3 Endpoint > - > > Key: HADOOP-9248 > URL: https://issues.apache.org/jira/browse/HADOOP-9248 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 > Environment: All environments connecting to S3 >Reporter: Timur Perelmutov > > http://wiki.apache.org/hadoop/AmazonS3 page describes configuration of Hadoop > with S3 as storage. Other systems like EMC Atmos now implement S3 Interface, > but in order to be able to connect to them, the endpoint needs to be > configurable. Please add a configuration parameter that would be propagated > to underlying jets3t library as s3service.s3-endpoint param. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10181) GangliaContext does not work with multicast ganglia setup
[ https://issues.apache.org/jira/browse/HADOOP-10181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280617#comment-14280617 ] Laurie Denness commented on HADOOP-10181: - +1 for this one.. We have 350 nodes so they span two subnets/VLANs, but we forward the Ganglia multicast between them. We just upgraded to CDH5 (Hadoop 2.5) and now we've lost our Ganglia metrics for half of the cluster (because the TTL is set to 1) depending on which half of the cluster you ask (as they will not span the two subnets) (Given we now have a tonne of missing metrics I would say this is more than Minor, as the alternative is use Unicast Ganglia and that is not ideal for other reasons) > GangliaContext does not work with multicast ganglia setup > - > > Key: HADOOP-10181 > URL: https://issues.apache.org/jira/browse/HADOOP-10181 > Project: Hadoop Common > Issue Type: Bug >Reporter: Andrew Otto >Priority: Minor > Labels: ganglia, hadoop, metrics, multicast > > The GangliaContext class which is used to send Hadoop metrics to Ganglia uses > a DatagramSocket to send these metrics. This works fine for Ganglia > multicast setups that are all on the same VLAN. However, when working with > multiple VLANs, a packet sent via DatagramSocket to a multicast address will > end up with a TTL of 1. Multicast TTL indicates the number of network hops > for which a particular multicast packet is valid. The packets sent by > GangliaContext do not make it to ganglia aggregrators on the same multicast > group, but in different VLANs. > To fix, we'd need a configuration property that specifies that multicast is > to be used, and another that allows setting of the multicast packet TTL. > With these set, we could then use MulticastSocket setTimeToLive() instead of > just plain ol' DatagramSocket. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11327) BloomFilter#not() omits the last bit, resulting in an incorrect filter
[
https://issues.apache.org/jira/browse/HADOOP-11327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280589#comment-14280589
]
Hadoop QA commented on HADOOP-11327:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692798/HADOOP-11327.v2.txt
against trunk revision 000ca83.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5419//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5419//console
This message is automatically generated.
> BloomFilter#not() omits the last bit, resulting in an incorrect filter
> --
>
> Key: HADOOP-11327
> URL: https://issues.apache.org/jira/browse/HADOOP-11327
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.5.1
>Reporter: Tim Luo
>Assignee: Eric Payne
>Priority: Minor
> Attachments: HADOOP-11327.v1.txt, HADOOP-11327.v2.txt
>
>
> There's an off-by-one error in {{BloomFilter#not()}}:
> {{BloomFilter#not}} calls {{BitSet#flip(0, vectorSize - 1)}}, but according
> to the javadoc for that method, {{toIndex}} is end-_exclusive_:
> {noformat}
> * @param toIndex index after the last bit to flip
> {noformat}
> This means that the last bit in the bit array is not flipped.
> Specifically, this was discovered in the following scenario:
> 1. A new/empty {{BloomFilter}} was created with vectorSize=7.
> 2. Invoke {{bloomFilter.not()}}; now expecting a bloom filter with all 7 bits
> (0 through 6) flipped to 1 and membershipTest(...) to always return true.
> 3. However, membershipTest(...) was found to often not return true, and upon
> inspection, the BitSet only had bits 0 through 5 flipped.
> The fix should be simple: remove the "- 1" from the call to {{BitSet#flip}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-10062) race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect results
[
https://issues.apache.org/jira/browse/HADOOP-10062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated HADOOP-10062:
-
Priority: Major (was: Minor)
> race condition in MetricsSystemImpl#publishMetricsNow that causes incorrect
> results
> ---
>
> Key: HADOOP-10062
> URL: https://issues.apache.org/jira/browse/HADOOP-10062
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 3.0.0
> Environment: CentOS 6.4, Oracle JDK 1.6.0_31, JDK1.7.0_45
>Reporter: Shinichi Yamashita
>Assignee: Sangjin Lee
> Attachments: HADOOP-10062-failed.txt, HADOOP-10062-success.txt,
> HADOOP-10062.003.patch, HADOOP-10062.patch, HADOOP-10062.patch
>
>
> TestMetricsSystemInpl#testMultiThreadedPublish failed with "Metrics not
> collected"
> {code}
> Running org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.688 sec <<<
> FAILURE! - in org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
> testMultiThreadedPublish(org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl)
> Time elapsed: 0.056 sec <<< FAILURE!
> java.lang.AssertionError: Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Passed
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at
> org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl.testMultiThreadedPublish(TestMetricsSystemImpl.java:232)
> Results :
> Failed tests:
> TestMetricsSystemImpl.testMultiThreadedPublish:232 Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Metric not collected!
> Passed
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11460) Deprecrate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11460: -- Status: Patch Available (was: Open) > Deprecrate shell vars > - > > Key: HADOOP-11460 > URL: https://issues.apache.org/jira/browse/HADOOP-11460 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer > Labels: scripts, shell > Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch > > > It is a very common shell pattern in 3.x to effectively replace sub-project > specific vars with generics. We should have a function that does this > replacement and provides a warning to the end user that the old shell var is > deprecated. Additionally, we should use this shell function to deprecate the > shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11460) Deprecrate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11460: -- Attachment: HADOOP-11460-01.patch -01: * Eliminate most (all?) of the usages of the deprecated env vars! > Deprecrate shell vars > - > > Key: HADOOP-11460 > URL: https://issues.apache.org/jira/browse/HADOOP-11460 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer > Labels: scripts, shell > Attachments: HADOOP-11460-00.patch, HADOOP-11460-01.patch > > > It is a very common shell pattern in 3.x to effectively replace sub-project > specific vars with generics. We should have a function that does this > replacement and provides a warning to the end user that the old shell var is > deprecated. Additionally, we should use this shell function to deprecate the > shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11460) Deprecrate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11460: -- Description: It is a very common shell pattern in 3.x to effectively replace sub-project specific vars with generics. We should have a function that does this replacement and provides a warning to the end user that the old shell var is deprecated. Additionally, we should use this shell function to deprecate the shell vars that are holdovers already. was:It is a very common shell pattern in 3.x to effectively replace sub-project specific vars with generics. We should have a function that does this replacement and provides a warning to the end user that the old shell var is deprecated. This will allow us an easier time to remove them in the future. > Deprecrate shell vars > - > > Key: HADOOP-11460 > URL: https://issues.apache.org/jira/browse/HADOOP-11460 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer > Labels: scripts, shell > Attachments: HADOOP-11460-00.patch > > > It is a very common shell pattern in 3.x to effectively replace sub-project > specific vars with generics. We should have a function that does this > replacement and provides a warning to the end user that the old shell var is > deprecated. Additionally, we should use this shell function to deprecate the > shell vars that are holdovers already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11171) Enable using a proxy server to connect to S3a.
[
https://issues.apache.org/jira/browse/HADOOP-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280510#comment-14280510
]
Steve Loughran commented on HADOOP-11171:
-
well, the tests pass, but they take ~5 minutes
{code}
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 295.421 sec -
in org.apache.hadoop.fs.s3a.TestS3AConfiguration
Results :
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 4:57.925s
{code}
Cause? Connection retries:
{code}
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
com.amazonaws.http.AmazonHttpClient.pauseBeforeNextRetry(AmazonHttpClient.java:864)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:353)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
at
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
at
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:282)
at
org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:48)
at
org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
Is there a way to tune the failure policy of the AWS lib so it doesn't retry on
these test runs?
> Enable using a proxy server to connect to S3a.
> --
>
> Key: HADOOP-11171
> URL: https://issues.apache.org/jira/browse/HADOOP-11171
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Affects Versions: 2.4.0
>Reporter: Thomas Demoor
>Assignee: Thomas Demoor
> Labels: amazon, s3
> Attachments: HADOOP-11171-2.patch, HADOOP-11171-3.patch,
> HADOOP-11171-4.patch, HADOOP-11171-5.patch, HADOOP-11171-6.patch,
> HADOOP-11171-7.patch, HADOOP-11171-8.patch, HADOOP-11171-9.patch,
> HADOOP-11171.patch
>
>
> This exposes the AWS SDK config for a proxy (host and port) to s3a through
> config settings.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11327) BloomFilter#not() omits the last bit, resulting in an incorrect filter
[
https://issues.apache.org/jira/browse/HADOOP-11327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated HADOOP-11327:
Attachment: HADOOP-11327.v2.txt
Thanks, [~jlowe], for the review and comments.
I have updated the test case in version 2 of the patch.
> BloomFilter#not() omits the last bit, resulting in an incorrect filter
> --
>
> Key: HADOOP-11327
> URL: https://issues.apache.org/jira/browse/HADOOP-11327
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Affects Versions: 2.5.1
>Reporter: Tim Luo
>Assignee: Eric Payne
>Priority: Minor
> Attachments: HADOOP-11327.v1.txt, HADOOP-11327.v2.txt
>
>
> There's an off-by-one error in {{BloomFilter#not()}}:
> {{BloomFilter#not}} calls {{BitSet#flip(0, vectorSize - 1)}}, but according
> to the javadoc for that method, {{toIndex}} is end-_exclusive_:
> {noformat}
> * @param toIndex index after the last bit to flip
> {noformat}
> This means that the last bit in the bit array is not flipped.
> Specifically, this was discovered in the following scenario:
> 1. A new/empty {{BloomFilter}} was created with vectorSize=7.
> 2. Invoke {{bloomFilter.not()}}; now expecting a bloom filter with all 7 bits
> (0 through 6) flipped to 1 and membershipTest(...) to always return true.
> 3. However, membershipTest(...) was found to often not return true, and upon
> inspection, the BitSet only had bits 0 through 5 flipped.
> The fix should be simple: remove the "- 1" from the call to {{BitSet#flip}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11460) Deprecrate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11460: -- Summary: Deprecrate shell vars (was: Add a way to deprecrate shell vars) > Deprecrate shell vars > - > > Key: HADOOP-11460 > URL: https://issues.apache.org/jira/browse/HADOOP-11460 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer > Labels: scripts, shell > Attachments: HADOOP-11460-00.patch > > > It is a very common shell pattern in 3.x to effectively replace sub-project > specific vars with generics. We should have a function that does this > replacement and provides a warning to the end user that the old shell var is > deprecated. This will allow us an easier time to remove them in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-10542) Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
[
https://issues.apache.org/jira/browse/HADOOP-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280446#comment-14280446
]
Ted Yu commented on HADOOP-10542:
-
If in is null, do we return an empty fileBlock or raise exception ?
I lean toward the latter. e.g. raise IOE with message saying object with key
blockToKey(block) wasn't found.
> Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
> ---
>
> Key: HADOOP-10542
> URL: https://issues.apache.org/jira/browse/HADOOP-10542
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.6.0
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> in = get(blockToKey(block), byteRangeStart);
> out = new BufferedOutputStream(new FileOutputStream(fileBlock));
> byte[] buf = new byte[bufferSize];
> int numRead;
> while ((numRead = in.read(buf)) >= 0) {
> {code}
> get() may return null.
> The while loop dereferences in without null check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-10533) S3 input stream NPEs in MapReduce job
[ https://issues.apache.org/jira/browse/HADOOP-10533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HADOOP-10533: Summary: S3 input stream NPEs in MapReduce job (was: S3 input stream NPEs in MapReduce jon) > S3 input stream NPEs in MapReduce job > - > > Key: HADOOP-10533 > URL: https://issues.apache.org/jira/browse/HADOOP-10533 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 1.0.0, 1.0.3, 3.0.0, 2.4.0 > Environment: Hadoop with default configurations >Reporter: Benjamin Kim >Assignee: Steve Loughran >Priority: Minor > Fix For: 2.5.0 > > > I'm running a wordcount MR as follows > hadoop jar WordCount.jar wordcount.WordCountDriver > s3n://bucket/wordcount/input s3n://bucket/wordcount/output > > s3n://bucket/wordcount/input is a s3 object that contains other input files. > However I get following NPE error > 12/10/02 18:56:23 INFO mapred.JobClient: map 0% reduce 0% > 12/10/02 18:56:54 INFO mapred.JobClient: map 50% reduce 0% > 12/10/02 18:56:56 INFO mapred.JobClient: Task Id : > attempt_201210021853_0001_m_01_0, Status : FAILED > java.lang.NullPointerException > at > org.apache.hadoop.fs.s3native.NativeS3FileSystem$NativeS3FsInputStream.close(NativeS3FileSystem.java:106) > at java.io.BufferedInputStream.close(BufferedInputStream.java:451) > at java.io.FilterInputStream.close(FilterInputStream.java:155) > at org.apache.hadoop.util.LineReader.close(LineReader.java:83) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:144) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:497) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > MR runs fine if i specify more specific input path such as > s3n://bucket/wordcount/input/file.txt > MR fails if I pass s3 folder as a parameter > In summary, > This works > hadoop jar ./hadoop-examples-1.0.3.jar wordcount > /user/hadoop/wordcount/input/ s3n://bucket/wordcount/output/ > This doesn't work > hadoop jar ./hadoop-examples-1.0.3.jar wordcount > s3n://bucket/wordcount/input/ s3n://bucket/wordcount/output/ > (both input path are directories) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11171) Enable using a proxy server to connect to S3a.
[ https://issues.apache.org/jira/browse/HADOOP-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Demoor updated HADOOP-11171: --- Attachment: HADOOP-11171-9.patch Rebased on trunk and reran the tests. Didn't apply as this shares the testclass with my endpoint feature that you merged in earlier today. > Enable using a proxy server to connect to S3a. > -- > > Key: HADOOP-11171 > URL: https://issues.apache.org/jira/browse/HADOOP-11171 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.4.0 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Attachments: HADOOP-11171-2.patch, HADOOP-11171-3.patch, > HADOOP-11171-4.patch, HADOOP-11171-5.patch, HADOOP-11171-6.patch, > HADOOP-11171-7.patch, HADOOP-11171-8.patch, HADOOP-11171-9.patch, > HADOOP-11171.patch > > > This exposes the AWS SDK config for a proxy (host and port) to s3a through > config settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11463) Replace method-local TransferManager object with S3AFileSystem#transfers
[
https://issues.apache.org/jira/browse/HADOOP-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280421#comment-14280421
]
Hadoop QA commented on HADOOP-11463:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692782/hadoop-11463-002.patch
against trunk revision 000ca83.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-aws.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5417//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/5417//console
This message is automatically generated.
> Replace method-local TransferManager object with S3AFileSystem#transfers
>
>
> Key: HADOOP-11463
> URL: https://issues.apache.org/jira/browse/HADOOP-11463
> Project: Hadoop Common
> Issue Type: Task
> Components: fs/s3
>Affects Versions: 2.7.0
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: hadoop-11463-001.patch, hadoop-11463-002.patch
>
>
> This is continuation of HADOOP-11446.
> The following changes are made according to Thomas Demoor's comments:
> 1. Replace method-local TransferManager object with S3AFileSystem#transfers
> 2. Do not shutdown TransferManager after purging existing multipart file -
> otherwise the current transfer is unable to proceed
> 3. Shutdown TransferManager instance in the close method of S3AFileSystem
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11350) The size of header buffer of HttpServer is too small when HTTPS is enabled
[
https://issues.apache.org/jira/browse/HADOOP-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280413#comment-14280413
]
Hudson commented on HADOOP-11350:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/])
HADOOP-11350. The size of header buffer of HttpServer is too small when HTTPS
is enabled. Contributed by Benoy Antony. (wheat9: rev
3ab3a6498812c9fa0c53dae02ce696033062af87)
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/HttpServerFunctionalTest.java
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
> The size of header buffer of HttpServer is too small when HTTPS is enabled
> --
>
> Key: HADOOP-11350
> URL: https://issues.apache.org/jira/browse/HADOOP-11350
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.7.0
>
> Attachments: HADOOP-11350.patch, HADOOP-11350.patch,
> HADOOP-11350.patch, HADOOP-11350.patch
>
>
> {code}
> curl -k -vvv -i -L --negotiate -u : https://:50070
> < HTTP/1.1 413 FULL head
> HTTP/1.1 413 FULL head
> < Connection: close
> Connection: close
> < Server: Jetty(6.1.26)
> Server: Jetty(6.1.26)
> {code}
> For some users, the spnego token too large for the default header buffer used
> by Jetty.
> Though the issue is fixed for HTTP connections (via HADOOP-8816), HTTPS
> connections needs to be fixed as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280410#comment-14280410 ] Hudson commented on HADOOP-11261: - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/]) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11318) Update the document for hadoop fs -stat
[
https://issues.apache.org/jira/browse/HADOOP-11318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280411#comment-14280411
]
Hudson commented on HADOOP-11318:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/])
HADOOP-11318. Update the document for hadoop fs -stat (aajisaka: rev
ce29074685abaa88ae9586fe8277aca23f9d54d6)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Stat.java
* hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
* hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
* hadoop-common-project/hadoop-common/CHANGES.txt
> Update the document for hadoop fs -stat
> ---
>
> Key: HADOOP-11318
> URL: https://issues.apache.org/jira/browse/HADOOP-11318
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: HADOOP-11318-001.patch, HADOOP-11318-002.patch,
> HADOOP-11318-002.patch, HADOOP-11318-003.patch
>
>
> In FileSystemShell.apt.vm,
> {code}
> stat
>Usage: <<>>
>Returns the stat information on the path.
> {code}
> Now {{-stat}} accepts the below formats.
> * %b: Size of file in blocks
> * %g: Group name of owner
> * %n: Filename
> * %o: Block size
> * %r: replication
> * %u: User name of owner
> * %y: UTC date as "-MM-dd HH:mm:ss"
> * %Y: Milliseconds since January 1, 1970 UTC
> They should be documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-8757) Metrics should disallow names with invalid characters
[ https://issues.apache.org/jira/browse/HADOOP-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280418#comment-14280418 ] Hudson commented on HADOOP-8757: FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/]) HADOOP-8757. Metrics should disallow names with invalid characters (rchiang via rkanter) (rkanter: rev b6ff9c03a4f8aba945e562a7ff60b0fc6a1cd040) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java > Metrics should disallow names with invalid characters > - > > Key: HADOOP-8757 > URL: https://issues.apache.org/jira/browse/HADOOP-8757 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Todd Lipcon >Assignee: Ray Chiang >Priority: Minor > Labels: newbie > Fix For: 2.7.0 > > Attachments: HADOOP-8757-01.patch, HADOOP-8757-02.patch, > HADOOP-8757.003.patch > > > Just spent a couple hours trying to figure out why a metric I added didn't > show up in JMX, only to eventually realize it was because I had a whitespace > in the property name. This didn't cause any errors to be logged -- the metric > just didn't show up in JMX. We should check that the name is valid and log an > error, or replace invalid characters with something like an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11479) hdfs crypto -createZone fails to impersonate the real user in a kerberised environment
[ https://issues.apache.org/jira/browse/HADOOP-11479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280359#comment-14280359 ] Ranadip commented on HADOOP-11479: -- Hi Charles, Yes, you are right on the user setup and namenode process is started by "hdfs". This is a kerberos enabled environment, so the real user is obtained from the Kerberos ticket. The Kerberos user, though not "hdfs" is a member of the hdfs supergroup. So, Hadoop should see the user as a superuser when executing this command. Moreover, the encryption zone is being created inside the Kerberos user's home directory where the user executing the command is in fact the owner of the EZ being created. However, providing key level access, specially READ access to an application user does ring some alarm bells. In effect, with this limitation that hdfs user needs those key accesses for all encryption zones, it means if the hdfs user credentials are compromised, the "baddie" can have access to the encrypted data as well (or am I missing something?) Should the process rather not use the real user (authenticated by Kerberos) from the UserGroupInformation check key level authorisations with KMS? > hdfs crypto -createZone fails to impersonate the real user in a kerberised > environment > -- > > Key: HADOOP-11479 > URL: https://issues.apache.org/jira/browse/HADOOP-11479 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.6.0 > Environment: CentOS >Reporter: Ranadip > Attachments: KMS-test-acl.txt > > > The problem occurs when KMS key level acl is created for the key before the > encryption zone is created. The command tried to create the encryption zone > using "hdfs" user's identity and not the real user's identity. > Steps: > In a kerberised environment: > 1. Create key level ACL in KMS for a new key. > 2. Create encryption key now. (Goes through fine) > 3. Create encryption zone. (Fails) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-8757) Metrics should disallow names with invalid characters
[ https://issues.apache.org/jira/browse/HADOOP-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280357#comment-14280357 ] Hudson commented on HADOOP-8757: FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/]) HADOOP-8757. Metrics should disallow names with invalid characters (rchiang via rkanter) (rkanter: rev b6ff9c03a4f8aba945e562a7ff60b0fc6a1cd040) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java > Metrics should disallow names with invalid characters > - > > Key: HADOOP-8757 > URL: https://issues.apache.org/jira/browse/HADOOP-8757 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Todd Lipcon >Assignee: Ray Chiang >Priority: Minor > Labels: newbie > Fix For: 2.7.0 > > Attachments: HADOOP-8757-01.patch, HADOOP-8757-02.patch, > HADOOP-8757.003.patch > > > Just spent a couple hours trying to figure out why a metric I added didn't > show up in JMX, only to eventually realize it was because I had a whitespace > in the property name. This didn't cause any errors to be logged -- the metric > just didn't show up in JMX. We should check that the name is valid and log an > error, or replace invalid characters with something like an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11318) Update the document for hadoop fs -stat
[
https://issues.apache.org/jira/browse/HADOOP-11318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280350#comment-14280350
]
Hudson commented on HADOOP-11318:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/])
HADOOP-11318. Update the document for hadoop fs -stat (aajisaka: rev
ce29074685abaa88ae9586fe8277aca23f9d54d6)
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Stat.java
* hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
* hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
> Update the document for hadoop fs -stat
> ---
>
> Key: HADOOP-11318
> URL: https://issues.apache.org/jira/browse/HADOOP-11318
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: HADOOP-11318-001.patch, HADOOP-11318-002.patch,
> HADOOP-11318-002.patch, HADOOP-11318-003.patch
>
>
> In FileSystemShell.apt.vm,
> {code}
> stat
>Usage: <<>>
>Returns the stat information on the path.
> {code}
> Now {{-stat}} accepts the below formats.
> * %b: Size of file in blocks
> * %g: Group name of owner
> * %n: Filename
> * %o: Block size
> * %r: replication
> * %u: User name of owner
> * %y: UTC date as "-MM-dd HH:mm:ss"
> * %Y: Milliseconds since January 1, 1970 UTC
> They should be documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280349#comment-14280349 ] Hudson commented on HADOOP-11261: - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/]) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11350) The size of header buffer of HttpServer is too small when HTTPS is enabled
[
https://issues.apache.org/jira/browse/HADOOP-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280352#comment-14280352
]
Hudson commented on HADOOP-11350:
-
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/])
HADOOP-11350. The size of header buffer of HttpServer is too small when HTTPS
is enabled. Contributed by Benoy Antony. (wheat9: rev
3ab3a6498812c9fa0c53dae02ce696033062af87)
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/HttpServerFunctionalTest.java
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
> The size of header buffer of HttpServer is too small when HTTPS is enabled
> --
>
> Key: HADOOP-11350
> URL: https://issues.apache.org/jira/browse/HADOOP-11350
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.7.0
>
> Attachments: HADOOP-11350.patch, HADOOP-11350.patch,
> HADOOP-11350.patch, HADOOP-11350.patch
>
>
> {code}
> curl -k -vvv -i -L --negotiate -u : https://:50070
> < HTTP/1.1 413 FULL head
> HTTP/1.1 413 FULL head
> < Connection: close
> Connection: close
> < Server: Jetty(6.1.26)
> Server: Jetty(6.1.26)
> {code}
> For some users, the spnego token too large for the default header buffer used
> by Jetty.
> Though the issue is fixed for HTTP connections (via HADOOP-8816), HTTPS
> connections needs to be fixed as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11463) Replace method-local TransferManager object with S3AFileSystem#transfers
[ https://issues.apache.org/jira/browse/HADOOP-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HADOOP-11463: Attachment: hadoop-11463-002.patch > Replace method-local TransferManager object with S3AFileSystem#transfers > > > Key: HADOOP-11463 > URL: https://issues.apache.org/jira/browse/HADOOP-11463 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Affects Versions: 2.7.0 >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: hadoop-11463-001.patch, hadoop-11463-002.patch > > > This is continuation of HADOOP-11446. > The following changes are made according to Thomas Demoor's comments: > 1. Replace method-local TransferManager object with S3AFileSystem#transfers > 2. Do not shutdown TransferManager after purging existing multipart file - > otherwise the current transfer is unable to proceed > 3. Shutdown TransferManager instance in the close method of S3AFileSystem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-8757) Metrics should disallow names with invalid characters
[ https://issues.apache.org/jira/browse/HADOOP-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280312#comment-14280312 ] Hudson commented on HADOOP-8757: FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/]) HADOOP-8757. Metrics should disallow names with invalid characters (rchiang via rkanter) (rkanter: rev b6ff9c03a4f8aba945e562a7ff60b0fc6a1cd040) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java > Metrics should disallow names with invalid characters > - > > Key: HADOOP-8757 > URL: https://issues.apache.org/jira/browse/HADOOP-8757 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Todd Lipcon >Assignee: Ray Chiang >Priority: Minor > Labels: newbie > Fix For: 2.7.0 > > Attachments: HADOOP-8757-01.patch, HADOOP-8757-02.patch, > HADOOP-8757.003.patch > > > Just spent a couple hours trying to figure out why a metric I added didn't > show up in JMX, only to eventually realize it was because I had a whitespace > in the property name. This didn't cause any errors to be logged -- the metric > just didn't show up in JMX. We should check that the name is valid and log an > error, or replace invalid characters with something like an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11318) Update the document for hadoop fs -stat
[
https://issues.apache.org/jira/browse/HADOOP-11318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280304#comment-14280304
]
Hudson commented on HADOOP-11318:
-
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/])
HADOOP-11318. Update the document for hadoop fs -stat (aajisaka: rev
ce29074685abaa88ae9586fe8277aca23f9d54d6)
* hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Stat.java
* hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
* hadoop-common-project/hadoop-common/CHANGES.txt
> Update the document for hadoop fs -stat
> ---
>
> Key: HADOOP-11318
> URL: https://issues.apache.org/jira/browse/HADOOP-11318
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: HADOOP-11318-001.patch, HADOOP-11318-002.patch,
> HADOOP-11318-002.patch, HADOOP-11318-003.patch
>
>
> In FileSystemShell.apt.vm,
> {code}
> stat
>Usage: <<>>
>Returns the stat information on the path.
> {code}
> Now {{-stat}} accepts the below formats.
> * %b: Size of file in blocks
> * %g: Group name of owner
> * %n: Filename
> * %o: Block size
> * %r: replication
> * %u: User name of owner
> * %y: UTC date as "-MM-dd HH:mm:ss"
> * %Y: Milliseconds since January 1, 1970 UTC
> They should be documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11483) HardLink.java should use the jdk7 createLink method
[ https://issues.apache.org/jira/browse/HADOOP-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280309#comment-14280309 ] Hudson commented on HADOOP-11483: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/]) HADOOP-11483. HardLink.java should use the jdk7 createLink method (aajisaka: rev db51548f706ccd2d0200745ab89e27610c6d10bc) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHardLink.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java * hadoop-common-project/hadoop-common/CHANGES.txt > HardLink.java should use the jdk7 createLink method > --- > > Key: HADOOP-11483 > URL: https://issues.apache.org/jira/browse/HADOOP-11483 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Colin Patrick McCabe >Assignee: Akira AJISAKA > Fix For: 2.7.0 > > Attachments: HDFS-7556-001.patch > > > Now that we are using jdk7, HardLink.java should use the jdk7 createLink > method rather than our shell commands or JNI methods. > Note that we cannot remove all of the JNI / shell commands unless we remove > the code which is checking the link count, something that jdk7 doesn't > provide (at least, I don't think it does) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11350) The size of header buffer of HttpServer is too small when HTTPS is enabled
[
https://issues.apache.org/jira/browse/HADOOP-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280306#comment-14280306
]
Hudson commented on HADOOP-11350:
-
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/])
HADOOP-11350. The size of header buffer of HttpServer is too small when HTTPS
is enabled. Contributed by Benoy Antony. (wheat9: rev
3ab3a6498812c9fa0c53dae02ce696033062af87)
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/HttpServerFunctionalTest.java
> The size of header buffer of HttpServer is too small when HTTPS is enabled
> --
>
> Key: HADOOP-11350
> URL: https://issues.apache.org/jira/browse/HADOOP-11350
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.7.0
>
> Attachments: HADOOP-11350.patch, HADOOP-11350.patch,
> HADOOP-11350.patch, HADOOP-11350.patch
>
>
> {code}
> curl -k -vvv -i -L --negotiate -u : https://:50070
> < HTTP/1.1 413 FULL head
> HTTP/1.1 413 FULL head
> < Connection: close
> Connection: close
> < Server: Jetty(6.1.26)
> Server: Jetty(6.1.26)
> {code}
> For some users, the spnego token too large for the default header buffer used
> by Jetty.
> Though the issue is fixed for HTTP connections (via HADOOP-8816), HTTPS
> connections needs to be fixed as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280303#comment-14280303 ] Hudson commented on HADOOP-11261: - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/]) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11350) The size of header buffer of HttpServer is too small when HTTPS is enabled
[
https://issues.apache.org/jira/browse/HADOOP-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280285#comment-14280285
]
Hudson commented on HADOOP-11350:
-
FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/])
HADOOP-11350. The size of header buffer of HttpServer is too small when HTTPS
is enabled. Contributed by Benoy Antony. (wheat9: rev
3ab3a6498812c9fa0c53dae02ce696033062af87)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/HttpServerFunctionalTest.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> The size of header buffer of HttpServer is too small when HTTPS is enabled
> --
>
> Key: HADOOP-11350
> URL: https://issues.apache.org/jira/browse/HADOOP-11350
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.7.0
>
> Attachments: HADOOP-11350.patch, HADOOP-11350.patch,
> HADOOP-11350.patch, HADOOP-11350.patch
>
>
> {code}
> curl -k -vvv -i -L --negotiate -u : https://:50070
> < HTTP/1.1 413 FULL head
> HTTP/1.1 413 FULL head
> < Connection: close
> Connection: close
> < Server: Jetty(6.1.26)
> Server: Jetty(6.1.26)
> {code}
> For some users, the spnego token too large for the default header buffer used
> by Jetty.
> Though the issue is fixed for HTTP connections (via HADOOP-8816), HTTPS
> connections needs to be fixed as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-8757) Metrics should disallow names with invalid characters
[ https://issues.apache.org/jira/browse/HADOOP-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280291#comment-14280291 ] Hudson commented on HADOOP-8757: FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/]) HADOOP-8757. Metrics should disallow names with invalid characters (rchiang via rkanter) (rkanter: rev b6ff9c03a4f8aba945e562a7ff60b0fc6a1cd040) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java > Metrics should disallow names with invalid characters > - > > Key: HADOOP-8757 > URL: https://issues.apache.org/jira/browse/HADOOP-8757 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Todd Lipcon >Assignee: Ray Chiang >Priority: Minor > Labels: newbie > Fix For: 2.7.0 > > Attachments: HADOOP-8757-01.patch, HADOOP-8757-02.patch, > HADOOP-8757.003.patch > > > Just spent a couple hours trying to figure out why a metric I added didn't > show up in JMX, only to eventually realize it was because I had a whitespace > in the property name. This didn't cause any errors to be logged -- the metric > just didn't show up in JMX. We should check that the name is valid and log an > error, or replace invalid characters with something like an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280282#comment-14280282 ] Hudson commented on HADOOP-11261: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/]) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11318) Update the document for hadoop fs -stat
[
https://issues.apache.org/jira/browse/HADOOP-11318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280283#comment-14280283
]
Hudson commented on HADOOP-11318:
-
FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/])
HADOOP-11318. Update the document for hadoop fs -stat (aajisaka: rev
ce29074685abaa88ae9586fe8277aca23f9d54d6)
* hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Stat.java
> Update the document for hadoop fs -stat
> ---
>
> Key: HADOOP-11318
> URL: https://issues.apache.org/jira/browse/HADOOP-11318
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: HADOOP-11318-001.patch, HADOOP-11318-002.patch,
> HADOOP-11318-002.patch, HADOOP-11318-003.patch
>
>
> In FileSystemShell.apt.vm,
> {code}
> stat
>Usage: <<>>
>Returns the stat information on the path.
> {code}
> Now {{-stat}} accepts the below formats.
> * %b: Size of file in blocks
> * %g: Group name of owner
> * %n: Filename
> * %o: Block size
> * %r: replication
> * %u: User name of owner
> * %y: UTC date as "-MM-dd HH:mm:ss"
> * %Y: Milliseconds since January 1, 1970 UTC
> They should be documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11483) HardLink.java should use the jdk7 createLink method
[ https://issues.apache.org/jira/browse/HADOOP-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280288#comment-14280288 ] Hudson commented on HADOOP-11483: - FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/]) HADOOP-11483. HardLink.java should use the jdk7 createLink method (aajisaka: rev db51548f706ccd2d0200745ab89e27610c6d10bc) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHardLink.java > HardLink.java should use the jdk7 createLink method > --- > > Key: HADOOP-11483 > URL: https://issues.apache.org/jira/browse/HADOOP-11483 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Colin Patrick McCabe >Assignee: Akira AJISAKA > Fix For: 2.7.0 > > Attachments: HDFS-7556-001.patch > > > Now that we are using jdk7, HardLink.java should use the jdk7 createLink > method rather than our shell commands or JNI methods. > Note that we cannot remove all of the JNI / shell commands unless we remove > the code which is checking the link count, something that jdk7 doesn't > provide (at least, I don't think it does) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11463) Replace method-local TransferManager object with S3AFileSystem#transfers
[ https://issues.apache.org/jira/browse/HADOOP-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11463: Component/s: fs/s3 Target Version/s: 2.7.0 Affects Version/s: 2.7.0 > Replace method-local TransferManager object with S3AFileSystem#transfers > > > Key: HADOOP-11463 > URL: https://issues.apache.org/jira/browse/HADOOP-11463 > Project: Hadoop Common > Issue Type: Task > Components: fs/s3 >Affects Versions: 2.7.0 >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: hadoop-11463-001.patch > > > This is continuation of HADOOP-11446. > The following changes are made according to Thomas Demoor's comments: > 1. Replace method-local TransferManager object with S3AFileSystem#transfers > 2. Do not shutdown TransferManager after purging existing multipart file - > otherwise the current transfer is unable to proceed > 3. Shutdown TransferManager instance in the close method of S3AFileSystem -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11463) Replace method-local TransferManager object with S3AFileSystem#transfers
[
https://issues.apache.org/jira/browse/HADOOP-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280258#comment-14280258
]
Steve Loughran commented on HADOOP-11463:
-
-1
I want one minor change, which is the {close()}} operation must check for
transfers != null before doing its work. There's some stuff done in the
{{initialize()}} method before the field is assigned, so there's a risk that if
something went wrong there, close() would fail.
Once that's done I'll approve the patch
> Replace method-local TransferManager object with S3AFileSystem#transfers
>
>
> Key: HADOOP-11463
> URL: https://issues.apache.org/jira/browse/HADOOP-11463
> Project: Hadoop Common
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: hadoop-11463-001.patch
>
>
> This is continuation of HADOOP-11446.
> The following changes are made according to Thomas Demoor's comments:
> 1. Replace method-local TransferManager object with S3AFileSystem#transfers
> 2. Do not shutdown TransferManager after purging existing multipart file -
> otherwise the current transfer is unable to proceed
> 3. Shutdown TransferManager instance in the close method of S3AFileSystem
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10542) Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
[
https://issues.apache.org/jira/browse/HADOOP-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280247#comment-14280247
]
Steve Loughran commented on HADOOP-10542:
-
this still exists in 2.6. Ted: do you have a patch for this?
> Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
> ---
>
> Key: HADOOP-10542
> URL: https://issues.apache.org/jira/browse/HADOOP-10542
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.6.0
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> in = get(blockToKey(block), byteRangeStart);
> out = new BufferedOutputStream(new FileOutputStream(fileBlock));
> byte[] buf = new byte[bufferSize];
> int numRead;
> while ((numRead = in.read(buf)) >= 0) {
> {code}
> get() may return null.
> The while loop dereferences in without null check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-10542) Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
[
https://issues.apache.org/jira/browse/HADOOP-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-10542:
Target Version/s: 2.7.0
Affects Version/s: 2.6.0
> Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
> ---
>
> Key: HADOOP-10542
> URL: https://issues.apache.org/jira/browse/HADOOP-10542
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.6.0
>Reporter: Ted Yu
>Priority: Minor
>
> {code}
> in = get(blockToKey(block), byteRangeStart);
> out = new BufferedOutputStream(new FileOutputStream(fileBlock));
> byte[] buf = new byte[bufferSize];
> int numRead;
> while ((numRead = in.read(buf)) >= 0) {
> {code}
> get() may return null.
> The while loop dereferences in without null check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HADOOP-10037) s3n read truncated, but doesn't throw exception
[ https://issues.apache.org/jira/browse/HADOOP-10037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-10037. - Resolution: Cannot Reproduce Fix Version/s: 2.6.0 closing as Cannot Reproduce, as it appears to have gone away for you. # Hadoop 2.6 is using a much later version of jets3t # Hadoop 2.6 also offers a (compatible) s3a fiesystem which uses the AWS SDK instead. If you do see this problem, try using s3a to see if it occurs there > s3n read truncated, but doesn't throw exception > > > Key: HADOOP-10037 > URL: https://issues.apache.org/jira/browse/HADOOP-10037 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.0.0-alpha > Environment: Ubuntu Linux 13.04 running on Amazon EC2 (cc2.8xlarge) >Reporter: David Rosenstrauch > Fix For: 2.6.0 > > Attachments: S3ReadFailedOnTruncation.html, S3ReadSucceeded.html > > > For months now we've been finding that we've been experiencing frequent data > truncation issues when reading from S3 using the s3n:// protocol. I finally > was able to gather some debugging output on the issue in a job I ran last > night, and so can finally file a bug report. > The job I ran last night was on a 16-node cluster (all of them AWS EC2 > cc2.8xlarge machines, running Ubuntu 13.04 and Cloudera CDH4.3.0). The job > was a Hadoop streaming job, which reads through a large number (i.e., > ~55,000) of files on S3, each of them approximately 300K bytes in size. > All of the files contain 46 columns of data in each record. But I added in > an extra check in my mapper code to count and verify the number of columns in > every record - throwing an error and crashing the map task if the column > count is wrong. > If you look in the attached task logs, you'll see 2 attempts on the same > task. The first one fails due to data truncated (i.e., my job intentionally > fails the map task due to the current record failing the column count check). > The task then gets retried on a different machine and runs to a succesful > completion. > You can see further evidence of the truncation further down in the task logs, > where it displays the count of the records read: the failed task says 32953 > records read, while the successful task says 63133. > Any idea what the problem might be here and/or how to work around it? This > issue is a very common occurrence on our clusters. E.g., in the job I ran > last night before I had gone to bed I had already encountered 8 such > failuers, and the job was only 10% complete. (~25,000 out of ~250,000 tasks.) > I realize that it's common for I/O errors to occur - possibly even frequently > - in a large Hadoop job. But I would think that if an I/O failure (like a > truncated read) did occur, that something in the underlying infrastructure > code (i.e., either in NativeS3FileSystem or in jets3t) should detect the > error and throw an IOException accordingly. It shouldn't be up to the > calling code to detect such failures, IMO. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-9248) Allow configuration of Amazon S3 Endpoint
[ https://issues.apache.org/jira/browse/HADOOP-9248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280235#comment-14280235 ] Steve Loughran commented on HADOOP-9248: HADOOP-11261 fixes this for the new S3a filesystem. On the assumption that once s3a has bedded in and people are confident it works in production, they can migrate off s3n —do we need this feature any more? > Allow configuration of Amazon S3 Endpoint > - > > Key: HADOOP-9248 > URL: https://issues.apache.org/jira/browse/HADOOP-9248 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3 > Environment: All environments connecting to S3 >Reporter: Timur Perelmutov > > http://wiki.apache.org/hadoop/AmazonS3 page describes configuration of Hadoop > with S3 as storage. Other systems like EMC Atmos now implement S3 Interface, > but in order to be able to connect to them, the endpoint needs to be > configurable. Please add a configuration parameter that would be propagated > to underlying jets3t library as s3service.s3-endpoint param. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-8870) NullPointerException when glob doesn't return files
[
https://issues.apache.org/jira/browse/HADOOP-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280232#comment-14280232
]
Steve Loughran commented on HADOOP-8870:
This doesn't occur branch-2 due to HADOOP-8906. Shall I close this as a "Cannot
reproduce?"
> NullPointerException when glob doesn't return files
> ---
>
> Key: HADOOP-8870
> URL: https://issues.apache.org/jira/browse/HADOOP-8870
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/s3
>Affects Versions: 0.20.205.0, 1.0.3
>Reporter: Jaka Jancar
>
> Reading
> {code}s3n://bucket/{a/,b/,c/}{code}
> if one of the globs matches nothing, I get:
> {code}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:992)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:177)
> at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
> at spark.HadoopRDD.(HadoopRDD.scala:51)
> at spark.SparkContext.hadoopFile(SparkContext.scala:186)
> at spark.SparkContext.textFile(SparkContext.scala:155)
> at
> com.celtra.analyzer.LogAnalyzer.analyzeSufficientS3Logs(LogAnalyzer.scala:52)
> at com.celtra.analyzer.App$.main(App.scala:164)
> at com.celtra.analyzer.App.main(App.scala)
> {code}
> I'm not sure whether this is specific to S3 or all filesystems.
> This was occuring in 0.20.205 and I confirmed it's still present in 1.0.3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
[ https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280227#comment-14280227 ] Steve Loughran commented on HADOOP-7352: HDFS & the FS spec do effectively require an IOE of some form to be raised here. Localfs does return null if a list operation fails due to an invalid path or other access problem. This is a bug: it must be escalated to an IOE for consistency with everything else. If someone were to write a patch I'd review it. > Contracts of LocalFileSystem and DistributedFileSystem should require > FileSystem::listStatus throw IOException not return null upon access error > > > Key: HADOOP-7352 > URL: https://issues.apache.org/jira/browse/HADOOP-7352 > Project: Hadoop Common > Issue Type: Improvement > Components: fs, fs/s3 >Reporter: Matt Foley >Assignee: Matt Foley > > In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should > throw FileNotFoundException instead of returning null, when the target > directory did not exist. > However, in LocalFileSystem implementation today, FileSystem::listStatus > still may return null, when the target directory exists but does not grant > read permission. This causes NPE in many callers, for all the reasons cited > in HADOOP-6201 and HDFS-538. See HADOOP-7327 and its linked issues for > examples. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-9577) Actual data loss using s3n (against US Standard region)
[ https://issues.apache.org/jira/browse/HADOOP-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-9577. Resolution: Won't Fix I'm going to close this as something we don't currently plan to fix in the Hadoop core codebase, given that Netflix S3mper and EMR itself both offer a solution, namely support on Amazon Dynamo for a consistent metadata store. The other way to get guaranteed create consistency is "don't use US East", which has no consistency guarantees —whereas everything else offers Create , but not Update or Delete > Actual data loss using s3n (against US Standard region) > --- > > Key: HADOOP-9577 > URL: https://issues.apache.org/jira/browse/HADOOP-9577 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 1.0.3 >Reporter: Joshua Caplan >Priority: Critical > > The implementation of needsTaskCommit() assumes that the FileSystem used for > writing temporary outputs is consistent. That happens not to be the case > when using the S3 native filesystem in the US Standard region. It is > actually quite common in larger jobs for the exists() call to return false > even if the task attempt wrote output minutes earlier, which essentially > cancels the commit operation with no error. That's real life data loss right > there, folks. > The saddest part is that the Hadoop APIs do not seem to provide any > legitimate means for the various RecordWriters to communicate with the > OutputCommitter. In my projects I have created a static map of semaphores > keyed by TaskAttemptID, which all my custom RecordWriters have to be aware > of. That's pretty lame. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-4436) S3 object names with arbitrary slashes confuse NativeS3FileSystem
[ https://issues.apache.org/jira/browse/HADOOP-4436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-4436. Resolution: Won't Fix Fix Version/s: 2.7.0 closing as WONTFIX...s3n, s3a and openstack clients all assume that a "/" paths represent directory delimiters in paths, an assumption that goes pretty deep. Sorry > S3 object names with arbitrary slashes confuse NativeS3FileSystem > - > > Key: HADOOP-4436 > URL: https://issues.apache.org/jira/browse/HADOOP-4436 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.18.1 >Reporter: David Phillips > Fix For: 2.7.0 > > > Consider a bucket with the following object names: > * / > * /foo > * foo//bar > NativeS3FileSystem treats an object named "/" as a directory. Doing an "fs > -lsr" causes an infinite loop. > I suggest we change NativeS3FileSystem to handle these by ignoring any such > "invalid" names. Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11262) Enable YARN to use S3A
[
https://issues.apache.org/jira/browse/HADOOP-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280205#comment-14280205
]
Steve Loughran commented on HADOOP-11262:
-
h3. TestS3A
# replace .* import with specific class imports
# you shouldn't have to set the s3a.impl file; it should instead go into
core-default.xml
# {{testS3AStatus()}}: assertTrue/false should include an error message,
ideally with the errant values
# {{testS3ACreateFileInSubDir}} should close FS in a try/finally block. You can
use the Java7 construct to do this now.
> Enable YARN to use S3A
> ---
>
> Key: HADOOP-11262
> URL: https://issues.apache.org/jira/browse/HADOOP-11262
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
>Reporter: Thomas Demoor
> Labels: amazon, s3
> Attachments: HADOOP-11262-2.patch, HADOOP-11262-3.patch,
> HADOOP-11262.patch
>
>
> Uses DelegateToFileSystem to expose S3A as an AbstractFileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-10643) Add NativeS3Fs that delgates calls from FileContext apis to native s3 fs implementation
[
https://issues.apache.org/jira/browse/HADOOP-10643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280198#comment-14280198
]
Steve Loughran commented on HADOOP-10643:
-
Sumit -why did you cut a check from {{AbstractFileSystem}}? That could have
implications elsewhere.
This should be retained, otherwise we've added a new way for things to fail,
different error messages, etc.
Can you add a new protected method to validate the URI; something which does
the check —and which your code overrides if it really, really doesn't want a
port to be specified. That won't impact anything else.
> Add NativeS3Fs that delgates calls from FileContext apis to native s3 fs
> implementation
> ---
>
> Key: HADOOP-10643
> URL: https://issues.apache.org/jira/browse/HADOOP-10643
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
>Affects Versions: 2.4.0
>Reporter: Sumit Kumar
>Assignee: Sumit Kumar
> Attachments: HADOOP-10643.patch
>
>
> The new set of file system related apis (FileContext/AbstractFileSystem)
> already support local filesytem, hdfs, viewfs) however they don't support
> s3n. This patch is to add that support using configurations like
> fs.AbstractFileSystem.s3n.impl = org.apache.hadoop.fs.s3native.NativeS3Fs
> This patch however doesn't provide a new implementation, instead relies on
> DelegateToFileSystem abstract class to delegate all calls from FileContext
> apis for s3n to the NativeS3FileSystem implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11171) Enable using a proxy server to connect to S3a.
[ https://issues.apache.org/jira/browse/HADOOP-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280194#comment-14280194 ] Steve Loughran commented on HADOOP-11171: - Thomas, this doesn't apply any more. Can you sync it up with trunk/branch-2 and I'll test & apply it > Enable using a proxy server to connect to S3a. > -- > > Key: HADOOP-11171 > URL: https://issues.apache.org/jira/browse/HADOOP-11171 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.4.0 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Attachments: HADOOP-11171-2.patch, HADOOP-11171-3.patch, > HADOOP-11171-4.patch, HADOOP-11171-5.patch, HADOOP-11171-6.patch, > HADOOP-11171-7.patch, HADOOP-11171-8.patch, HADOOP-11171.patch > > > This exposes the AWS SDK config for a proxy (host and port) to s3a through > config settings. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11350) The size of header buffer of HttpServer is too small when HTTPS is enabled
[
https://issues.apache.org/jira/browse/HADOOP-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280096#comment-14280096
]
Hudson commented on HADOOP-11350:
-
FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/809/])
HADOOP-11350. The size of header buffer of HttpServer is too small when HTTPS
is enabled. Contributed by Benoy Antony. (wheat9: rev
3ab3a6498812c9fa0c53dae02ce696033062af87)
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/HttpServerFunctionalTest.java
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> The size of header buffer of HttpServer is too small when HTTPS is enabled
> --
>
> Key: HADOOP-11350
> URL: https://issues.apache.org/jira/browse/HADOOP-11350
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.7.0
>
> Attachments: HADOOP-11350.patch, HADOOP-11350.patch,
> HADOOP-11350.patch, HADOOP-11350.patch
>
>
> {code}
> curl -k -vvv -i -L --negotiate -u : https://:50070
> < HTTP/1.1 413 FULL head
> HTTP/1.1 413 FULL head
> < Connection: close
> Connection: close
> < Server: Jetty(6.1.26)
> Server: Jetty(6.1.26)
> {code}
> For some users, the spnego token too large for the default header buffer used
> by Jetty.
> Though the issue is fixed for HTTP connections (via HADOOP-8816), HTTPS
> connections needs to be fixed as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-8757) Metrics should disallow names with invalid characters
[ https://issues.apache.org/jira/browse/HADOOP-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280102#comment-14280102 ] Hudson commented on HADOOP-8757: FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/809/]) HADOOP-8757. Metrics should disallow names with invalid characters (rchiang via rkanter) (rkanter: rev b6ff9c03a4f8aba945e562a7ff60b0fc6a1cd040) * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java > Metrics should disallow names with invalid characters > - > > Key: HADOOP-8757 > URL: https://issues.apache.org/jira/browse/HADOOP-8757 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Todd Lipcon >Assignee: Ray Chiang >Priority: Minor > Labels: newbie > Fix For: 2.7.0 > > Attachments: HADOOP-8757-01.patch, HADOOP-8757-02.patch, > HADOOP-8757.003.patch > > > Just spent a couple hours trying to figure out why a metric I added didn't > show up in JMX, only to eventually realize it was because I had a whitespace > in the property name. This didn't cause any errors to be logged -- the metric > just didn't show up in JMX. We should check that the name is valid and log an > error, or replace invalid characters with something like an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280093#comment-14280093 ] Hudson commented on HADOOP-11261: - FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/809/]) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11318) Update the document for hadoop fs -stat
[
https://issues.apache.org/jira/browse/HADOOP-11318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280094#comment-14280094
]
Hudson commented on HADOOP-11318:
-
FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/809/])
HADOOP-11318. Update the document for hadoop fs -stat (aajisaka: rev
ce29074685abaa88ae9586fe8277aca23f9d54d6)
* hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Stat.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
> Update the document for hadoop fs -stat
> ---
>
> Key: HADOOP-11318
> URL: https://issues.apache.org/jira/browse/HADOOP-11318
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: HADOOP-11318-001.patch, HADOOP-11318-002.patch,
> HADOOP-11318-002.patch, HADOOP-11318-003.patch
>
>
> In FileSystemShell.apt.vm,
> {code}
> stat
>Usage: <<>>
>Returns the stat information on the path.
> {code}
> Now {{-stat}} accepts the below formats.
> * %b: Size of file in blocks
> * %g: Group name of owner
> * %n: Filename
> * %o: Block size
> * %r: replication
> * %u: User name of owner
> * %y: UTC date as "-MM-dd HH:mm:ss"
> * %Y: Milliseconds since January 1, 1970 UTC
> They should be documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11483) HardLink.java should use the jdk7 createLink method
[ https://issues.apache.org/jira/browse/HADOOP-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280099#comment-14280099 ] Hudson commented on HADOOP-11483: - FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/809/]) HADOOP-11483. HardLink.java should use the jdk7 createLink method (aajisaka: rev db51548f706ccd2d0200745ab89e27610c6d10bc) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHardLink.java * hadoop-common-project/hadoop-common/CHANGES.txt > HardLink.java should use the jdk7 createLink method > --- > > Key: HADOOP-11483 > URL: https://issues.apache.org/jira/browse/HADOOP-11483 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Colin Patrick McCabe >Assignee: Akira AJISAKA > Fix For: 2.7.0 > > Attachments: HDFS-7556-001.patch > > > Now that we are using jdk7, HardLink.java should use the jdk7 createLink > method rather than our shell commands or JNI methods. > Note that we cannot remove all of the JNI / shell commands unless we remove > the code which is checking the link count, something that jdk7 doesn't > provide (at least, I don't think it does) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-8757) Metrics should disallow names with invalid characters
[ https://issues.apache.org/jira/browse/HADOOP-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280088#comment-14280088 ] Hudson commented on HADOOP-8757: FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/]) HADOOP-8757. Metrics should disallow names with invalid characters (rchiang via rkanter) (rkanter: rev b6ff9c03a4f8aba945e562a7ff60b0fc6a1cd040) * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMetricsRegistry.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java > Metrics should disallow names with invalid characters > - > > Key: HADOOP-8757 > URL: https://issues.apache.org/jira/browse/HADOOP-8757 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Todd Lipcon >Assignee: Ray Chiang >Priority: Minor > Labels: newbie > Fix For: 2.7.0 > > Attachments: HADOOP-8757-01.patch, HADOOP-8757-02.patch, > HADOOP-8757.003.patch > > > Just spent a couple hours trying to figure out why a metric I added didn't > show up in JMX, only to eventually realize it was because I had a whitespace > in the property name. This didn't cause any errors to be logged -- the metric > just didn't show up in JMX. We should check that the name is valid and log an > error, or replace invalid characters with something like an underscore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11483) HardLink.java should use the jdk7 createLink method
[ https://issues.apache.org/jira/browse/HADOOP-11483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280085#comment-14280085 ] Hudson commented on HADOOP-11483: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/]) HADOOP-11483. HardLink.java should use the jdk7 createLink method (aajisaka: rev db51548f706ccd2d0200745ab89e27610c6d10bc) * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HardLink.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestHardLink.java > HardLink.java should use the jdk7 createLink method > --- > > Key: HADOOP-11483 > URL: https://issues.apache.org/jira/browse/HADOOP-11483 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Colin Patrick McCabe >Assignee: Akira AJISAKA > Fix For: 2.7.0 > > Attachments: HDFS-7556-001.patch > > > Now that we are using jdk7, HardLink.java should use the jdk7 createLink > method rather than our shell commands or JNI methods. > Note that we cannot remove all of the JNI / shell commands unless we remove > the code which is checking the link count, something that jdk7 doesn't > provide (at least, I don't think it does) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280079#comment-14280079 ] Hudson commented on HADOOP-11261: - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/]) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * hadoop-common-project/hadoop-common/CHANGES.txt * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11318) Update the document for hadoop fs -stat
[
https://issues.apache.org/jira/browse/HADOOP-11318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280080#comment-14280080
]
Hudson commented on HADOOP-11318:
-
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/])
HADOOP-11318. Update the document for hadoop fs -stat (aajisaka: rev
ce29074685abaa88ae9586fe8277aca23f9d54d6)
* hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
* hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Stat.java
* hadoop-common-project/hadoop-common/CHANGES.txt
> Update the document for hadoop fs -stat
> ---
>
> Key: HADOOP-11318
> URL: https://issues.apache.org/jira/browse/HADOOP-11318
> Project: Hadoop Common
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.5.1
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Fix For: 2.7.0
>
> Attachments: HADOOP-11318-001.patch, HADOOP-11318-002.patch,
> HADOOP-11318-002.patch, HADOOP-11318-003.patch
>
>
> In FileSystemShell.apt.vm,
> {code}
> stat
>Usage: <<>>
>Returns the stat information on the path.
> {code}
> Now {{-stat}} accepts the below formats.
> * %b: Size of file in blocks
> * %g: Group name of owner
> * %n: Filename
> * %o: Block size
> * %r: replication
> * %u: User name of owner
> * %y: UTC date as "-MM-dd HH:mm:ss"
> * %Y: Milliseconds since January 1, 1970 UTC
> They should be documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11350) The size of header buffer of HttpServer is too small when HTTPS is enabled
[
https://issues.apache.org/jira/browse/HADOOP-11350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280082#comment-14280082
]
Hudson commented on HADOOP-11350:
-
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/])
HADOOP-11350. The size of header buffer of HttpServer is too small when HTTPS
is enabled. Contributed by Benoy Antony. (wheat9: rev
3ab3a6498812c9fa0c53dae02ce696033062af87)
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestSSLHttpServer.java
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
* hadoop-common-project/hadoop-common/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/HttpServerFunctionalTest.java
> The size of header buffer of HttpServer is too small when HTTPS is enabled
> --
>
> Key: HADOOP-11350
> URL: https://issues.apache.org/jira/browse/HADOOP-11350
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Benoy Antony
>Assignee: Benoy Antony
> Fix For: 2.7.0
>
> Attachments: HADOOP-11350.patch, HADOOP-11350.patch,
> HADOOP-11350.patch, HADOOP-11350.patch
>
>
> {code}
> curl -k -vvv -i -L --negotiate -u : https://:50070
> < HTTP/1.1 413 FULL head
> HTTP/1.1 413 FULL head
> < Connection: close
> Connection: close
> < Server: Jetty(6.1.26)
> Server: Jetty(6.1.26)
> {code}
> For some users, the spnego token too large for the default header buffer used
> by Jetty.
> Though the issue is fixed for HTTP connections (via HADOOP-8816), HTTPS
> connections needs to be fixed as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280063#comment-14280063 ] Hudson commented on HADOOP-11261: - FAILURE: Integrated in Hadoop-trunk-Commit #6876 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6876/]) HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) (stevel: rev 000ca83ea3aeb3687625c857bcc0762fd2887db2) * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AConfiguration.java * hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * hadoop-common-project/hadoop-common/CHANGES.txt > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-11261. - Resolution: Fixed Fix Version/s: 2.7.0 > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280055#comment-14280055 ] Steve Loughran commented on HADOOP-11261: - All tests pass, reviewed code looks good +1, committing Thomas: you are going to need to write a site documentation patch which covers this and the other configuration parameters you've got in pending changes > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Fix For: 2.7.0 > > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11261) Set custom endpoint for S3A
[ https://issues.apache.org/jira/browse/HADOOP-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-11261: Assignee: Thomas Demoor > Set custom endpoint for S3A > --- > > Key: HADOOP-11261 > URL: https://issues.apache.org/jira/browse/HADOOP-11261 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Reporter: Thomas Demoor >Assignee: Thomas Demoor > Labels: amazon, s3 > Attachments: HADOOP-11261-2.patch, HADOOP-11261-3.patch, > JIRA-11261.patch > > > Use a config setting to allow customizing the used AWS region. > It also enables using a custom url pointing to an S3-compatible object store. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-11335) KMS ACL in meta data or database
[
https://issues.apache.org/jira/browse/HADOOP-11335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280013#comment-14280013
]
Arun Suresh commented on HADOOP-11335:
--
Thanks for working this [~dian.fu]
I have taken an first pass at the latest patch.. my initial comments follow
(will send out more comments once I do further reviews).
1) JavaKeyStoreProvider:
* createKey() : I think it might be a bit odd that we have metadata with
version == 0
* deleteKey() : Am sorry, I might be missing something but, I did not quite
understand why we can't just delete the metadata from the cache when the key is
deleted
2) KeyProvider:
* The Metadata Class now has a dependency on {{KeyOpType}} which is defined
in {{KeyProviderAuthorizationExtension}}. The Extension classes were meant to
add functionality to a KeyProvider. It seems a bit weird that the KeyProvider
class should have a dependency on an Extension.
* Not very confortable adding setters to Metadata. I Guess the original
implementaion conciously made a choice to not allow modification of metadata
once it is created (except for the version)
3) KeyProviderExtension:
* Do we really need the read and write locks ? The undelying KeyProvider
should take care of the synchronization. (for eg. {{JavaKeyStoreProvider}})
does infact use write and read locks for createKey etc.. this would probably
lead to unnecessary double locking.
4) KeyProviderAuthorizationExtension:
* Same as above.. do we really need the read and write locks ? I feel the
Extension class should handle its own concurrency semantics
5) MetadataKeyAuthorizer
* Remove commented code
Looking at the commented code in {{MetadataKeyAuthorizer}}, I see that you had
initially toyed with having an extended {{MetadataWithACL}} class. Any reason
why you did not pursue that design ? It seems to me like that could have been a
way to probably avoid having to modify the {{JavaKeyStoreProvider}} and
{{KeyProvider}}. One suggestion would have been to templatized {{KeyProvider}}
like so :
{noformat}
public class KeyProvider
...
{noformat}
and have different implemenetation of a {{KeyProvider}} like :
{noformat}
public classs KeyProviderWithACls extends KeyProvider
...
{noformat}
> KMS ACL in meta data or database
>
>
> Key: HADOOP-11335
> URL: https://issues.apache.org/jira/browse/HADOOP-11335
> Project: Hadoop Common
> Issue Type: Improvement
> Components: kms
>Affects Versions: 2.6.0
>Reporter: Jerry Chen
>Assignee: Dian Fu
> Labels: Security
> Attachments: HADOOP-11335.001.patch, HADOOP-11335.002.patch, KMS ACL
> in metadata or database.pdf
>
> Original Estimate: 504h
> Remaining Estimate: 504h
>
> Currently Hadoop KMS has implemented ACL for keys and the per key ACL are
> stored in the configuration file kms-acls.xml.
> The management of ACL in configuration file would not be easy in enterprise
> usage and it is put difficulties for backup and recovery.
> It is ideal to store the ACL for keys in the key meta data similar to what
> file system ACL does. In this way, the backup and recovery that works on
> keys should work for ACL for keys too.
> On the other hand, with the ACL in meta data, the ACL of each key can be
> easily manipulate with API or command line tool and take effect instantly.
> This is very important for enterprise level access control management. This
> feature can be addressed by separate JIRA. While with the configuration file,
> these would be hard to provide.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HADOOP-11485) Pluggable shell integration
[ https://issues.apache.org/jira/browse/HADOOP-11485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279946#comment-14279946 ] Allen Wittenauer commented on HADOOP-11485: --- I'm making HADOOP-11460 required, as it is nearly impossible to give 3rd parties guidance on what they can/cannot change without removing some of the cruft generated by previous shell work. > Pluggable shell integration > --- > > Key: HADOOP-11485 > URL: https://issues.apache.org/jira/browse/HADOOP-11485 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: scripts, shell > Attachments: HADOOP-11485-00.patch > > > It would be useful to provide a way for core and non-core Hadoop components > to plug into the shell infrastructure. This would allow us to pull the HDFS, > MapReduce, and YARN shell functions out of hadoop-functions.sh. > Additionally, it should let 3rd parties such as HBase influence things like > classpaths at runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11460) Add a way to deprecrate shell vars
[ https://issues.apache.org/jira/browse/HADOOP-11460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11460: -- Attachment: HADOOP-11460-00.patch -00: * initial version * deprecate a bunch of hdfs, mapred, and yarn vars > Add a way to deprecrate shell vars > -- > > Key: HADOOP-11460 > URL: https://issues.apache.org/jira/browse/HADOOP-11460 > Project: Hadoop Common > Issue Type: New Feature > Components: scripts >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer > Labels: scripts, shell > Attachments: HADOOP-11460-00.patch > > > It is a very common shell pattern in 3.x to effectively replace sub-project > specific vars with generics. We should have a function that does this > replacement and provides a warning to the end user that the old shell var is > deprecated. This will allow us an easier time to remove them in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
