[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303568#comment-15303568 ] Ling Zhou commented on HADOOP-12756: Thanks Kai. There are two configuration files. test/resources/contract-test-options.xml and test/resources/auth-keys.xml. It seems these two files should not be included in source code, as what .gitingore has excluded. Maybe we can provide these two files separately? > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303561#comment-15303561 ] Chris Nauroth commented on HADOOP-12756: The situation is similar to the testing challenges we face for other similar modules: hadoop-aws, hadoop-azure and hadoop-openstack. Right now, these modules don't really get tested during pre-commit, except for a few true unit tests that don't require integration with an external service. Instead, it's up to reviewers to get credentials to the external service, either using a personal account or an employer's account, and configure the tests to run from their development environments. There is enough activity happening in these modules that it would be in our best interest to get the tests running from pre-commit. I haven't put much thought into how this would work. Kai is right that the credentials need to go somewhere accessible by each Jenkins host that runs a Hadoop pre-commit build. However, we wouldn't want those credentials accessible by the whole Internet or really even the whole Apache contributor community. I expect it will take coordination with the Apache infra team to get this done correctly. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13199) Add doc for distcp -filters
[ https://issues.apache.org/jira/browse/HADOOP-13199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-13199: Attachment: HADOOP-13199.002.patch Patch 002: * Add option -filters to DistCp.md.vm * Back out alphabetic listing in patch 001 > Add doc for distcp -filters > --- > > Key: HADOOP-13199 > URL: https://issues.apache.org/jira/browse/HADOOP-13199 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Trivial > Labels: supportability > Attachments: HADOOP-13199.001.patch, HADOOP-13199.002.patch > > > Update distcp doc to reflect -filters option added by HADOOP-1540. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303548#comment-15303548 ] Kai Zheng commented on HADOOP-11540: The check styles are not suitable to fix and the failure isn't related. Checked it built on windows. > Raw Reed-Solomon coder using Intel ISA-L library > > > Key: HADOOP-11540 > URL: https://issues.apache.org/jira/browse/HADOOP-11540 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Kai Zheng > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, > HADOOP-11540-v10.patch, HADOOP-11540-v11.patch, HADOOP-11540-v2.patch, > HADOOP-11540-v4.patch, HADOOP-11540-v5.patch, HADOOP-11540-v6.patch, > HADOOP-11540-v7.patch, HADOOP-11540-v8.patch, HADOOP-11540-v9.patch, > HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel > ISAL-v1.pdf > > > This is to provide RS codec implementation using Intel ISA-L library for > encoding and decoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303503#comment-15303503
]
stack commented on HADOOP-12910:
bq. Why making the entire async feature unavailable in branch 2?
I'd think a new API on HDFS with a new semantic showing up late in the life of
H2 would come as a surprise to most and seems like a natural H3 differentiator.
But no matter. Sounds like you are targeting H2.
bq. I suggest returning Future (or a sub-interface to support callbacks) in
branch 2 and CompletableFuture (or our own implementation of CompletableFuture)
in trunk. In this way, trunk is backward compatible to branch 2 since
CompletableFuture implements Future.
Suggest it should be same in branch 2 and branch 3 given how much work the spec
and implementation will be. You can't consume the API in a
non-blocking/asynchronous way if you can't register a callback. So a Future
alone is not sufficient, or to put it another way, if H2 returns a Future,
only, and H3 allows registering callbacks, then the implementation and consumer
will have to change going from H2 to H3.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13210) MetricsSinkAdapter always reset retryDelay and retryCount
[
https://issues.apache.org/jira/browse/HADOOP-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303497#comment-15303497
]
Yiqun Lin commented on HADOOP-13210:
The failed unit test {{TestGangliaMetrics}} also appeared in HADOOP-13206 and
the other one {{TestReloadingX509TrustManager}} looks not related. Thanks for
review.
> MetricsSinkAdapter always reset retryDelay and retryCount
> -
>
> Key: HADOOP-13210
> URL: https://issues.apache.org/jira/browse/HADOOP-13210
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Yiqun Lin
> Attachments: HADOOP-13210.001.patch
>
>
> In {{MetricsSinkAdapter#publishMetricsFromQueue}}, it always reset the
> {{retryDelay}} and {{retryCount}} in each loop.
> {code}
> void publishMetricsFromQueue() {
> int retryDelay = firstRetryDelay;
> int n = retryCount;
> int minDelay = Math.min(500, retryDelay * 1000); // millis
> Random rng = new Random(System.nanoTime());
> while (!stopping) {
> try {
> queue.consumeAll(this);
> refreshQueueSizeGauge();
> retryDelay = firstRetryDelay;
> n = retryCount;
> inError = false;
> } catch (InterruptedException e) {
> LOG.info(name +" thread interrupted.");
> } catch (Exception e) {
> if (n > 0) {
> int retryWindow = Math.max(0, 1000 / 2 * retryDelay - minDelay);
> int awhile = rng.nextInt(retryWindow) + minDelay;
> if (!inError) {
> LOG.error("Got sink exception, retry in "+ awhile +"ms", e);
> }
> retryDelay *= retryBackoff;
> try { Thread.sleep(awhile); }
> catch (InterruptedException e2) {
> LOG.info(name +" thread interrupted while waiting for retry", e2);
> }
> --n;
> } else {
> if (!inError) {
> LOG.error("Got sink exception and over retry limit, "+
> "suppressing further error messages", e);
> }
> queue.clear();
> refreshQueueSizeGauge();
> inError = true; // Don't keep complaining ad infinitum
> }
> }
> }
> }
> {code}
> It looks the change of these in catch block code will not comes into effect.
> I think we should remove these two line because the values have been
> initalized before in {{publishMetricsFromQueue}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13070) classloading isolation improvements for cleaner and stricter dependencies
[ https://issues.apache.org/jira/browse/HADOOP-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303494#comment-15303494 ] Sangjin Lee commented on HADOOP-13070: -- I would greatly appreciate feedback on the proposal or thoughts and suggestions in general. Thanks! > classloading isolation improvements for cleaner and stricter dependencies > - > > Key: HADOOP-13070 > URL: https://issues.apache.org/jira/browse/HADOOP-13070 > Project: Hadoop Common > Issue Type: Improvement > Components: util >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Attachments: classloading-improvements-ideas-v.3.pdf, > classloading-improvements-ideas.pdf, classloading-improvements-ideas.v.2.pdf > > > Related to HADOOP-11656, we would like to make a number of improvements in > terms of classloading isolation so that user-code can run safely without > worrying about dependency collisions with the Hadoop dependencies. > By the same token, it should raised the quality of the user code and its > specified classpath so that users get clear signals if they specify incorrect > classpaths. > This will contain a proposal that will include several improvements some of > which may not be backward compatible. As such, it should be targeted to the > next major revision of Hadoop. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13211) Swift driver should have a configurable retry feature when ecounter 5xx error
Chen He created HADOOP-13211: Summary: Swift driver should have a configurable retry feature when ecounter 5xx error Key: HADOOP-13211 URL: https://issues.apache.org/jira/browse/HADOOP-13211 Project: Hadoop Common Issue Type: New Feature Components: fs/swift Affects Versions: 2.7.2 Reporter: Chen He Assignee: Chen He In current code. if Swift driver meets a HTTP 5xx, it will throw exception and stop. As a driver, it will be more sophisticate if it can retry a configurable times before report failure. There are two reasons that I can image: 1. if the server is really busy, it is possible that the server will drop some requests to avoid DDoS attack. 2. If server accidentally unavailable for a short period of time and come back again, we may not need to fail the whole driver. Just record the exception and retry may be more flexible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library
[
https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303458#comment-15303458
]
Hadoop QA commented on HADOOP-11540:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 3s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-common-project/hadoop-common: The patch generated
14 new + 28 unchanged - 0 fixed = 42 total (was 28) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 45s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806555/HADOOP-11540-v11.patch
|
| JIRA Issue | HADOOP-11540 |
| Optional Tests | asflicense xml compile javac javadoc mvninstall
mvnsite unit findbugs checkstyle cc |
| uname | Linux d8012662ee5a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8c84a2a |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9598/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9598/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9598/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9598/testReport/ |
|
[jira] [Commented] (HADOOP-13210) MetricsSinkAdapter always reset retryDelay and retryCount
[
https://issues.apache.org/jira/browse/HADOOP-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303435#comment-15303435
]
Hadoop QA commented on HADOOP-13210:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 31s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
| | hadoop.security.ssl.TestReloadingX509TrustManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806554/HADOOP-13210.001.patch
|
| JIRA Issue | HADOOP-13210 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d6ab6e4207ef 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8c84a2a |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9597/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9597/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9597/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9597/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> MetricsSinkAdapter always reset retryDelay and
[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303407#comment-15303407 ] Kai Zheng commented on HADOOP-12756: Thanks Mingfei. The problem is it's hard to distinguish Hadoop users from Hadoop community developers. The bundle of codes should be automatically tested by Yetus to ensure it's well maintained across release and release. The required account info credential must be somewhere in configuration or scripts and there is no mechanism to protect them from being known by some people. IMO, Aliyun should be aware of this and OK about the info being used in the open source project in the way when providing it. Otherwise, I don't aware any way it could proceed. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11540) Raw Reed-Solomon coder using Intel ISA-L library
[ https://issues.apache.org/jira/browse/HADOOP-11540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11540: --- Attachment: HADOOP-11540-v11.patch Rebased the patch on the latest trunk, and double checked all the comments are addressed. > Raw Reed-Solomon coder using Intel ISA-L library > > > Key: HADOOP-11540 > URL: https://issues.apache.org/jira/browse/HADOOP-11540 > Project: Hadoop Common > Issue Type: Sub-task >Affects Versions: HDFS-7285 >Reporter: Zhe Zhang >Assignee: Kai Zheng > Labels: hdfs-ec-3.0-must-do > Attachments: HADOOP-11540-initial.patch, HADOOP-11540-v1.patch, > HADOOP-11540-v10.patch, HADOOP-11540-v11.patch, HADOOP-11540-v2.patch, > HADOOP-11540-v4.patch, HADOOP-11540-v5.patch, HADOOP-11540-v6.patch, > HADOOP-11540-v7.patch, HADOOP-11540-v8.patch, HADOOP-11540-v9.patch, > HADOOP-11540-with-11996-codes.patch, Native Erasure Coder Performance - Intel > ISAL-v1.pdf > > > This is to provide RS codec implementation using Intel ISA-L library for > encoding and decoding. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13210) MetricsSinkAdapter always reset retryDelay and retryCount
[
https://issues.apache.org/jira/browse/HADOOP-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HADOOP-13210:
---
Status: Patch Available (was: Open)
Attach a patch for this.
> MetricsSinkAdapter always reset retryDelay and retryCount
> -
>
> Key: HADOOP-13210
> URL: https://issues.apache.org/jira/browse/HADOOP-13210
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Yiqun Lin
> Attachments: HADOOP-13210.001.patch
>
>
> In {{MetricsSinkAdapter#publishMetricsFromQueue}}, it always reset the
> {{retryDelay}} and {{retryCount}} in each loop.
> {code}
> void publishMetricsFromQueue() {
> int retryDelay = firstRetryDelay;
> int n = retryCount;
> int minDelay = Math.min(500, retryDelay * 1000); // millis
> Random rng = new Random(System.nanoTime());
> while (!stopping) {
> try {
> queue.consumeAll(this);
> refreshQueueSizeGauge();
> retryDelay = firstRetryDelay;
> n = retryCount;
> inError = false;
> } catch (InterruptedException e) {
> LOG.info(name +" thread interrupted.");
> } catch (Exception e) {
> if (n > 0) {
> int retryWindow = Math.max(0, 1000 / 2 * retryDelay - minDelay);
> int awhile = rng.nextInt(retryWindow) + minDelay;
> if (!inError) {
> LOG.error("Got sink exception, retry in "+ awhile +"ms", e);
> }
> retryDelay *= retryBackoff;
> try { Thread.sleep(awhile); }
> catch (InterruptedException e2) {
> LOG.info(name +" thread interrupted while waiting for retry", e2);
> }
> --n;
> } else {
> if (!inError) {
> LOG.error("Got sink exception and over retry limit, "+
> "suppressing further error messages", e);
> }
> queue.clear();
> refreshQueueSizeGauge();
> inError = true; // Don't keep complaining ad infinitum
> }
> }
> }
> }
> {code}
> It looks the change of these in catch block code will not comes into effect.
> I think we should remove these two line because the values have been
> initalized before in {{publishMetricsFromQueue}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13210) MetricsSinkAdapter always reset retryDelay and retryCount
[
https://issues.apache.org/jira/browse/HADOOP-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HADOOP-13210:
---
Attachment: HADOOP-13210.001.patch
> MetricsSinkAdapter always reset retryDelay and retryCount
> -
>
> Key: HADOOP-13210
> URL: https://issues.apache.org/jira/browse/HADOOP-13210
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Reporter: Yiqun Lin
> Attachments: HADOOP-13210.001.patch
>
>
> In {{MetricsSinkAdapter#publishMetricsFromQueue}}, it always reset the
> {{retryDelay}} and {{retryCount}} in each loop.
> {code}
> void publishMetricsFromQueue() {
> int retryDelay = firstRetryDelay;
> int n = retryCount;
> int minDelay = Math.min(500, retryDelay * 1000); // millis
> Random rng = new Random(System.nanoTime());
> while (!stopping) {
> try {
> queue.consumeAll(this);
> refreshQueueSizeGauge();
> retryDelay = firstRetryDelay;
> n = retryCount;
> inError = false;
> } catch (InterruptedException e) {
> LOG.info(name +" thread interrupted.");
> } catch (Exception e) {
> if (n > 0) {
> int retryWindow = Math.max(0, 1000 / 2 * retryDelay - minDelay);
> int awhile = rng.nextInt(retryWindow) + minDelay;
> if (!inError) {
> LOG.error("Got sink exception, retry in "+ awhile +"ms", e);
> }
> retryDelay *= retryBackoff;
> try { Thread.sleep(awhile); }
> catch (InterruptedException e2) {
> LOG.info(name +" thread interrupted while waiting for retry", e2);
> }
> --n;
> } else {
> if (!inError) {
> LOG.error("Got sink exception and over retry limit, "+
> "suppressing further error messages", e);
> }
> queue.clear();
> refreshQueueSizeGauge();
> inError = true; // Don't keep complaining ad infinitum
> }
> }
> }
> }
> {code}
> It looks the change of these in catch block code will not comes into effect.
> I think we should remove these two line because the values have been
> initalized before in {{publishMetricsFromQueue}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13210) MetricsSinkAdapter always reset retryDelay and retryCount
Yiqun Lin created HADOOP-13210:
--
Summary: MetricsSinkAdapter always reset retryDelay and retryCount
Key: HADOOP-13210
URL: https://issues.apache.org/jira/browse/HADOOP-13210
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Reporter: Yiqun Lin
In {{MetricsSinkAdapter#publishMetricsFromQueue}}, it always reset the
{{retryDelay}} and {{retryCount}} in each loop.
{code}
void publishMetricsFromQueue() {
int retryDelay = firstRetryDelay;
int n = retryCount;
int minDelay = Math.min(500, retryDelay * 1000); // millis
Random rng = new Random(System.nanoTime());
while (!stopping) {
try {
queue.consumeAll(this);
refreshQueueSizeGauge();
retryDelay = firstRetryDelay;
n = retryCount;
inError = false;
} catch (InterruptedException e) {
LOG.info(name +" thread interrupted.");
} catch (Exception e) {
if (n > 0) {
int retryWindow = Math.max(0, 1000 / 2 * retryDelay - minDelay);
int awhile = rng.nextInt(retryWindow) + minDelay;
if (!inError) {
LOG.error("Got sink exception, retry in "+ awhile +"ms", e);
}
retryDelay *= retryBackoff;
try { Thread.sleep(awhile); }
catch (InterruptedException e2) {
LOG.info(name +" thread interrupted while waiting for retry", e2);
}
--n;
} else {
if (!inError) {
LOG.error("Got sink exception and over retry limit, "+
"suppressing further error messages", e);
}
queue.clear();
refreshQueueSizeGauge();
inError = true; // Don't keep complaining ad infinitum
}
}
}
}
{code}
It looks the change of these in catch block code will not comes into effect. I
think we should remove these two line because the values have been initalized
before in {{publishMetricsFromQueue}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303361#comment-15303361 ] shimingfei commented on HADOOP-12756: - Hi Kai, The credential info doesn't change, it is created for Hadoop community to do the functional test, and we can't expose it to all Hadoop users. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303355#comment-15303355 ] Kai Zheng commented on HADOOP-12756: bq. But how will the account info and other test properties be configured and tests can be properly triggered? Do these info change? Does the account info need to be protected or is it suitable to be exposed in this open source world? If the aliyun keys can be configured in the test resource file, why such info isn't suitable in the similar way? > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303346#comment-15303346 ] Ling Zhou commented on HADOOP-12756: Hi everyone. Our patch is updated. The code is after the latest hadoop trunk. 1. OSS keys are passed via the test/resources/auth-keys.xml file, the same way as S3A. 2. Module is renamed to hadoop-aliyun. 3. Since httpclient in hadoop trunk is updated to 4.5.2, the dependency conflict issue is solved. We have prepared individual Aliyun OSS account to test this patch. But how will the account info and other test properties be configured and tests can be properly triggered? Thank you for your response. > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shimingfei updated HADOOP-12756: Attachment: (was: 0001-HADOOP-12756.-Aliyun-OSS-filesystem-integration-with.patch) > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shimingfei updated HADOOP-12756: Attachment: HADOOP-12756-v02.patch > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS > integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303332#comment-15303332
]
Yongjun Zhang commented on HADOOP-12847:
Thanks [~jojochuang] for the work here, +1 on rev 10. Will commit soon,
> hadoop daemonlog should support https and SPNEGO for Kerberized cluster
> ---
>
> Key: HADOOP-12847
> URL: https://issues.apache.org/jira/browse/HADOOP-12847
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12847.001.patch, HADOOP-12847.002.patch,
> HADOOP-12847.003.patch, HADOOP-12847.004.patch, HADOOP-12847.005.patch,
> HADOOP-12847.006.patch, HADOOP-12847.008.patch, HADOOP-12847.009.patch,
> HADOOP-12847.010.branch-2.patch, HADOOP-12847.010.patch
>
>
> {{hadoop daemonlog}} is a simple, yet useful tool for debugging.
> However, it does not support https, nor does it support a Kerberized Hadoop
> cluster.
> Using {{AuthenticatedURL}}, it will be able to support SPNEGO negotiation
> with a Kerberized name node web ui. It will also fall back to simple
> authentication if the cluster is not Kerberized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shimingfei updated HADOOP-12756: Attachment: 0001-HADOOP-12756.-Aliyun-OSS-filesystem-integration-with.patch > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: > 0001-HADOOP-12756.-Aliyun-OSS-filesystem-integration-with.patch, HCFS User > manual.md, OSS integration.pdf, OSS integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
[ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ling Zhou updated HADOOP-12756: --- Attachment: (was: 0001-OSS-filesystem-integration-with-Hadoop.patch) > Incorporate Aliyun OSS file system implementation > - > > Key: HADOOP-12756 > URL: https://issues.apache.org/jira/browse/HADOOP-12756 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.0 >Reporter: shimingfei >Assignee: shimingfei > Attachments: HCFS User manual.md, OSS integration.pdf, OSS > integration.pdf > > > Aliyun OSS is widely used among China’s cloud users, but currently it is not > easy to access data laid on OSS storage from user’s Hadoop/Spark application, > because of no original support for OSS in Hadoop. > This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, > Spark/Hadoop applications can read/write data from OSS without any code > change. Narrowing the gap between user’s APP and data storage, like what have > been done for S3 in Hadoop -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303295#comment-15303295
]
Andrew Wang commented on HADOOP-12910:
--
We could shade Deferred so it's not an external dependency, or maybe even
copy-paste it into Hadoop. It's been pretty stable if you look at the commit
history:
https://github.com/OpenTSDB/async/commits/master/src/Deferred.java
bq. I suggest returning Future (or a sub-interface to support callbacks) in
branch 2 and CompletableFuture (or our own implementation of CompletableFuture)
in trunk. In this way, trunk is backward compatible to branch 2 since
CompletableFuture implements Future.
I like this suggestion. If there's demand for callbacks in branch-2 then we
might as well use Deferred everywhere, but if Future is sufficient for the
current set of usecases, then let's go with this plan.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[
https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-12893:
-
Attachment: HADOOP-12893.007.patch
I made a few improvements to Xiao's patch:
* Tweaked the pom dependencies so it works with a clean m2 cache
* Removed "-" from new module name for consistency
* Changed the LICENSE.txt wording to reflect that these new additions are only
for the binary distribution
* Pulled mockito and slf4j to their own copy of the MIT license, since the
existing version for bootstrap has the Twitter copyright
I wrote the following script to make sure all the generated JARs have the
LICENSE and NOTICE:
{code}
for f in $(find . -name "hadoop*SNAPSHOT.jar"); do
jar -tf $f | grep "LICENSE" > /dev/null
RET1=$?
jar -tf $f | grep "NOTICE" > /dev/null
RET2=$?
if [ $RET1 -ne 0 -a $RET2 -ne 0 ]; then
echo $f "missing LICENSE and NOTICE!";
elif [ $RET1 -ne 0 ]; then
echo $f "missing LICENSE!";
elif [ $RET2 -ne 0 ]; then
echo $f "missing NOTICE!";
fi
done
{code}
This is the output:
{quote}
./hadoop-project-dist/target/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT.jar
missing LICENSE and NOTICE!
./hadoop-project-dist/target/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT/share/hadoop/UNDEF/hadoop-project-dist-3.0.0-alpha1-SNAPSHOT.jar
missing LICENSE and NOTICE!
./hadoop-build-tools/target/hadoop-build-tools-3.0.0-alpha1-SNAPSHOT.jar
missing LICENSE and NOTICE!
{quote}
This is because these inherit from hadoop-main rather than hadoop-project,
which is where the remote-resources-plugin is called. There's not much in these
JARs, so we might not need to generate/deploy them. If we do, then we can do
some special-case L work.
> Verify LICENSE.txt and NOTICE.txt
> -
>
> Key: HADOOP-12893
> URL: https://issues.apache.org/jira/browse/HADOOP-12893
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-12893.002.patch, HADOOP-12893.003.patch,
> HADOOP-12893.004.patch, HADOOP-12893.005.patch, HADOOP-12893.006.patch,
> HADOOP-12893.007.patch, HADOOP-12893.01.patch
>
>
> We have many bundled dependencies in both the source and the binary artifacts
> that are not in LICENSE.txt and NOTICE.txt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13209) replace slaves with workers
[ https://issues.apache.org/jira/browse/HADOOP-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Smith updated HADOOP-13209: Attachment: HADOOP-13209.v01.patch > replace slaves with workers > --- > > Key: HADOOP-13209 > URL: https://issues.apache.org/jira/browse/HADOOP-13209 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: John Smith > Attachments: HADOOP-13209.v01.patch > > > slaves.sh and the slaves file should get replace with workers.sh and a > workers file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13209) replace slaves with workers
[ https://issues.apache.org/jira/browse/HADOOP-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Smith updated HADOOP-13209: Status: Patch Available (was: Open) > replace slaves with workers > --- > > Key: HADOOP-13209 > URL: https://issues.apache.org/jira/browse/HADOOP-13209 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts >Reporter: John Smith > Attachments: HADOOP-13209.v01.patch > > > slaves.sh and the slaves file should get replace with workers.sh and a > workers file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303246#comment-15303246
]
Hadoop QA commented on HADOOP-12847:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 27s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 10s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 10s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 10 unchanged - 13 fixed = 10 total (was 23) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 50 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 34s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 42s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 0s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Timed out junit tests |
org.apache.hadoop.http.TestHttpServerLifecycle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:babe025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806509/HADOOP-12847.010.branch-2.patch
|
| JIRA Issue | HADOOP-12847 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7d2b58a749bf 3.13.0-36-lowlatency #63-Ubuntu SMP
[jira] [Commented] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303230#comment-15303230
]
Chris Nauroth commented on HADOOP-13171:
The new stats are very thorough. The refactoring of {{ProgressListener}} and
pulling encryption logic out of the output streams is nice too. Thank you,
Steve!
{{TestS3ADirectoryPerformance#testListOperations}} is timing out for me
consistently in my runs against US-west-2. It appears to be making progress,
but not quickly enough.
{code}
int scale = (int)getOperationCount();
int width = 2 * scale;
int depth = 2 * scale;
int files = 2 * scale;
{code}
Are you sure this is what you wanted for these factors? Please check my math
here, but I believe the interaction between width and depth means the total
number of directories grows exponentially with respect to scale, something like
(2*scale) ^ (2*scale) + 1 (for the root). Then, considering the file count per
directory, we have a total of ((2*scale) ^ (2*scale) + 1) * (2 * scale) files.
The default operation count is 2005, so these are big numbers. Even if I
down-tune to scale.test.operation.count=4, it still takes a long time.
Please add an {{@Override}} annotation to
{{ProgressableProgressListener#progressChanged}}.
I think {{ProgressableProgressListener#getLastBytesTransferred}} is no longer
called anywhere and can be removed.
Maybe I'm missing something, but it appears that after execution of the base
{{FileSystem#initialize}}, it's impossible for {{FileSystem#statistics}} to be
null. Therefore, the null guards in the increment methods should be
unnecessary. If you prefer to keep them in the spirit of defensive coding, I'm
fine with that too.
{code}
private Iterator> iterator =
opsCount.entrySet().iterator();
{code}
I suggest changing to the following...
{code}
private Iterator> iterator =
Collections.unmodifiableSet(opsCount.entrySet()).iterator();
{code}
...so that we block callers from removing entries through the iterator. That's
probably applicable to the equivalent code up in hadoop-common too, but
probably best to keep the scope focused just on the hadoop-aws code here.
The JavaDocs for {{S3ATestUtils#createSubdirs}} aren't at the right indentation
level.
Repeating a comment from HADOOP-13131, {{getS3AFS}} might instead be an
override of {{getFileSystem}} with a covariant return type.
{code}
@Test
public void testCostOfCopyFromLocalFile() throws Throwable {
describe("testCostOfCopyFromLocalFile");
File tmpFile = File.createTempFile("tests3acost", ".txt");
{code}
I'd prefer to avoid accessing /tmp due to the potential for platform-specific
oddities, e.g. HADOOP-761 and MAPREDUCE-5191. Can we switch to something based
on {{getContract().getTestPath()}}?
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13073) RawLocalFileSystem does not react on changing umask
[
https://issues.apache.org/jira/browse/HADOOP-13073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303228#comment-15303228
]
Hadoop QA commented on HADOOP-13073:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-common-project/hadoop-common: The patch generated
6 new + 32 unchanged - 0 fixed = 38 total (was 32) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 28s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12801709/HADOOP-13073.01.patch
|
| JIRA Issue | HADOOP-13073 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f9a0cd3e4683 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5b41b28 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9595/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9595/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9595/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RawLocalFileSystem does not react on changing umask
> ---
>
> Key: HADOOP-13073
> URL: https://issues.apache.org/jira/browse/HADOOP-13073
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Andras Bokor
>Assignee: Andras Bokor
>
[jira] [Created] (HADOOP-13209) replace slaves with workers
John Smith created HADOOP-13209: --- Summary: replace slaves with workers Key: HADOOP-13209 URL: https://issues.apache.org/jira/browse/HADOOP-13209 Project: Hadoop Common Issue Type: Improvement Components: scripts Reporter: John Smith slaves.sh and the slaves file should get replace with workers.sh and a workers file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303154#comment-15303154
]
Hadoop QA commented on HADOOP-12847:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 9 unchanged - 13 fixed = 9 total (was 22) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 51s
{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 15s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806500/HADOOP-12847.010.patch
|
| JIRA Issue | HADOOP-12847 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8b8451f6ab47 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5b41b28 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9593/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9593/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> hadoop daemonlog should support https and SPNEGO for Kerberized cluster
> ---
>
> Key: HADOOP-12847
> URL: https://issues.apache.org/jira/browse/HADOOP-12847
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12847.001.patch, HADOOP-12847.002.patch,
> HADOOP-12847.003.patch,
[jira] [Assigned] (HADOOP-7363) TestRawLocalFileSystemContract is needed
[ https://issues.apache.org/jira/browse/HADOOP-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor reassigned HADOOP-7363: Assignee: Andras Bokor (was: Matt Foley) > TestRawLocalFileSystemContract is needed > > > Key: HADOOP-7363 > URL: https://issues.apache.org/jira/browse/HADOOP-7363 > Project: Hadoop Common > Issue Type: Test > Components: fs >Affects Versions: 0.23.0 >Reporter: Matt Foley >Assignee: Andras Bokor > Attachments: HADOOP-7363.01.patch > > > FileSystemContractBaseTest is supposed to be run with each concrete > FileSystem implementation to insure adherence to the "contract" for > FileSystem behavior. However, currently only HDFS and S3 do so. > RawLocalFileSystem, at least, needs to be added. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13073) RawLocalFileSystem does not react on changing umask
[ https://issues.apache.org/jira/browse/HADOOP-13073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Bokor updated HADOOP-13073: -- Affects Version/s: (was: 0.23.0) Status: Patch Available (was: Open) > RawLocalFileSystem does not react on changing umask > --- > > Key: HADOOP-13073 > URL: https://issues.apache.org/jira/browse/HADOOP-13073 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Andras Bokor >Assignee: Andras Bokor > Attachments: HADOOP-13073.01.patch > > > FileSystemContractBaseTest#testMkdirsWithUmask is changing umask under the > filesystem. RawLocalFileSystem reads the config on startup so it will not > react if we change the umask. > It blocks [HADOOP-7363|https://issues.apache.org/jira/browse/HADOOP-7363] > since testMkdirsWithUmask test will never work with RawLocalFileSystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12855) Add option to disable JVMPauseMonitor across services
[
https://issues.apache.org/jira/browse/HADOOP-12855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303121#comment-15303121
]
John Zhuge edited comment on HADOOP-12855 at 5/26/16 11:12 PM:
---
Hi [~ste...@apache.org], looking back, I do not like the designs in
HADOOP-12946 and HADOOP-12908 which both rely heavily on static variable and
atomic operation.
{{JVMPauseMonitor}} should NOT be embedded in other service. JVM should start
one and only copy of {{JVMPauseMonitor}} and then start other services. A
typical main function:
{code}
main(String args[]) {
new JVMPauseMonitor(new Configuration()).start();
DataNode datanode = createDataNode(args, null, resources);
}
{code}
was (Author: jzhuge):
Hi [~ste...@apache.org], looking back, I do not like the designs in
HADOOP-12946 and HADOOP-12908 which both rely heavily on static variable and
atomic operation.
{{JVMPauseMonitor}} should NOT be embedded in other service. When one JVM
starts, it should start one and only copy of {{JVMPauseMonitor}} and then start
other services. A typical main function can be:
{code}
main(String args[]) {
Configuration conf = new SomeConfiguration();
new JVMPauseMonitor(conf).start();
DataNode datanode = createDataNode(args, conf, resources);
}
{code}
> Add option to disable JVMPauseMonitor across services
> -
>
> Key: HADOOP-12855
> URL: https://issues.apache.org/jira/browse/HADOOP-12855
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, test
>Affects Versions: 2.8.0
> Environment: JVMs with miniHDFS and miniYarn clusters
>Reporter: Steve Loughran
>Assignee: John Zhuge
> Attachments: HADOOP-12855-001.patch, HADOOP-12855-002.patch,
> HADOOP-12855-003.patch, HADOOP-12855-004.patch, HADOOP-12855-005.patch,
> HADOOP-12855-option-001.patch, HADOOP-12855-option-002.patch
>
>
> Now that the YARN and HDFS services automatically start a JVM pause monitor,
> if you start up the mini HDFS and YARN clusters, with history server, you are
> spinning off 5 + threads, all looking for JVM pauses, all printing things out
> when it happens.
> We do not need these monitors in minicluster testing; they merely add load
> and noise to tests.
> Rather than retrofit new options everywhere, how about having a
> "jvm.pause.monitor.enabled" flag (default true), which, when set, starts off
> the monitor thread.
> That way, the existing code is unchanged, there is always a JVM pause monitor
> for the various services —it just isn't spinning up threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12855) Add option to disable JVMPauseMonitor across services
[
https://issues.apache.org/jira/browse/HADOOP-12855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303121#comment-15303121
]
John Zhuge commented on HADOOP-12855:
-
Hi [~ste...@apache.org], looking back, I do not like the designs in
HADOOP-12946 and HADOOP-12908 which both rely heavily on static variable and
atomic operation.
{{JVMPauseMonitor}} should NOT be embedded in other service. When one JVM
starts, it should start one and only copy of {{JVMPauseMonitor}} and then start
other services. A typical main function can be:
{code}
main(String args[]) {
Configuration conf = new SomeConfiguration();
new JVMPauseMonitor(conf).start();
DataNode datanode = createDataNode(args, conf, resources);
}
{code}
> Add option to disable JVMPauseMonitor across services
> -
>
> Key: HADOOP-12855
> URL: https://issues.apache.org/jira/browse/HADOOP-12855
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, test
>Affects Versions: 2.8.0
> Environment: JVMs with miniHDFS and miniYarn clusters
>Reporter: Steve Loughran
>Assignee: John Zhuge
> Attachments: HADOOP-12855-001.patch, HADOOP-12855-002.patch,
> HADOOP-12855-003.patch, HADOOP-12855-004.patch, HADOOP-12855-005.patch,
> HADOOP-12855-option-001.patch, HADOOP-12855-option-002.patch
>
>
> Now that the YARN and HDFS services automatically start a JVM pause monitor,
> if you start up the mini HDFS and YARN clusters, with history server, you are
> spinning off 5 + threads, all looking for JVM pauses, all printing things out
> when it happens.
> We do not need these monitors in minicluster testing; they merely add load
> and noise to tests.
> Rather than retrofit new options everywhere, how about having a
> "jvm.pause.monitor.enabled" flag (default true), which, when set, starts off
> the monitor thread.
> That way, the existing code is unchanged, there is always a JVM pause monitor
> for the various services —it just isn't spinning up threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-12847:
-
Attachment: HADOOP-12847.010.branch-2.patch
Attach the branch-2 patch. The default NN port has been changed in trunk, so
need to update the doc.
> hadoop daemonlog should support https and SPNEGO for Kerberized cluster
> ---
>
> Key: HADOOP-12847
> URL: https://issues.apache.org/jira/browse/HADOOP-12847
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12847.001.patch, HADOOP-12847.002.patch,
> HADOOP-12847.003.patch, HADOOP-12847.004.patch, HADOOP-12847.005.patch,
> HADOOP-12847.006.patch, HADOOP-12847.008.patch, HADOOP-12847.009.patch,
> HADOOP-12847.010.branch-2.patch, HADOOP-12847.010.patch
>
>
> {{hadoop daemonlog}} is a simple, yet useful tool for debugging.
> However, it does not support https, nor does it support a Kerberized Hadoop
> cluster.
> Using {{AuthenticatedURL}}, it will be able to support SPNEGO negotiation
> with a Kerberized name node web ui. It will also fall back to simple
> authentication if the cluster is not Kerberized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-12847:
-
Attachment: HADOOP-12847.010.patch
> hadoop daemonlog should support https and SPNEGO for Kerberized cluster
> ---
>
> Key: HADOOP-12847
> URL: https://issues.apache.org/jira/browse/HADOOP-12847
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12847.001.patch, HADOOP-12847.002.patch,
> HADOOP-12847.003.patch, HADOOP-12847.004.patch, HADOOP-12847.005.patch,
> HADOOP-12847.006.patch, HADOOP-12847.008.patch, HADOOP-12847.009.patch,
> HADOOP-12847.010.patch
>
>
> {{hadoop daemonlog}} is a simple, yet useful tool for debugging.
> However, it does not support https, nor does it support a Kerberized Hadoop
> cluster.
> Using {{AuthenticatedURL}}, it will be able to support SPNEGO negotiation
> with a Kerberized name node web ui. It will also fall back to simple
> authentication if the cluster is not Kerberized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303073#comment-15303073
]
Hadoop QA commented on HADOOP-12847:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 19s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} hadoop-common-project/hadoop-common: The patch
generated 0 new + 9 unchanged - 13 fixed = 9 total (was 22) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 40s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.ssl.TestReloadingX509TrustManager |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806485/HADOOP-12847.010.patch
|
| JIRA Issue | HADOOP-12847 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6b1e37dbbbc1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 55c3e2d |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9592/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9592/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9592/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9592/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> hadoop daemonlog should support https and SPNEGO for Kerberized cluster
>
[jira] [Updated] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-12847:
-
Attachment: (was: HADOOP-12847.010.patch)
> hadoop daemonlog should support https and SPNEGO for Kerberized cluster
> ---
>
> Key: HADOOP-12847
> URL: https://issues.apache.org/jira/browse/HADOOP-12847
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12847.001.patch, HADOOP-12847.002.patch,
> HADOOP-12847.003.patch, HADOOP-12847.004.patch, HADOOP-12847.005.patch,
> HADOOP-12847.006.patch, HADOOP-12847.008.patch, HADOOP-12847.009.patch
>
>
> {{hadoop daemonlog}} is a simple, yet useful tool for debugging.
> However, it does not support https, nor does it support a Kerberized Hadoop
> cluster.
> Using {{AuthenticatedURL}}, it will be able to support SPNEGO negotiation
> with a Kerberized name node web ui. It will also fall back to simple
> authentication if the cluster is not Kerberized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13175) Remove hadoop-ant from hadoop-tools
[ https://issues.apache.org/jira/browse/HADOOP-13175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HADOOP-13175: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) > Remove hadoop-ant from hadoop-tools > --- > > Key: HADOOP-13175 > URL: https://issues.apache.org/jira/browse/HADOOP-13175 > Project: Hadoop Common > Issue Type: Task >Reporter: Chris Douglas > Attachments: HADOOP-13175.001.patch > > > The hadoop-ant code is an ancient kludge unlikely to have any users, still. > We can delete it from trunk as a "scream test" for 3.x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13131) Add tests to verify that S3A supports SSE-S3 encryption
[
https://issues.apache.org/jira/browse/HADOOP-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303022#comment-15303022
]
Chris Nauroth commented on HADOOP-13131:
One more thing: consider adding a JUnit {{Timeout}} to {{AbstractS3ATestBase}}.
> Add tests to verify that S3A supports SSE-S3 encryption
> ---
>
> Key: HADOOP-13131
> URL: https://issues.apache.org/jira/browse/HADOOP-13131
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13131-001.patch, HADOOP-13131-002.patch,
> HADOOP-13131-003.patch, HADOOP-13131-004.patch, HADOOP-13131-005.patch,
> HADOOP-13131-branch-2-006.patch, HADOOP-13131-branch-2-007.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Although S3A claims to support server-side S3 encryption (and does, if you
> set the option), we don't have any test to verify this. Of course, as the
> encryption is transparent, it's hard to test.
> Here's what I propose
> # a test which sets encryption = AES256; expects things to work as normal.
> # a test which sets encyption = DES and expects any operation creating a file
> or directory to fail with a 400 "bad request" error
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client
[
https://issues.apache.org/jira/browse/HADOOP-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303015#comment-15303015
]
Yongjun Zhang commented on HADOOP-13206:
Hi [~zhz],
Thanks for the new rev.
Some questions and comments:
{code}
60try {
61 serviceMatch =
NetUtils.createSocketAddr(token.getService().toString()).
62 equals(NetUtils.createSocketAddr(service.toString()));
63} catch (IllegalArgumentException e) {
64 SecurityUtil.LOG.debug("service " + service + " or token
service " +
65 token.getService() + " is not in host:port format.");
66}
{code}
1. Do we expect the to be either host name or ip address, or only host
name is allowed?
2. Do we intend to support both hostname and ip address formats here? Based on
my read of the jira description, seems we intend to support both
3. Is the msg level DEBUG sufficient? I guess we might see too many messages if
we change it to WARN?
4. Suggest to do
{code}
64 SecurityUtil.LOG.("service " + service + " or token
service " +
65 token.getService() + " is not in host:port format.", e);
{code}
Thanks.
> Delegation token cannot be fetched and used by different versions of client
> ---
>
> Key: HADOOP-13206
> URL: https://issues.apache.org/jira/browse/HADOOP-13206
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.3.0, 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HADOOP-13206.00.patch, HADOOP-13206.01.patch
>
>
> We have observed that an HDFS delegation token fetched by a 2.3.0 client
> cannot be used by a 2.6.1 client, and vice versa. Through some debugging I
> found that it's a mismatch between the token's {{service}} and the
> {{service}} of the filesystem (e.g. {{webhdfs://host.something.com:50070/}}).
> One would be in numerical IP address and one would be in non-numerical
> hostname format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303008#comment-15303008
]
Tsz Wo Nicholas Sze commented on HADOOP-12910:
--
> Can we introduce our own ListenableFuture class? Or use callbacks instead of
> Future? Listener or callback is very useful for writing event-driven programs.
It seems like a good idea to have our own Future sub-interface.
> Use Deferred, ...
I guess it may not be a good idea since it introduces an external dependency at
API level.
> Target this for 3.0 and use CompletableFuture. ...
Why making the entire async feature unavailable in branch 2? CompletableFuture
is nice-to-have but not must-have.
I suggest returning Future (or a sub-interface to support callbacks) in branch
2 and CompletableFuture (or our own implementation of CompletableFuture) in
trunk. In this way, trunk is backward compatible to branch 2 since
CompletableFuture implements Future.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12847) hadoop daemonlog should support https and SPNEGO for Kerberized cluster
[
https://issues.apache.org/jira/browse/HADOOP-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-12847:
-
Attachment: HADOOP-12847.010.patch
v10. minor code refactoring.
> hadoop daemonlog should support https and SPNEGO for Kerberized cluster
> ---
>
> Key: HADOOP-12847
> URL: https://issues.apache.org/jira/browse/HADOOP-12847
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
> Attachments: HADOOP-12847.001.patch, HADOOP-12847.002.patch,
> HADOOP-12847.003.patch, HADOOP-12847.004.patch, HADOOP-12847.005.patch,
> HADOOP-12847.006.patch, HADOOP-12847.008.patch, HADOOP-12847.009.patch,
> HADOOP-12847.010.patch
>
>
> {{hadoop daemonlog}} is a simple, yet useful tool for debugging.
> However, it does not support https, nor does it support a Kerberized Hadoop
> cluster.
> Using {{AuthenticatedURL}}, it will be able to support SPNEGO negotiation
> with a Kerberized name node web ui. It will also fall back to simple
> authentication if the cluster is not Kerberized.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13131) Add tests to verify that S3A supports SSE-S3 encryption
[ https://issues.apache.org/jira/browse/HADOOP-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302954#comment-15302954 ] Steve Loughran commented on HADOOP-13131: - thanks for the feedback, I'll review. And yes, I did try to cleanup some of the setup. Even so, good point about checking both codepaths, hadn't thought of that. > Add tests to verify that S3A supports SSE-S3 encryption > --- > > Key: HADOOP-13131 > URL: https://issues.apache.org/jira/browse/HADOOP-13131 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: HADOOP-13131-001.patch, HADOOP-13131-002.patch, > HADOOP-13131-003.patch, HADOOP-13131-004.patch, HADOOP-13131-005.patch, > HADOOP-13131-branch-2-006.patch, HADOOP-13131-branch-2-007.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Although S3A claims to support server-side S3 encryption (and does, if you > set the option), we don't have any test to verify this. Of course, as the > encryption is transparent, it's hard to test. > Here's what I propose > # a test which sets encryption = AES256; expects things to work as normal. > # a test which sets encyption = DES and expects any operation creating a file > or directory to fail with a 400 "bad request" error -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13207) Specify FileSystem listStatus and listFiles
[
https://issues.apache.org/jira/browse/HADOOP-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13207:
Attachment: HADOOP-13207-branch-2-001.patch
Patch 001; no proofreading. Covers listStatus, listFiles. Avoids globStatus and
intends to stay that way.
No tests; there some against localFS, a couple in Swift I recall writing in a
hurry. They can be added to a new AbstractListOperations contract test.
As most implementations of the more complex ones (iterators, aggregators) defer
to the baseline ones, even the object stores *should* work; I'll just need to
spin waiting for the directory creations to become visible
> Specify FileSystem listStatus and listFiles
> ---
>
> Key: HADOOP-13207
> URL: https://issues.apache.org/jira/browse/HADOOP-13207
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-13207-branch-2-001.patch
>
>
> The many `listStatus`, `listLocatedStatus` and `listFiles` operations have
> not been completely covered in the FS specification. There's lots of implicit
> use of {{listStatus()}} path, but no coverage or tests of the others.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13131) Add tests to verify that S3A supports SSE-S3 encryption
[
https://issues.apache.org/jira/browse/HADOOP-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302937#comment-15302937
]
Chris Nauroth commented on HADOOP-13131:
Oh, I see now that HADOOP-13171 is refactoring encryption logic out of
{{S3AFastOutputStream}} and into {{S3AFileSystem}} for better encapsulation. I
retract my statement about adding a test covering {{S3AFastOutputStream}}.
After that refactoring, the additional test wouldn't provide much benefit.
> Add tests to verify that S3A supports SSE-S3 encryption
> ---
>
> Key: HADOOP-13131
> URL: https://issues.apache.org/jira/browse/HADOOP-13131
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13131-001.patch, HADOOP-13131-002.patch,
> HADOOP-13131-003.patch, HADOOP-13131-004.patch, HADOOP-13131-005.patch,
> HADOOP-13131-branch-2-006.patch, HADOOP-13131-branch-2-007.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Although S3A claims to support server-side S3 encryption (and does, if you
> set the option), we don't have any test to verify this. Of course, as the
> encryption is transparent, it's hard to test.
> Here's what I propose
> # a test which sets encryption = AES256; expects things to work as normal.
> # a test which sets encyption = DES and expects any operation creating a file
> or directory to fail with a 400 "bad request" error
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13208) listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories
[
https://issues.apache.org/jira/browse/HADOOP-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302913#comment-15302913
]
Steve Loughran commented on HADOOP-13208:
-
One little detail, nothing has been written down about how listFiles() really
works, there are no FS contract tests. HADOOP-13207 covers this.
> listFiles(recursive=true) to do a bulk listObjects instead of walking the
> pseudo-tree of directories
>
>
> Key: HADOOP-13208
> URL: https://issues.apache.org/jira/browse/HADOOP-13208
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> A major cost in split calculation against object stores turns out be listing
> the directory tree itself. That's because against S3, it takes S3A two HEADs
> and two lists to list the content of any directory path (2 HEADs + 1 list for
> getFileStatus(); the next list to query the contents).
> Listing a directory could be improved slightly by combining the final two
> listings. However, a listing of a directory tree will still be
> O(directories). In contrast, a recursive {{listFiles()}} operation should be
> implementable by a bulk listing of all descendant paths; one List operation
> per thousand descendants.
> As the result of this call is an iterator, the ongoing listing can be
> implemented within the iterator itself
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13208) listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories
Steve Loughran created HADOOP-13208:
---
Summary: listFiles(recursive=true) to do a bulk listObjects
instead of walking the pseudo-tree of directories
Key: HADOOP-13208
URL: https://issues.apache.org/jira/browse/HADOOP-13208
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
A major cost in split calculation against object stores turns out be listing
the directory tree itself. That's because against S3, it takes S3A two HEADs
and two lists to list the content of any directory path (2 HEADs + 1 list for
getFileStatus(); the next list to query the contents).
Listing a directory could be improved slightly by combining the final two
listings. However, a listing of a directory tree will still be O(directories).
In contrast, a recursive {{listFiles()}} operation should be implementable by a
bulk listing of all descendant paths; one List operation per thousand
descendants.
As the result of this call is an iterator, the ongoing listing can be
implemented within the iterator itself
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13131) Add tests to verify that S3A supports SSE-S3 encryption
[
https://issues.apache.org/jira/browse/HADOOP-13131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302902#comment-15302902
]
Chris Nauroth commented on HADOOP-13131:
Patch v007 looks good. Thank you, Steve. I had a successful full test run in
parallel mode against US-west-2.
{{S3AFastOutputStream}} also supports server-side encryption. What would you
think of adding a test covering that?
Aside from that, I have some minor comments. Much of this is subjective, so
feel free to take it or leave it as you please.
Should {{S3AFileSystem#getObjectMetadata}} be annotated {{VisibleForTesting}}?
I think I see why there are 2 overloads of {{getObjectMetadata}}. It
simplifies HADOOP-13171, right? However, I'm not sure why
{{getObjectMetadata(String)}} is protected. Should that be private?
{code}
the S3 protocols to the extent that the amazon s3 SDK is capable of talking
{code}
Suggest capitalizing "Amazon S3".
{code}
@BeforeClass
public static void nameThread() {
Thread.currentThread().setName("JUnit");
}
{code}
A possible evolution of this could be to switch to a {{Before}} method and
change the thread name to "JUnit-" + {{methodName}}, so that the thread name is
specific to the test being run.
{code}
public TestS3AEncryption() {
}
{code}
No-op default constructor can be removed?
{code}
@Override
protected AbstractFSContract createContract(Configuration conf) {
S3ATestUtils.disableFilesystemCaching(conf);
conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM,
AES256);
return new S3AContract(conf);
}
{code}
Would it be better to do this config setup by overriding
{{AbstractFSContractTestBase#createConfiguration}}? Then, the subclasses
wouldn't need to instantiate {{S3AContract}} and could instead rely on
{{AbstractS3ATestBase#createContract}}.
{code}
protected S3AFileSystem getS3AFS() {
return (S3AFileSystem) getFileSystem();
}
{code}
Instead of adding a new method, I suggest overriding
{{AbstractFSContractTestBase#getFileSystem}}, with the method's return type
changed to {{S3AFileSystem}}, since Java allows covariant return types.
> Add tests to verify that S3A supports SSE-S3 encryption
> ---
>
> Key: HADOOP-13131
> URL: https://issues.apache.org/jira/browse/HADOOP-13131
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13131-001.patch, HADOOP-13131-002.patch,
> HADOOP-13131-003.patch, HADOOP-13131-004.patch, HADOOP-13131-005.patch,
> HADOOP-13131-branch-2-006.patch, HADOOP-13131-branch-2-007.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Although S3A claims to support server-side S3 encryption (and does, if you
> set the option), we don't have any test to verify this. Of course, as the
> encryption is transparent, it's hard to test.
> Here's what I propose
> # a test which sets encryption = AES256; expects things to work as normal.
> # a test which sets encyption = DES and expects any operation creating a file
> or directory to fail with a 400 "bad request" error
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-13207) Specify FileSystem listStatus and listFiles
[
https://issues.apache.org/jira/browse/HADOOP-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-13207 started by Steve Loughran.
---
> Specify FileSystem listStatus and listFiles
> ---
>
> Key: HADOOP-13207
> URL: https://issues.apache.org/jira/browse/HADOOP-13207
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation, fs
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> The many `listStatus`, `listLocatedStatus` and `listFiles` operations have
> not been completely covered in the FS specification. There's lots of implicit
> use of {{listStatus()}} path, but no coverage or tests of the others.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13207) Specify FileSystem listStatus and listFiles
Steve Loughran created HADOOP-13207:
---
Summary: Specify FileSystem listStatus and listFiles
Key: HADOOP-13207
URL: https://issues.apache.org/jira/browse/HADOOP-13207
Project: Hadoop Common
Issue Type: Improvement
Components: documentation, fs
Affects Versions: 2.8.0
Reporter: Steve Loughran
Assignee: Steve Loughran
The many `listStatus`, `listLocatedStatus` and `listFiles` operations have not
been completely covered in the FS specification. There's lots of implicit use
of {{listStatus()}} path, but no coverage or tests of the others.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client
[
https://issues.apache.org/jira/browse/HADOOP-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302862#comment-15302862
]
Hadoop QA commented on HADOOP-13206:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 6s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 7s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} hadoop-common-project/hadoop-common: The patch generated
1 new + 2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 41s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 37s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806456/HADOOP-13206.01.patch
|
| JIRA Issue | HADOOP-13206 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0c3b20dbec3f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f4b9bcd |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9591/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9591/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9591/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9591/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302854#comment-15302854
]
stack commented on HADOOP-12910:
Good suggestion. We'll be back.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12910) Add new FileSystem API to support asynchronous method calls
[
https://issues.apache.org/jira/browse/HADOOP-12910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302850#comment-15302850
]
Tsz Wo Nicholas Sze commented on HADOOP-12910:
--
Okay, let's replay it.
> ... . It's not fair to push the burden of supporting multiple APIs onto our
> downstreams, ...
We are not going to support multiple APIs. Once we have decided the async API,
the unstable API can be removed. That is the meaning of "unstable".
The down streams are intelligent people. They can decide whether they want to
use the unstable API. It is even more unfair if we delay to provide any async
API to the down streams. No?
[~andrew.wang], is it your intention to slow down the async hdfs development? I
hope not.
> Add new FileSystem API to support asynchronous method calls
> ---
>
> Key: HADOOP-12910
> URL: https://issues.apache.org/jira/browse/HADOOP-12910
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: HADOOP-12910-HDFS-9924.000.patch,
> HADOOP-12910-HDFS-9924.001.patch, HADOOP-12910-HDFS-9924.002.patch
>
>
> Add a new API, namely FutureFileSystem (or AsynchronousFileSystem, if it is a
> better name). All the APIs in FutureFileSystem are the same as FileSystem
> except that the return type is wrapped by Future, e.g.
> {code}
> //FileSystem
> public boolean rename(Path src, Path dst) throws IOException;
> //FutureFileSystem
> public Future rename(Path src, Path dst) throws IOException;
> {code}
> Note that FutureFileSystem does not extend FileSystem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-7352) Contracts of LocalFileSystem and DistributedFileSystem should require FileSystem::listStatus throw IOException not return null upon access error
[
https://issues.apache.org/jira/browse/HADOOP-7352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302838#comment-15302838
]
Steve Loughran commented on HADOOP-7352:
Looking at the codepath fpr {{FileStatus[] listStatus(Path f, PathFilter
filter)
., it invokes {{abstract FileStatus[] listStatus(Path f) }} via
{{listStatus(ArrayList results, Path f, PathFilter filter)}}, and
includes a check for the return value being null.
That is, if you go LocalFileSystem.listStatus(path) , in some circumstances
null can be returned, but in listStatus(path, filter), you'll get an IOE back.
It should be an IOE everywhere.
> Contracts of LocalFileSystem and DistributedFileSystem should require
> FileSystem::listStatus throw IOException not return null upon access error
>
>
> Key: HADOOP-7352
> URL: https://issues.apache.org/jira/browse/HADOOP-7352
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs, fs/s3
>Reporter: Matt Foley
>Assignee: Matt Foley
>
> In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should
> throw FileNotFoundException instead of returning null, when the target
> directory did not exist.
> However, in LocalFileSystem implementation today, FileSystem::listStatus
> still may return null, when the target directory exists but does not grant
> read permission. This causes NPE in many callers, for all the reasons cited
> in HADOOP-6201 and HDFS-538. See HADOOP-7327 and its linked issues for
> examples.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client
[
https://issues.apache.org/jira/browse/HADOOP-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HADOOP-13206:
---
Attachment: HADOOP-13206.01.patch
Thanks [~yzhangal] for the review! Updating patch to address the comment.
Failed unit tests are unrelated and pass locally (for v0 patch).
> Delegation token cannot be fetched and used by different versions of client
> ---
>
> Key: HADOOP-13206
> URL: https://issues.apache.org/jira/browse/HADOOP-13206
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.3.0, 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HADOOP-13206.00.patch, HADOOP-13206.01.patch
>
>
> We have observed that an HDFS delegation token fetched by a 2.3.0 client
> cannot be used by a 2.6.1 client, and vice versa. Through some debugging I
> found that it's a mismatch between the token's {{service}} and the
> {{service}} of the filesystem (e.g. {{webhdfs://host.something.com:50070/}}).
> One would be in numerical IP address and one would be in non-numerical
> hostname format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-11820) aw jira testing, ignore
[
https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-11820:
--
Comment: was deleted
(was: | (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 00s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
00s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} machack {color} | {color:blue} 0m 00s
{color} | {color:blue} Applied YARN-5121 so that OS X works {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 08s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 13s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 32s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestSymlinkLocalFSFileContext |
| | hadoop.fs.TestSymlinkLocalFSFileSystem |
| | hadoop.net.unix.TestDomainSocket |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806168/socket.patch |
| JIRA Issue | HADOOP-11820 |
| Optional Tests | compile javac mvninstall unit |
| uname | Darwin Gavins-Mac-mini.local 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 |
| Build tool | maven |
| Personality |
/Users/jenkins/jenkins-home/workspace/Precommit-HADOOP-OSX/patchprocess/apache-yetus-8bf2ab7/precommit/personality/hadoop.sh
|
| git revision | trunk / 77d5ce9 |
| Default Java | 1.8.0_74 |
| unit |
https://builds.apache.org/job/Precommit-HADOOP-OSX/25/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/Precommit-HADOOP-OSX/25/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/Precommit-HADOOP-OSX/25/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/Precommit-HADOOP-OSX/25/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
)
> aw jira testing, ignore
> ---
>
> Key: HADOOP-11820
> URL: https://issues.apache.org/jira/browse/HADOOP-11820
> Project: Hadoop Common
> Issue Type: Task
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
> Attachments: 0001-domainsocket.patch, YARN-5132-v1.patch, socket.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client
[
https://issues.apache.org/jira/browse/HADOOP-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302734#comment-15302734
]
Yongjun Zhang commented on HADOOP-13206:
Hi [~zhz],
Thanks for reporting the issue and the patch. One minor comment, seems it's
better if we do the service match checking after we find {{if
(kindName.equals(token.getKind())}} is true.
> Delegation token cannot be fetched and used by different versions of client
> ---
>
> Key: HADOOP-13206
> URL: https://issues.apache.org/jira/browse/HADOOP-13206
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.3.0, 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HADOOP-13206.00.patch
>
>
> We have observed that an HDFS delegation token fetched by a 2.3.0 client
> cannot be used by a 2.6.1 client, and vice versa. Through some debugging I
> found that it's a mismatch between the token's {{service}} and the
> {{service}} of the filesystem (e.g. {{webhdfs://host.something.com:50070/}}).
> One would be in numerical IP address and one would be in non-numerical
> hostname format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client
[
https://issues.apache.org/jira/browse/HADOOP-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302723#comment-15302723
]
Hadoop QA commented on HADOOP-13206:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 40s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-common-project/hadoop-common: The patch generated
2 new + 2 unchanged - 0 fixed = 4 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 44s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
| | hadoop.net.TestDNS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806446/HADOOP-13206.00.patch
|
| JIRA Issue | HADOOP-13206 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 25cac643f459 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fed9bf0 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9590/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9590/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9590/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9590/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Commented] (HADOOP-12488) DomainSocket: Solaris does not support timeouts on AF_UNIX sockets
[
https://issues.apache.org/jira/browse/HADOOP-12488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302693#comment-15302693
]
Allen Wittenauer commented on HADOOP-12488:
---
In order to move this forward, I have a suggestion: let's merge HADOOP-12487
and HADOOP-12488 as a single patch. This gets us around the problem that they
are both dependent upon each other. There's pretty much no other way for us to
test and commit these without one being made an independent patch.
> DomainSocket: Solaris does not support timeouts on AF_UNIX sockets
> --
>
> Key: HADOOP-12488
> URL: https://issues.apache.org/jira/browse/HADOOP-12488
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: net
>Affects Versions: 2.7.1
> Environment: Solaris
>Reporter: Alan Burlison
>Assignee: Alan Burlison
> Attachments: HADOOP-12488.001.patch, HADOOP-12488.002.patch
>
>
> From the hadoop-common-dev mailing list:
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201509.mbox/%3c560b99f6.6010...@oracle.com%3E
> http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201510.mbox/%3c560ea6bf.2070...@oracle.com%3E
> {quote}
> Now that the Hadoop native code builds on Solaris I've been chipping
> away at all the test failures. About 50% of the failures involve
> DomainSocket, either directly or indirectly. That seems to be mainly
> because the tests use DomainSocket to do single-node testing, whereas in
> production it seems that DomainSocket is less commonly used
> (https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html).
> The particular problem on Solaris is that socket read/write timeouts
> (the SO_SNDTIMEO and SO_RCVTIMEO socket options) are not supported for
> UNIX domain (PF_UNIX) sockets. Those options are however supported for
> PF_INET sockets. That's because the socket implementation on Solaris is
> split roughly into two parts, for inet sockets and for STREAMS sockets,
> and the STREAMS implementation lacks support for SO_SNDTIMEO and
> SO_RCVTIMEO. As an aside, performance of sockets that use loopback or
> the host's own IP is slightly better than that of UNIX domain sockets on
> Solaris.
> I'm investigating getting timeouts supported for PF_UNIX sockets added
> to Solaris, but in the meantime I'm also looking how this might be
> worked around in Hadoop. One way would be to implement timeouts by
> wrapping all the read/write/send/recv etc calls in DomainSocket.c with
> either poll() or select().
> The basic idea is to add two new fields to DomainSocket.c to hold the
> read/write timeouts. On platforms that support SO_SNDTIMEO and
> SO_RCVTIMEO these would be unused as setsockopt() would be used to set
> the socket timeouts. On platforms such as Solaris the JNI code would use
> the values to implement the timeouts appropriately.
> To prevent the code in DomainSocket.c becoming a #ifdef hairball, the
> current socket IO function calls such as accept(), send(), read() etc
> would be replaced with a macros such as HD_ACCEPT. On platforms that
> provide timeouts these would just expand to the normal socket functions,
> on platforms that don't support timeouts it would expand to wrappers
> that implements timeouts for them.
> The only caveats are that all code that does anything to a PF_UNIX
> socket would *always* have to do so via DomainSocket. As far as I can
> tell that's not an issue, but it would have to be borne in mind if any
> changes were made in this area.
> Before I set about doing this, does the approach seem reasonable?
> {quote}
> {quote}
> Unfortunately it's not a simple as I'd hoped. For some reason I don't
> really understand, nearly all the JNI methods are declared as static and
> therefore don't get a "this" pointer and as a consequence all the class
> data members that are needed by the JNI code have to be passed in as
> parameters. That also means it's not possible to store the timeouts in
> the DomainSocket fields from within the JNI code. Most of the JNI
> methods should be instance methods rather than static ones, but making
> that change would require some significant surgery to DomainSocket.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client
[
https://issues.apache.org/jira/browse/HADOOP-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HADOOP-13206:
---
Attachment: HADOOP-13206.00.patch
> Delegation token cannot be fetched and used by different versions of client
> ---
>
> Key: HADOOP-13206
> URL: https://issues.apache.org/jira/browse/HADOOP-13206
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.3.0, 2.6.1
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HADOOP-13206.00.patch
>
>
> We have observed that an HDFS delegation token fetched by a 2.3.0 client
> cannot be used by a 2.6.1 client, and vice versa. Through some debugging I
> found that it's a mismatch between the token's {{service}} and the
> {{service}} of the filesystem (e.g. {{webhdfs://host.something.com:50070/}}).
> One would be in numerical IP address and one would be in non-numerical
> hostname format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client
Zhe Zhang created HADOOP-13206:
--
Summary: Delegation token cannot be fetched and used by different
versions of client
Key: HADOOP-13206
URL: https://issues.apache.org/jira/browse/HADOOP-13206
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.6.1, 2.3.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
We have observed that an HDFS delegation token fetched by a 2.3.0 client cannot
be used by a 2.6.1 client, and vice versa. Through some debugging I found that
it's a mismatch between the token's {{service}} and the {{service}} of the
filesystem (e.g. {{webhdfs://host.something.com:50070/}}). One would be in
numerical IP address and one would be in non-numerical hostname format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13206) Delegation token cannot be fetched and used by different versions of client
[
https://issues.apache.org/jira/browse/HADOOP-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HADOOP-13206:
---
Status: Patch Available (was: Open)
> Delegation token cannot be fetched and used by different versions of client
> ---
>
> Key: HADOOP-13206
> URL: https://issues.apache.org/jira/browse/HADOOP-13206
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.1, 2.3.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
>
> We have observed that an HDFS delegation token fetched by a 2.3.0 client
> cannot be used by a 2.6.1 client, and vice versa. Through some debugging I
> found that it's a mismatch between the token's {{service}} and the
> {{service}} of the filesystem (e.g. {{webhdfs://host.something.com:50070/}}).
> One would be in numerical IP address and one would be in non-numerical
> hostname format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11820) aw jira testing, ignore
[
https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302458#comment-15302458
]
Hadoop QA commented on HADOOP-11820:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 00s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
00s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} machack {color} | {color:blue} 0m 01s
{color} | {color:blue} Applied YARN-5121 so that YARN native on OS X works
{color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 09s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 37s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 7m 37s {color} |
{color:red} root generated 3 new + 25 unchanged - 2 fixed = 28 total (was 27)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 37s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 13s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 15s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 08s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestSymlinkLocalFSFileContext |
| | hadoop.fs.TestSymlinkLocalFSFileSystem |
| | hadoop.net.unix.TestDomainSocket |
| | hadoop.fs.TestEnhancedByteBufferAccess |
| | hadoop.hdfs.client.impl.TestBlockReaderFactory |
| | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation |
| | hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.namenode.TestEditLog |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead |
| | hadoop.hdfs.TestDFSInputStream |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestLargeBlock |
| | hadoop.hdfs.TestParallelShortCircuitRead |
| | hadoop.hdfs.TestParallelShortCircuitReadNoChecksum |
| | hadoop.hdfs.TestParallelShortCircuitReadUnCached |
| | hadoop.hdfs.TestParallelUnixDomainRead |
| | hadoop.hdfs.TestRead |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.tracing.TestTracingShortCircuitLocalRead |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806392/0001-domainsocket.patch
|
| JIRA Issue | HADOOP-11820 |
| Optional Tests | compile cc javac unit mvninstall |
| uname | Darwin Gavins-Mac-mini.local 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 |
| Build tool | maven |
| Personality |
/Users/jenkins/jenkins-home/workspace/Precommit-HADOOP-OSX/patchprocess/apache-yetus-8bf2ab7/precommit/personality/hadoop.sh
|
| git revision | trunk / fed9bf0 |
| Default Java | 1.8.0_74 |
| cc |
https://builds.apache.org/job/Precommit-HADOOP-OSX/28/artifact/patchprocess/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/Precommit-HADOOP-OSX/28/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/Precommit-HADOOP-OSX/28/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
[jira] [Resolved] (HADOOP-4109) FileSystem support for posix truncate method
[
https://issues.apache.org/jira/browse/HADOOP-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-4109.
---
Resolution: Duplicate
Truncate has been implemented, tracked in HDFS-3107.
> FileSystem support for posix truncate method
>
>
> Key: HADOOP-4109
> URL: https://issues.apache.org/jira/browse/HADOOP-4109
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Pete Wyckoff
>
> {code}
> void truncate(Path path, long offset) throws IOException;
> {code}
> from man truncate:
> DESCRIPTION
>The truncate and ftruncate functions cause the regular file named by
> path or referenced by fd to be truncated to a size of precisely length bytes.
>If the file previously was larger than this size, the extra data is
> lost. If the file previously was shorter, it is extended, and the extended
> part reads as zero bytes.
>The file pointer is not changed.
>If the size changed, then the ctime and mtime fields for the file are
> updated, and suid and sgid mode bits may be cleared.
>With ftruncate, the file must be open for writing; with truncate, the
> file must be writable.
> RETURN VALUE
>On success, zero is returned. On error, -1 is returned, and errno is
> set appropriately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11820) aw jira testing, ignore
[
https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302425#comment-15302425
]
Hadoop QA commented on HADOOP-11820:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 51s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 23s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 42s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 22s
{color} | {color:red} root: The patch generated 4 new + 174 unchanged - 5 fixed
= 178 total (was 179) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 23s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 0s {color} |
{color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 22s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 117m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
| | hadoop.hdfs.TestAsyncDFSRename |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806392/0001-domainsocket.patch
|
| JIRA Issue | HADOOP-11820 |
| Optional Tests | asflicense compile cc mvnsite javac unit javadoc
mvninstall findbugs checkstyle |
| uname | Linux a3397f726a00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / fed9bf0 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302190#comment-15302190
]
Hudson commented on HADOOP-12727:
-
SUCCESS: Integrated in Hadoop-trunk-Commit #9865 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9865/])
HADOOP-12727. Minor cleanups needed for CMake 3.X (Alan Burlison via aw) (aw:
rev fed9bf0ec1946674a58745455e478fc377ee4f98)
* hadoop-common-project/hadoop-common/HadoopCommon.cmake
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302180#comment-15302180
]
Hadoop QA commented on HADOOP-13171:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s {color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} branch-2 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} branch-2 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red} hadoop-tools/hadoop-aws: The patch generated 12 new + 53
unchanged - 13 fixed = 65 total (was 66) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.8.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s
{color} | {color:green} hadoop-aws in the patch passed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 54s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:babe025 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806385/HADOOP-13171-branch-2-007.patch
|
| JIRA Issue | HADOOP-13171 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 38d15c101fe2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
[jira] [Updated] (HADOOP-11820) aw jira testing, ignore
[ https://issues.apache.org/jira/browse/HADOOP-11820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11820: -- Attachment: 0001-domainsocket.patch > aw jira testing, ignore > --- > > Key: HADOOP-11820 > URL: https://issues.apache.org/jira/browse/HADOOP-11820 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer > Attachments: 0001-domainsocket.patch, YARN-5132-v1.patch, socket.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HADOOP-12727:
--
Resolution: Fixed
Fix Version/s: 3.0.0-alpha1
Status: Resolved (was: Patch Available)
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Fix For: 3.0.0-alpha1
>
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302158#comment-15302158
]
Allen Wittenauer commented on HADOOP-12727:
---
Those unit tests are known broken on OS X.
+1 committing to trunk.
Thanks!
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302151#comment-15302151
]
Hadoop QA commented on HADOOP-12727:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 00s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 00s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} machack {color} | {color:blue} 0m 01s
{color} | {color:blue} Applied YARN-5121 so that YARN native on OS X works
{color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 11s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 7m 33s {color} |
{color:red} root generated 2 new + 25 unchanged - 3 fixed = 27 total (was 28)
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 22s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestSymlinkLocalFSFileContext |
| | hadoop.fs.TestSymlinkLocalFSFileSystem |
| | hadoop.net.unix.TestDomainSocket |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12793718/HADOOP-12727.001.patch
|
| JIRA Issue | HADOOP-12727 |
| Optional Tests | compile cc javac unit |
| uname | Darwin Gavins-Mac-mini.local 13.2.0 Darwin Kernel Version 13.2.0: Thu
Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64 |
| Build tool | maven |
| Personality |
/Users/jenkins/jenkins-home/workspace/Precommit-HADOOP-OSX/patchprocess/apache-yetus-8bf2ab7/precommit/personality/hadoop.sh
|
| git revision | trunk / 77202fa |
| Default Java | 1.8.0_74 |
| cc |
https://builds.apache.org/job/Precommit-HADOOP-OSX/27/artifact/patchprocess/diff-compile-cc-root.txt
|
| unit |
https://builds.apache.org/job/Precommit-HADOOP-OSX/27/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/Precommit-HADOOP-OSX/27/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/Precommit-HADOOP-OSX/27/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/Precommit-HADOOP-OSX/27/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Attachment: HADOOP-13171-branch-2-007.patch
Patch 007
* lots more counting of what goes on in directory list (listFiles, listStatus,
globStatus) calls
* the full suite of getFileStatus operations
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Patch Available (was: Open)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch, HADOOP-13171-branch-2-007.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13171) Add StorageStatistics to S3A; instrument some more operations
[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-13171:
Status: Open (was: Patch Available)
> Add StorageStatistics to S3A; instrument some more operations
> -
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-13171-branch-2-001.patch,
> HADOOP-13171-branch-2-002.patch, HADOOP-13171-branch-2-003.patch,
> HADOOP-13171-branch-2-004.patch, HADOOP-13171-branch-2-005.patch,
> HADOOP-13171-branch-2-006.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13205) S3A to support custom retry policies; maybe failfast on unknown host
Steve Loughran created HADOOP-13205: --- Summary: S3A to support custom retry policies; maybe failfast on unknown host Key: HADOOP-13205 URL: https://issues.apache.org/jira/browse/HADOOP-13205 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 2.8.0 Reporter: Steve Loughran Noticed today that when connections are down, S3A retries on UnknownHostExceptions logging noisily in the process. # it should be possible to define or customize retry policies for an FS instance (fail fast, exponential backoff, etc) # we may want to explicitly have a fail-fast-if-offline retry policy, catching the common connectivity ones. Testing will be fun here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13202) Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might be changed
[
https://issues.apache.org/jira/browse/HADOOP-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302113#comment-15302113
]
Hadoop QA commented on HADOOP-13202:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s
{color} | {color:red} hadoop-common-project/hadoop-common: The patch generated
1 new + 47 unchanged - 0 fixed = 48 total (was 47) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 51s {color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12806365/HADOOP-13202.01.patch
|
| JIRA Issue | HADOOP-13202 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9f96fb531c7e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 77202fa |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9587/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9587/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9587/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9587/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/9587/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HADOOP-13205) S3A to support custom retry policies; maybe failfast on unknown host
[
https://issues.apache.org/jira/browse/HADOOP-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302106#comment-15302106
]
Steve Loughran commented on HADOOP-13205:
-
Stack trace you get today when offline
{code}
2016-05-26 13:40:00,979 [Thread-0] INFO http.AmazonHttpClient
(AmazonHttpClient.java:executeHelper(496)) - Unable to execute HTTP request:
steve-eu2.s3.amazonaws.com
java.net.UnknownHostException: steve-eu2.s3.amazonaws.com: unknown error
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:262)
at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:161)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:328)
at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:612)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:884)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:728)
at
com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at
com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1107)
at
com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1070)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:304)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:283)
at
org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:59)
at
org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase.setUp(S3AScaleTestBase.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
> S3A to support custom retry policies; maybe failfast on unknown host
>
>
> Key: HADOOP-13205
> URL: https://issues.apache.org/jira/browse/HADOOP-13205
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> Noticed today that when connections are down, S3A retries on
> UnknownHostExceptions logging noisily in the process.
> # it should be possible to define or customize retry policies for an FS
> instance (fail fast, exponential backoff, etc)
> # we may want to explicitly have a fail-fast-if-offline retry policy,
> catching the common connectivity ones.
> Testing will be fun here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13204) Über-jira: S3a phase III: scale and tuning
Steve Loughran created HADOOP-13204: --- Summary: Über-jira: S3a phase III: scale and tuning Key: HADOOP-13204 URL: https://issues.apache.org/jira/browse/HADOOP-13204 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.8.0 Reporter: Steve Loughran Assignee: Steve Loughran S3A Phase III work; post 2.8. Areas could include * customisation * performance enhancement * management and metrics * error/failure handling And of course any bugs which surface -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12727) Minor cleanups needed for CMake 3.X
[
https://issues.apache.org/jira/browse/HADOOP-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302092#comment-15302092
]
Allen Wittenauer commented on HADOOP-12727:
---
YARN-5121 needed a rebase. Let's try this again on OS X.
> Minor cleanups needed for CMake 3.X
> ---
>
> Key: HADOOP-12727
> URL: https://issues.apache.org/jira/browse/HADOOP-12727
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: native
>Affects Versions: 2.7.1
>Reporter: Alan Burlison
>Assignee: Alan Burlison
>Priority: Minor
> Attachments: HADOOP-12727.001.patch
>
>
> On switching from CMake 2.8.6 to 3.3.2 a couple of minor issues popped up:
> \\
> \\
> * There's a syntax error in
> {{hadoop-common-project/hadoop-common/src/CMakeLists.txt}} that generates a
> warning in 3.X
> * {{CMAKE_SHARED_LINKER_FLAGS}} is being incorrectly set in
> {{hadoop-common-project/hadoop-common/HadoopCommon.cmake}} - despite the name
> it contains the flags passed to {{ar}} not to the linker. 2.8.6 ignores the
> incorrect flags, 3.3.2 doesn't and building static libraries fails as a
> result. See http://public.kitware.com/pipermail/cmake/2016-January/062447.html
> Patch to follow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13202) Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might be changed
[ https://issues.apache.org/jira/browse/HADOOP-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki reassigned HADOOP-13202: --- Assignee: Kai Sasaki > Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might > be changed > > > Key: HADOOP-13202 > URL: https://issues.apache.org/jira/browse/HADOOP-13202 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: zhengbing li >Assignee: Kai Sasaki > Attachments: HADOOP-13202.01.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Current implementation: > return (vectorSize + 7) / 8; > when vectorSize is 2147483647（the max value of Int), error > :"java.lang.NegativeArraySizeException" will report > the implementation might be changed > return (int)(((long)vectorSize + 7) / 8); -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13202) Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might be changed
[ https://issues.apache.org/jira/browse/HADOOP-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HADOOP-13202: Attachment: HADOOP-13202.01.patch > Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might > be changed > > > Key: HADOOP-13202 > URL: https://issues.apache.org/jira/browse/HADOOP-13202 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: zhengbing li >Assignee: Kai Sasaki > Attachments: HADOOP-13202.01.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Current implementation: > return (vectorSize + 7) / 8; > when vectorSize is 2147483647（the max value of Int), error > :"java.lang.NegativeArraySizeException" will report > the implementation might be changed > return (int)(((long)vectorSize + 7) / 8); -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13202) Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might be changed
[ https://issues.apache.org/jira/browse/HADOOP-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HADOOP-13202: Fix Version/s: 3.0.0-alpha1 > Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might > be changed > > > Key: HADOOP-13202 > URL: https://issues.apache.org/jira/browse/HADOOP-13202 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: zhengbing li >Assignee: Kai Sasaki > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13202.01.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Current implementation: > return (vectorSize + 7) / 8; > when vectorSize is 2147483647（the max value of Int), error > :"java.lang.NegativeArraySizeException" will report > the implementation might be changed > return (int)(((long)vectorSize + 7) / 8); -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13202) Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might be changed
[ https://issues.apache.org/jira/browse/HADOOP-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HADOOP-13202: Affects Version/s: (was: 3.0.0-alpha1) > Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might > be changed > > > Key: HADOOP-13202 > URL: https://issues.apache.org/jira/browse/HADOOP-13202 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: zhengbing li >Assignee: Kai Sasaki > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13202.01.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Current implementation: > return (vectorSize + 7) / 8; > when vectorSize is 2147483647（the max value of Int), error > :"java.lang.NegativeArraySizeException" will report > the implementation might be changed > return (int)(((long)vectorSize + 7) / 8); -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13202) Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might be changed
[ https://issues.apache.org/jira/browse/HADOOP-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HADOOP-13202: Affects Version/s: 3.0.0-alpha1 > Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might > be changed > > > Key: HADOOP-13202 > URL: https://issues.apache.org/jira/browse/HADOOP-13202 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: zhengbing li >Assignee: Kai Sasaki > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-13202.01.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Current implementation: > return (vectorSize + 7) / 8; > when vectorSize is 2147483647（the max value of Int), error > :"java.lang.NegativeArraySizeException" will report > the implementation might be changed > return (int)(((long)vectorSize + 7) / 8); -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13202) Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might be changed
[ https://issues.apache.org/jira/browse/HADOOP-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Sasaki updated HADOOP-13202: Status: Patch Available (was: Open) > Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might > be changed > > > Key: HADOOP-13202 > URL: https://issues.apache.org/jira/browse/HADOOP-13202 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.2 >Reporter: zhengbing li >Assignee: Kai Sasaki > Attachments: HADOOP-13202.01.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > Current implementation: > return (vectorSize + 7) / 8; > when vectorSize is 2147483647（the max value of Int), error > :"java.lang.NegativeArraySizeException" will report > the implementation might be changed > return (int)(((long)vectorSize + 7) / 8); -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13202) Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might be changed
[
https://issues.apache.org/jira/browse/HADOOP-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15302059#comment-15302059
]
Kai Sasaki commented on HADOOP-13202:
-
Though {{Math.toIntExact}} is a JDK8 API, HADOOP-11858 set minimum version
JDK8. So it seems okay from Hadoop 3 to use.
> Implementation of getNBytes in org.apache.hadoop.util.bloom.BloomFilter might
> be changed
>
>
> Key: HADOOP-13202
> URL: https://issues.apache.org/jira/browse/HADOOP-13202
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: zhengbing li
>Assignee: Kai Sasaki
> Attachments: HADOOP-13202.01.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Current implementation:
> return (vectorSize + 7) / 8;
> when vectorSize is 2147483647（the max value of Int), error
> :"java.lang.NegativeArraySizeException" will report
> the implementation might be changed
> return (int)(((long)vectorSize + 7) / 8);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12782) Faster LDAP group name resolution with ActiveDirectory
[ https://issues.apache.org/jira/browse/HADOOP-12782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15301874#comment-15301874 ] Kai Zheng commented on HADOOP-12782: Committed to branch-2. > Faster LDAP group name resolution with ActiveDirectory > -- > > Key: HADOOP-12782 > URL: https://issues.apache.org/jira/browse/HADOOP-12782 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Fix For: 3.0.0-alpha1 > > Attachments: HADOOP-12782.001.patch, HADOOP-12782.002.patch, > HADOOP-12782.003.patch, HADOOP-12782.004.patch, HADOOP-12782.005.patch, > HADOOP-12782.006.patch, HADOOP-12782.007.patch, HADOOP-12782.008.patch, > HADOOP-12782.009.patch, HADOOP-12782.branch-2.010.patch > > > The typical LDAP group name resolution works well under typical scenarios. > However, we have seen cases where a user is mapped to many groups (in an > extreme case, a user is mapped to more than 100 groups). The way it's being > implemented now makes this case super slow resolving groups from > ActiveDirectory. > The current LDAP group resolution implementation sends two queries to a > ActiveDirectory server. The first query returns a user object, which contains > DN (distinguished name). The second query looks for groups where the user DN > is a member. If a user is mapped to many groups, the second query returns all > group objects associated with the user, and is thus very slow. > After studying a user object in ActiveDirectory, I found a user object > actually contains a "memberOf" field, which is the DN of all group objects > where the user belongs to. Assuming that an organization has no recursive > group relation (that is, a user A is a member of group G1, and group G1 is a > member of group G2), we can use this properties to avoid the second query, > which can potentially run very slow. > I propose that we add a configuration to only enable this feature for users > who want to reduce group resolution time and who does not have recursive > groups, so that existing behavior will not be broken. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12893) Verify LICENSE.txt and NOTICE.txt
[
https://issues.apache.org/jira/browse/HADOOP-12893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-12893:
---
Attachment: HADOOP-12893.006.patch
Patch 6 addresses the comments above:
- Only the 'bundled'==Y deps are included. (I didn't rename the column in the
doc, but please feel free to do so)
- jdiff is hence not included, because it's not bundled.
(Verified by {{find ./target/ -name "*jar"|xargs -n1 basename|sort |uniq|grep
-v 3.0.0-alpha}} )
Let me be thorough to hopefully help review:
- I put license text into the {{Licenses}} tab in the doc, and update the
generate script to read that
- Still had to do the manual 80-char-wrap, because I don't know how to preserve
line change after saving that as a tsv, or how to decently break the content
with 80-char and preserve paragraphs.
- The license texts are all copied from the link after it.
- For the W3C license, the
[website|https://www.w3.org/Consortium/Legal/2002/copyright-software-20021231]
lists the license and disclaimers separately. Only the 'License' section is
included. Please let me know if this is not right.
If anyone could try the patch or would like to make any changes, please do. :)
> Verify LICENSE.txt and NOTICE.txt
> -
>
> Key: HADOOP-12893
> URL: https://issues.apache.org/jira/browse/HADOOP-12893
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Xiao Chen
>Priority: Blocker
> Attachments: HADOOP-12893.002.patch, HADOOP-12893.003.patch,
> HADOOP-12893.004.patch, HADOOP-12893.005.patch, HADOOP-12893.006.patch,
> HADOOP-12893.01.patch
>
>
> We have many bundled dependencies in both the source and the binary artifacts
> that are not in LICENSE.txt and NOTICE.txt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
