[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361364#comment-17361364
]
yikf commented on HADOOP-17758:
---
Oh, Thanks, you are right, My bad, If something is added to the queue,
ReferenceQueue.enqueue will be called, i.e. lock.notifyAll be called, That will
not wait forerver.
I will revert.
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17753) Keep restrict-imports-enforcer-rule for Guava Lists in hadoop-main pom
[ https://issues.apache.org/jira/browse/HADOOP-17753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HADOOP-17753. --- Fix Version/s: 3.4.0 Resolution: Fixed > Keep restrict-imports-enforcer-rule for Guava Lists in hadoop-main pom > -- > > Key: HADOOP-17753 > URL: https://issues.apache.org/jira/browse/HADOOP-17753 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17753) Keep restrict-imports-enforcer-rule for Guava Lists in hadoop-main pom
[
https://issues.apache.org/jira/browse/HADOOP-17753?focusedWorklogId=609991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609991
]
ASF GitHub Bot logged work on HADOOP-17753:
---
Author: ASF GitHub Bot
Created on: 10/Jun/21 23:24
Start Date: 10/Jun/21 23:24
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3087:
URL: https://github.com/apache/hadoop/pull/3087#issuecomment-859150552
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 0s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 17s | | trunk passed |
| +1 :green_heart: | compile | 20m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 12s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 24m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 7m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 27s | |
branch/hadoop-cloud-storage-project/hadoop-cloud-storage no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 44m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 36m 23s | | the patch passed |
| +1 :green_heart: | compile | 20m 38s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 38s | | the patch passed |
| +1 :green_heart: | compile | 18m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 20m 18s | | the patch passed |
| +1 :green_heart: | xml | 0m 25s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 7m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 7m 41s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
hadoop-cloud-storage-project/hadoop-cloud-storage has no data from spotbugs |
| +1 :green_heart: | shadedclient | 44m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 771m 14s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 42s | | The patch does not
generate ASF License warnings. |
| | | 1174m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3087/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3087 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 054b60f681cc 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 89e961b20d08edec96845dda43f5bc07c4813019 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #3087: HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom
hadoop-yetus commented on pull request #3087:
URL: https://github.com/apache/hadoop/pull/3087#issuecomment-859150552
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 0s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 17s | | trunk passed |
| +1 :green_heart: | compile | 20m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 18m 12s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 24m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 7m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 7m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 27s | |
branch/hadoop-cloud-storage-project/hadoop-cloud-storage no spotbugs output
file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 44m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 36m 23s | | the patch passed |
| +1 :green_heart: | compile | 20m 38s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 20m 38s | | the patch passed |
| +1 :green_heart: | compile | 18m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 18m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 20m 18s | | the patch passed |
| +1 :green_heart: | xml | 0m 25s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 7m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 7m 41s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 28s | |
hadoop-cloud-storage-project/hadoop-cloud-storage has no data from spotbugs |
| +1 :green_heart: | shadedclient | 44m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 771m 14s | | root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 42s | | The patch does not
generate ASF License warnings. |
| | | 1174m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3087/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3087 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux 054b60f681cc 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 89e961b20d08edec96845dda43f5bc07c4813019 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3087/5/testReport/ |
| Max. process+thread count | 3662 (vs. ulimit of 5500) |
| modules | C: hadoop-maven-plugins hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-registry hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs
[GitHub] [hadoop] goiri commented on a change in pull request #3086: HDFS-16039. RBF: Some indicators of RBFMetrics count inaccurately
goiri commented on a change in pull request #3086:
URL: https://github.com/apache/hadoop/pull/3086#discussion_r649553648
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/RBFMetrics.java
##
@@ -165,11 +172,46 @@ public RBFMetrics(Router router) throws IOException {
// Initialize the cache for the DN reports
Configuration conf = router.getConfig();
-this.timeOut = conf.getTimeDuration(RBFConfigKeys.DN_REPORT_TIME_OUT,
-RBFConfigKeys.DN_REPORT_TIME_OUT_MS_DEFAULT, TimeUnit.MILLISECONDS);
this.topTokenRealOwners = conf.getInt(
RBFConfigKeys.DFS_ROUTER_METRICS_TOP_NUM_TOKEN_OWNERS_KEY,
RBFConfigKeys.DFS_ROUTER_METRICS_TOP_NUM_TOKEN_OWNERS_KEY_DEFAULT);
+// Initialize the cache for the DN reports
+this.dnReportTimeOut = conf.getTimeDuration(
+RBFConfigKeys.DN_REPORT_TIME_OUT,
+RBFConfigKeys.DN_REPORT_TIME_OUT_MS_DEFAULT, TimeUnit.MILLISECONDS);
+long dnCacheExpire = conf.getTimeDuration(
+RBFConfigKeys.DN_REPORT_CACHE_EXPIRE,
+RBFConfigKeys.DN_REPORT_CACHE_EXPIRE_MS_DEFAULT,
TimeUnit.MILLISECONDS);
+this.dnCache = CacheBuilder.newBuilder()
Review comment:
getNodeUsage() already uses it, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #3086: HDFS-16039. RBF: Some indicators of RBFMetrics count inaccurately
goiri commented on a change in pull request #3086:
URL: https://github.com/apache/hadoop/pull/3086#discussion_r649552979
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/RBFMetrics.java
##
@@ -165,11 +172,46 @@ public RBFMetrics(Router router) throws IOException {
// Initialize the cache for the DN reports
Configuration conf = router.getConfig();
-this.timeOut = conf.getTimeDuration(RBFConfigKeys.DN_REPORT_TIME_OUT,
-RBFConfigKeys.DN_REPORT_TIME_OUT_MS_DEFAULT, TimeUnit.MILLISECONDS);
this.topTokenRealOwners = conf.getInt(
RBFConfigKeys.DFS_ROUTER_METRICS_TOP_NUM_TOKEN_OWNERS_KEY,
RBFConfigKeys.DFS_ROUTER_METRICS_TOP_NUM_TOKEN_OWNERS_KEY_DEFAULT);
+// Initialize the cache for the DN reports
+this.dnReportTimeOut = conf.getTimeDuration(
+RBFConfigKeys.DN_REPORT_TIME_OUT,
+RBFConfigKeys.DN_REPORT_TIME_OUT_MS_DEFAULT, TimeUnit.MILLISECONDS);
+long dnCacheExpire = conf.getTimeDuration(
+RBFConfigKeys.DN_REPORT_CACHE_EXPIRE,
+RBFConfigKeys.DN_REPORT_CACHE_EXPIRE_MS_DEFAULT,
TimeUnit.MILLISECONDS);
+this.dnCache = CacheBuilder.newBuilder()
Review comment:
RouterRpcServer has a similar cache, can we use that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2349: MAPREDUCE-7282. Move away from V2 commit algorithm
steveloughran commented on pull request #2349: URL: https://github.com/apache/hadoop/pull/2349#issuecomment-859061589 Once the manifest committer #2971 is in, the abfs and gcs stores will get something faster than v1 but with its task failure semantics * all treewalking for task attempt listing will be in task commit * no dir renames in task commit, just saving of the manifest * job commit: parallel load of manifests, merge of list of directories to create, parallel set of mkdirs and then the parallelized renames. In this world, we can just leave people using v1/v2 alone, and for (spark) jobs in azure and google cloud say "use the manifest committer". Which means I can just close this as a wontfix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-15566) Support OpenTelemetry
[ https://issues.apache.org/jira/browse/HADOOP-15566?focusedWorklogId=609928=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609928 ] ASF GitHub Bot logged work on HADOOP-15566: --- Author: ASF GitHub Bot Created on: 10/Jun/21 20:50 Start Date: 10/Jun/21 20:50 Worklog Time Spent: 10m Work Description: steveloughran commented on pull request #2816: URL: https://github.com/apache/hadoop/pull/2816#issuecomment-85905 ..yes. I need to catch up with my reviewing. I've not forgotten this, and I'm not giving it any less attention than all those other PRs I need to review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609928) Time Spent: 2.5h (was: 2h 20m) > Support OpenTelemetry > - > > Key: HADOOP-15566 > URL: https://issues.apache.org/jira/browse/HADOOP-15566 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics, tracing >Affects Versions: 3.1.0 >Reporter: Todd Lipcon >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available, security > Attachments: HADOOP-15566.000.WIP.patch, OpenTelemetry Support Scope > Doc v2.pdf, OpenTracing Support Scope Doc.pdf, Screen Shot 2018-06-29 at > 11.59.16 AM.png, ss-trace-s3a.png > > Time Spent: 2.5h > Remaining Estimate: 0h > > The HTrace incubator project has voted to retire itself and won't be making > further releases. The Hadoop project currently has various hooks with HTrace. > It seems in some cases (eg HDFS-13702) these hooks have had measurable > performance overhead. Given these two factors, I think we should consider > removing the HTrace integration. If there is someone willing to do the work, > replacing it with OpenTracing might be a better choice since there is an > active community. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #2816: HADOOP-15566 initial changes for opentelemetry - WIP
steveloughran commented on pull request #2816: URL: https://github.com/apache/hadoop/pull/2816#issuecomment-85905 ..yes. I need to catch up with my reviewing. I've not forgotten this, and I'm not giving it any less attention than all those other PRs I need to review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3095: HDFS-16061. DFTestUtil.waitReplication can produce false positives
hadoop-yetus commented on pull request #3095:
URL: https://github.com/apache/hadoop/pull/3095#issuecomment-859025279
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 18s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 86 unchanged - 2
fixed = 86 total (was 88) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 252m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3095/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 336m 55s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestFileCorruption |
| | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3095/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3095 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a756d5c27042 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / da16c59f102f81ff2be92b85f10f487fb7b2541c |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3095/1/testReport/ |
| Max. process+thread count | 2995 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3095/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This
[GitHub] [hadoop] amahussein opened a new pull request #3097: HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently
amahussein opened a new pull request #3097: URL: https://github.com/apache/hadoop/pull/3097 HDFS-15671 TestBalancerRPCDelay#testBalancerRPCDelayQpsDefault fails on Trunk Some blocks cannot be replicated on the existing nodes which causes the unit test to fail during the creation of the file and waiting for the replications. Setting `dfs.namenode.redundancy.considerLoad` to false. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3065: HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
hadoop-yetus commented on pull request #3065:
URL: https://github.com/apache/hadoop/pull/3065#issuecomment-858964433
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 22m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 18 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | cc | 1m 18s | | the patch passed |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | cc | 1m 8s | | the patch passed |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3065/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 1337 unchanged
- 13 fixed = 1338 total (was 1350) |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 357m 17s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3065/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 0m 37s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3065/5/artifact/out/results-asflicense.txt)
| The patch generated 2 ASF License warnings. |
| | | 471m 35s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3065/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3065 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat
xml |
| uname | Linux 9cf7a549a107 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Comment Edited] (HADOOP-11461) Namenode stdout log contains IllegalAccessException
[
https://issues.apache.org/jira/browse/HADOOP-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361154#comment-17361154
]
Vicente Oyanedel edited comment on HADOOP-11461 at 6/10/21, 6:26 PM:
-
Well, that's sort of relieving. Actually, we're having this issue on a
production AWS EMR cluster that can be hardly updated to Hadoop 3.
The solution of adding the exception kind of looks simpler and might give us
some time until dev-ops decide to update the EMR, Hadoop, and Spark.
The issue I'm having right now is that due AWS limitations, modifying
/etc/alternatives/jre/lib/logging.properties file doesn't affect the whole
cluster and probably the change will be wiped out at any time when the cluster
it's rebuilt.
So my question is if there's any other configuration I can do to avoid this
stack trace to fill up the disks? Probably a solution that involves Hadoop/Yarn
configurations, and not configuring a global exception at JRE level. At first
hand, I'm limited to [this set of
configurations|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html#emr-5310-class].
I'm thinking something like configuring Hadoop's log4j.properties might work,
but there are so many levels of logs that it's really hard to be sure.
Does anybody know if log4j it's involved in this issue?
I'll keep you guys informed of any findings I make, as I'm pretty sure this
issue may be happening to other EMR users.
was (Author: vichoko):
Well, that's sort of relieving. Actually, we're having this issue on a
production AWS EMR cluster that can be hardly updated to Hadoop 3.
The solution of adding the exception kind of looks simpler and might give us
some time until dev-ops decide to update the EMR, Hadoop, and Spark.
The issue I'm having right now is that due AWS limitations, modifying
/etc/alternatives/jre/lib/logging.properties file doesn't affect the whole
cluster and probably the change will be wiped out at any time when the cluster
it's rebuilt.
So my question is if there's any other configuration I can do to avoid this
stack trace to fill up the disks? Probably a solution that involves Hadoop/Yarn
configurations, and not configuring a global exception at JRE level. At first
hand, I'm limited to [this set of
configurations|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html#emr-5310-class].
I'm thinking something like configuring Hadoop's log4j.properties might work,
but there are so many levels of logs that it's really hard to be sure.
Does anybody know if log4j it's involved in this issue?
I'll keep you guys informed of any findings I make, as I'm pretty sure this
issue may be happening to other EMR users.
> Namenode stdout log contains IllegalAccessException
> ---
>
> Key: HADOOP-11461
> URL: https://issues.apache.org/jira/browse/HADOOP-11461
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.7.0
>Reporter: Mohammad Islam
>Assignee: Mohammad Islam
>Priority: Major
>
> We frequently see the following exception in namenode out log file.
> {noformat}
> Nov 19, 2014 8:11:19 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
> Nov 19, 2014 8:11:19 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> resolve
> SEVERE: null
> java.lang.IllegalAccessException: Class
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can
> not access a member of class javax.ws.rs.co
> re.Response with modifiers "protected"
> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
> at java.lang.Class.newInstance(Class.java:368)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
> at
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
> at
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
> at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:120)
> at
> com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
>
[jira] [Comment Edited] (HADOOP-11461) Namenode stdout log contains IllegalAccessException
[
https://issues.apache.org/jira/browse/HADOOP-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361154#comment-17361154
]
Vicente Oyanedel edited comment on HADOOP-11461 at 6/10/21, 6:26 PM:
-
Well, that's sort of relieving. Actually, we're having this issue on a
production AWS EMR cluster that can be hardly updated to Hadoop 3.
The solution of adding the exception kind of looks simpler and might give us
some time until dev-ops decide to update the EMR, Hadoop, and Spark.
The issue I'm having right now is that due AWS limitations, modifying
/etc/alternatives/jre/lib/logging.properties file doesn't affect the whole
cluster and probably the change will be wiped out at any time when the cluster
it's rebuilt.
So my question is if there's any other configuration I can do to avoid this
stack trace to fill up the disks? Probably a solution that involves Hadoop/Yarn
configurations, and not configuring a global exception at JRE level. At first
hand, I'm limited to [this set of
configurations|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html#emr-5310-class].
I'm thinking something like configuring Hadoop's log4j.properties might work,
but there are so many levels of logs that it's really hard to be sure.
Does anybody know if log4j it's involved in this issue?
I'll keep you guys informed of any findings I make, as I'm pretty sure this
issue may be happening to other EMR users.
was (Author: vichoko):
Well, that's sort of relieving. Actually, we're having this issue on a
production AWS EMR cluster that can be hardly updated to Hadoop 3.
The solution of adding the exception kind of looks simpler and might give us
some time until dev-ops decide to update the EMR, Hadoop, and Spark.
The issue I'm having right now is that due AWS limitations, modifying
/etc/alternatives/jre/lib/logging.properties file doesn't affect the whole
cluster and probably the change will be wiped out at any time when the cluster
it's rebuilt.
So my question is if there's any other configuration I can do to avoid this
stack trace to fill up the disks? Probably a solution that involves Hadoop/Yarn
configurations, and not configuring a global exception at JRE level. At first
hand, I'm limited to [this set of
configurations|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html#emr-5310-class].
I'm thinking something like configuring Hadoop's log4j.properties might work,
but there are so many levels of logs that it's really hard to be sure.
Does anybody know if log4j it's involved in this issue?
> Namenode stdout log contains IllegalAccessException
> ---
>
> Key: HADOOP-11461
> URL: https://issues.apache.org/jira/browse/HADOOP-11461
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.7.0
>Reporter: Mohammad Islam
>Assignee: Mohammad Islam
>Priority: Major
>
> We frequently see the following exception in namenode out log file.
> {noformat}
> Nov 19, 2014 8:11:19 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
> Nov 19, 2014 8:11:19 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> resolve
> SEVERE: null
> java.lang.IllegalAccessException: Class
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can
> not access a member of class javax.ws.rs.co
> re.Response with modifiers "protected"
> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
> at java.lang.Class.newInstance(Class.java:368)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
> at
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
> at
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
> at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:120)
> at
> com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
>
[jira] [Comment Edited] (HADOOP-11461) Namenode stdout log contains IllegalAccessException
[
https://issues.apache.org/jira/browse/HADOOP-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361154#comment-17361154
]
Vicente Oyanedel edited comment on HADOOP-11461 at 6/10/21, 6:25 PM:
-
Well, that's sort of relieving. Actually, we're having this issue on a
production AWS EMR cluster that can be hardly updated to Hadoop 3.
The solution of adding the exception kind of looks simpler and might give us
some time until dev-ops decide to update the EMR, Hadoop, and Spark.
The issue I'm having right now is that due AWS limitations, modifying
/etc/alternatives/jre/lib/logging.properties file doesn't affect the whole
cluster and probably the change will be wiped out at any time when the cluster
it's rebuilt.
So my question is if there's any other configuration I can do to avoid this
stack trace to fill up the disks? Probably a solution that involves Hadoop/Yarn
configurations, and not configuring a global exception at JRE level. At first
hand, I'm limited to [this set of
configurations|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html#emr-5310-class].
I'm thinking something like configuring Hadoop's log4j.properties might work,
but there are so many levels of logs that it's really hard to be sure.
Does anybody know if log4j it's involved in this issue?
was (Author: vichoko):
Well that's sort of relieving. Actually we're having this issue on a production
AWS EMR cluster that can be hardly updated to Hadoop 3.
The solution of adding the exception kind of looks simpler and might give us
some time until dev-ops decide to update the EMR, Hadoop and Spark.
The issue I'm having right now is that due AWS limitations, modifying
/etc/alternatives/jre/lib/logging.properties file doesn't affect the whole
cluster and probably the change will be wiped out at any time when the cluster
it's rebuilt.
So my question is if there's any other configuration I can do to avoid this
stack trace to fill up the disks? Probably a solution that involves Hadoop/Yarn
configurations, and not configuring a global exception at JRE level.
I'm thinking something like configuring Hadoop's log4j.properties might work,
but there's so many levels of logs that it's really hard to be sure.
Anybody knows if log4j it's involved in this issue?
> Namenode stdout log contains IllegalAccessException
> ---
>
> Key: HADOOP-11461
> URL: https://issues.apache.org/jira/browse/HADOOP-11461
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.7.0
>Reporter: Mohammad Islam
>Assignee: Mohammad Islam
>Priority: Major
>
> We frequently see the following exception in namenode out log file.
> {noformat}
> Nov 19, 2014 8:11:19 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
> Nov 19, 2014 8:11:19 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> resolve
> SEVERE: null
> java.lang.IllegalAccessException: Class
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can
> not access a member of class javax.ws.rs.co
> re.Response with modifiers "protected"
> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
> at java.lang.Class.newInstance(Class.java:368)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
> at
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
> at
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
> at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:120)
> at
> com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
[jira] [Commented] (HADOOP-11461) Namenode stdout log contains IllegalAccessException
[
https://issues.apache.org/jira/browse/HADOOP-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361154#comment-17361154
]
Vicente Oyanedel commented on HADOOP-11461:
---
Well that's sort of relieving. Actually we're having this issue on a production
AWS EMR cluster that can be hardly updated to Hadoop 3.
The solution of adding the exception kind of looks simpler and might give us
some time until dev-ops decide to update the EMR, Hadoop and Spark.
The issue I'm having right now is that due AWS limitations, modifying
/etc/alternatives/jre/lib/logging.properties file doesn't affect the whole
cluster and probably the change will be wiped out at any time when the cluster
it's rebuilt.
So my question is if there's any other configuration I can do to avoid this
stack trace to fill up the disks? Probably a solution that involves Hadoop/Yarn
configurations, and not configuring a global exception at JRE level.
I'm thinking something like configuring Hadoop's log4j.properties might work,
but there's so many levels of logs that it's really hard to be sure.
Anybody knows if log4j it's involved in this issue?
> Namenode stdout log contains IllegalAccessException
> ---
>
> Key: HADOOP-11461
> URL: https://issues.apache.org/jira/browse/HADOOP-11461
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.7.0
>Reporter: Mohammad Islam
>Assignee: Mohammad Islam
>Priority: Major
>
> We frequently see the following exception in namenode out log file.
> {noformat}
> Nov 19, 2014 8:11:19 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator
> attachTypes
> INFO: Couldn't find JAX-B element for class javax.ws.rs.core.Response
> Nov 19, 2014 8:11:19 PM
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8
> resolve
> SEVERE: null
> java.lang.IllegalAccessException: Class
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8 can
> not access a member of class javax.ws.rs.co
> re.Response with modifiers "protected"
> at sun.reflect.Reflection.ensureMemberAccess(Reflection.java:109)
> at java.lang.Class.newInstance(Class.java:368)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator$8.resolve(WadlGeneratorJAXBGrammarGenerator.java:467)
> at
> com.sun.jersey.server.wadl.WadlGenerator$ExternalGrammarDefinition.resolve(WadlGenerator.java:181)
> at
> com.sun.jersey.server.wadl.ApplicationDescription.resolve(ApplicationDescription.java:81)
> at
> com.sun.jersey.server.wadl.generators.WadlGeneratorJAXBGrammarGenerator.attachTypes(WadlGeneratorJAXBGrammarGenerator.java:518)
> at com.sun.jersey.server.wadl.WadlBuilder.generate(WadlBuilder.java:124)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:104)
> at
> com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.getApplication(WadlApplicationContextImpl.java:120)
> at
> com.sun.jersey.server.impl.wadl.WadlMethodFactory$WadlOptionsMethodDispatcher.dispatch(WadlMethodFactory.java:98)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
> at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
>
[GitHub] [hadoop] smengcl commented on pull request #3078: HDFS-16055. Quota is not preserved in snapshot INode
smengcl commented on pull request #3078: URL: https://github.com/apache/hadoop/pull/3078#issuecomment-858869425 Thanks @szetszwo for the +1. There are still a few UTs failing because themselves needs to be fixed after this patch (because modified current dir entry no longer exists in some test cases and the result is often hard-coded). I am fixing them one-by-one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17759) Remove Hadoop 3.1.4 from the download page
[ https://issues.apache.org/jira/browse/HADOOP-17759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17759: --- Issue Type: Task (was: Bug) > Remove Hadoop 3.1.4 from the download page > -- > > Key: HADOOP-17759 > URL: https://issues.apache.org/jira/browse/HADOOP-17759 > Project: Hadoop Common > Issue Type: Task > Components: documentation >Reporter: Akira Ajisaka >Priority: Major > > Since Hadoop 3.1.x is EoL, 3.1.4 should be removed from > https://hadoop.apache.org/releases.html. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17759) Remove Hadoop 3.1.4 from the download page
[ https://issues.apache.org/jira/browse/HADOOP-17759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361132#comment-17361132 ] Akira Ajisaka commented on HADOOP-17759: After removing 3.1.4 from the download page, we have to clean up the release artifacts [https://dist.apache.org/repos/dist/release/hadoop/common/hadoop-3.1.4/] > Remove Hadoop 3.1.4 from the download page > -- > > Key: HADOOP-17759 > URL: https://issues.apache.org/jira/browse/HADOOP-17759 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Reporter: Akira Ajisaka >Priority: Major > > Since Hadoop 3.1.x is EoL, 3.1.4 should be removed from > https://hadoop.apache.org/releases.html. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev resolved HADOOP-16960. Resolution: Won't Fix > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > Attachments: HADOOP-16960.001.patch, HADOOP-16960.002.patch, > HADOOP-16960.003.patch, HADOOP-16960.004.patch > > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17759) Remove Hadoop 3.1.4 from the download page
Akira Ajisaka created HADOOP-17759: -- Summary: Remove Hadoop 3.1.4 from the download page Key: HADOOP-17759 URL: https://issues.apache.org/jira/browse/HADOOP-17759 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Akira Ajisaka Since Hadoop 3.1.x is EoL, 3.1.4 should be removed from https://hadoop.apache.org/releases.html. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16960) Enable setting clientId and tenantId for ADL MsiTokenProvider
[ https://issues.apache.org/jira/browse/HADOOP-16960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vasil Vasilev updated HADOOP-16960: --- Status: Open (was: Patch Available) > Enable setting clientId and tenantId for ADL MsiTokenProvider > - > > Key: HADOOP-16960 > URL: https://issues.apache.org/jira/browse/HADOOP-16960 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/adl >Reporter: Vasil Vasilev >Priority: Major > Attachments: HADOOP-16960.001.patch, HADOOP-16960.002.patch, > HADOOP-16960.003.patch, HADOOP-16960.004.patch > > > AdlFileSystem is [currently > using|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-azure-datalake/src/main/java/org/apache/hadoop/fs/adl/AdlFileSystem.java#L318] > a [deprecated constructor for > MsiTokenProvider|https://github.com/Azure/azure-data-lake-store-java/blob/2.3.6/src/main/java/com/microsoft/azure/datalake/store/oauth2/MsiTokenProvider.java#L42], > only passing a port setting which is not used anymore. > The purpose of this improvement is to update AdlFileSystem to pass optional > clientId and tenantId to MsiTokenProvider. This can ultimately enable MSI > authentication on VMs with attached multiple MSIs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17728) Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp
[
https://issues.apache.org/jira/browse/HADOOP-17728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated HADOOP-17728:
---
Fix Version/s: (was: 3.3.1)
3.3.2
Updated the fix version because this commit is not in branch-3.3.1.
FYI: The commit message is "HDFS-16033 Fix issue of the
StatisticsDataReferenceCleaner cleanUp (#3042)"
> Deadlock in FileSystem StatisticsDataReferenceCleaner cleanUp
> -
>
> Key: HADOOP-17728
> URL: https://issues.apache.org/jira/browse/HADOOP-17728
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.2.1
>Reporter: yikf
>Assignee: yikf
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Cleaner thread will be blocked if we remove reference from ReferenceQueue
> unless the `queue.enqueue` called.
>
> As shown below, We call ReferenceQueue.remove() now while cleanUp, Call
> chain as follow：
> *StatisticsDataReferenceCleaner#queue.remove() ->
> ReferenceQueue.remove(0) -> lock.wait(0)*
> But, lock.notifyAll is called when queue.enqueue only, so Cleaner thread
> will be blocked.
>
> ThreadDump:
> {code:java}
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x7f7afc088800
> nid=0x2119 in Object.wait() [0x7f7b0023]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0xc00c2f58> (a java.lang.ref.Reference$Lock)
> at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361075#comment-17361075
]
Jim Brennan commented on HADOOP-17758:
--
{quote}
We need a timeout to wake me up.
{quote}
I still don't understand why you need to wake up if the queue is still empty.
There is nothing to clean up until something is added to the queue.
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361068#comment-17361068
]
yikf commented on HADOOP-17758:
---
Cleanup is a persistent process. If the queue is empty, the current thread will
wait forever It's not a deadlock, though and not notify, so cleanUp will not
work.
We need a timeout to wake me up.
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361053#comment-17361053
]
Jim Brennan commented on HADOOP-17758:
--
I'm sorry, but am I missing something? What is the point of using a timeout,
if nothing has been added to the queue? We'll just loop around again and wait
on the same lock until something does get queued.
One thread waiting on a lock is not a deadlock. Is this causing some other
thread to block as well?
My inclination would be to revert [HADOOP-17728].
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?focusedWorklogId=609783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609783
]
ASF GitHub Bot logged work on HADOOP-17758:
---
Author: ASF GitHub Bot
Created on: 10/Jun/21 15:53
Start Date: 10/Jun/21 15:53
Worklog Time Spent: 10m
Work Description: yikf commented on pull request #3096:
URL: https://github.com/apache/hadoop/pull/3096#issuecomment-858740406
Code conflicting, Will reopen new PR later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609783)
Time Spent: 0.5h (was: 20m)
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] yikf commented on pull request #3096: [HADOOP-17758][HDFS][FOLLOWUP] NPE and excessive warnings after HADOOP-17728
yikf commented on pull request #3096: URL: https://github.com/apache/hadoop/pull/3096#issuecomment-858740406 Code conflicting, Will reopen new PR later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?focusedWorklogId=609779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609779
]
ASF GitHub Bot logged work on HADOOP-17758:
---
Author: ASF GitHub Bot
Created on: 10/Jun/21 15:42
Start Date: 10/Jun/21 15:42
Worklog Time Spent: 10m
Work Description: yikf closed pull request #3096:
URL: https://github.com/apache/hadoop/pull/3096
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609779)
Time Spent: 20m (was: 10m)
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] yikf closed pull request #3096: [HADOOP-17758][HDFS][FOLLOWUP] NPE and excessive warnings after HADOOP-17728
yikf closed pull request #3096: URL: https://github.com/apache/hadoop/pull/3096 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-17758:
Labels: pull-request-available (was: )
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?focusedWorklogId=609777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609777
]
ASF GitHub Bot logged work on HADOOP-17758:
---
Author: ASF GitHub Bot
Created on: 10/Jun/21 15:41
Start Date: 10/Jun/21 15:41
Worklog Time Spent: 10m
Work Description: yikf opened a new pull request #3096:
URL: https://github.com/apache/hadoop/pull/3096
After HADOOP-17728, `StatisticsDataReferenceCleaner` may appear the
following questions, That's beacuse `ReferenceQueue.remove` might return `null`.
2021-06-09 21:51:12,334 WARN
[org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread but
it will continue to run
java.lang.NullPointerException
at
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
at java.lang.Thread.run(Thread.java:748)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609777)
Remaining Estimate: 0h
Time Spent: 10m
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] yikf opened a new pull request #3096: [HADOOP-17758][HDFS][FOLLOWUP] NPE and excessive warnings after HADOOP-17728
yikf opened a new pull request #3096: URL: https://github.com/apache/hadoop/pull/3096 After HADOOP-17728, `StatisticsDataReferenceCleaner` may appear the following questions, That's beacuse `ReferenceQueue.remove` might return `null`. 2021-06-09 21:51:12,334 WARN [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner] fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread but it will continue to run java.lang.NullPointerException at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020) at java.lang.Thread.run(Thread.java:748) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361023#comment-17361023
]
yikf edited comment on HADOOP-17758 at 6/10/21, 3:21 PM:
-
gentle ping [~Jim_Brennan]
Thanks for your reporting, We change it beacuse that It will wait forerver if
we have't the timeout and the RerferenceQueue is empty;
eg (Suppose we hav't the timeout):
# we call the RerferenceQueue.remove()
# if we poll is empty from the queue, RerferenceQueue will
lock.wait(0)
# cleanUp cant work due to the DeadLock
>From above, we nee the timeout to prevent deadLock.
reference JDK ReferenctQueue:
{code:java}
public Reference remove(long timeout)
throws IllegalArgumentException, InterruptedException
{
if (timeout < 0) {
throw new IllegalArgumentException("Negative timeout value");
}
synchronized (lock) {
Reference r = reallyPoll();
if (r != null) return r;
long start = (timeout == 0) ? 0 : System.nanoTime();
for (;;) {
lock.wait(timeout);
r = reallyPoll();
if (r != null) return r;
if (timeout != 0) {
long end = System.nanoTime();
timeout -= (end - start) / 1000_000;
if (timeout <= 0) return null;
start = end;
}
}
}
}
{code}
But i ignore it that the return of method of the remove may be is null, i will
fix the issue, Thanks~
was (Author: kaifeiyi):
Thanks for your reporting, We change it beacuse that It will wait forerver if
we have't the timeout and the RerferenceQueue is empty;
eg (Suppose we hav't the timeout):
# we call the RerferenceQueue.remove()
# if we poll is empty from the queue, RerferenceQueue will
lock.wait(0)
# cleanUp cant work due to the DeadLock
>From above, we nee the timeout to prevent deadLock.
reference JDK ReferenctQueue:
{code:java}
public Reference remove(long timeout)
throws IllegalArgumentException, InterruptedException
{
if (timeout < 0) {
throw new IllegalArgumentException("Negative timeout value");
}
synchronized (lock) {
Reference r = reallyPoll();
if (r != null) return r;
long start = (timeout == 0) ? 0 : System.nanoTime();
for (;;) {
lock.wait(timeout);
r = reallyPoll();
if (r != null) return r;
if (timeout != 0) {
long end = System.nanoTime();
timeout -= (end - start) / 1000_000;
if (timeout <= 0) return null;
start = end;
}
}
}
}
{code}
But i ignore it that the return of method of the remove may be is null, i will
fix the issue, Thanks~
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
[
https://issues.apache.org/jira/browse/HADOOP-17758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361023#comment-17361023
]
yikf commented on HADOOP-17758:
---
Thanks for your reporting, We change it beacuse that It will wait forerver if
we have't the timeout and the RerferenceQueue is empty;
eg (Suppose we hav't the timeout):
# we call the RerferenceQueue.remove()
# if we poll is empty from the queue, RerferenceQueue will
lock.wait(0)
# cleanUp cant work due to the DeadLock
>From above, we nee the timeout to prevent deadLock.
reference JDK ReferenctQueue:
{code:java}
public Reference remove(long timeout)
throws IllegalArgumentException, InterruptedException
{
if (timeout < 0) {
throw new IllegalArgumentException("Negative timeout value");
}
synchronized (lock) {
Reference r = reallyPoll();
if (r != null) return r;
long start = (timeout == 0) ? 0 : System.nanoTime();
for (;;) {
lock.wait(timeout);
r = reallyPoll();
if (r != null) return r;
if (timeout != 0) {
long end = System.nanoTime();
timeout -= (end - start) / 1000_000;
if (timeout <= 0) return null;
start = end;
}
}
}
}
{code}
But i ignore it that the return of method of the remove may be is null, i will
fix the issue, Thanks~
> NPE and excessive warnings after HADOOP-17728
> -
>
> Key: HADOOP-17758
> URL: https://issues.apache.org/jira/browse/HADOOP-17758
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Priority: Major
>
> I'm noticing these warnings and NPE's when just running a simple pi test on a
> one node cluster:
> {noformat}
> 2021-06-09 21:51:12,334 WARN
> [org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
> fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread
> but it will continue to run
> java.lang.NullPointerException
> at
> org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
> at java.lang.Thread.run(Thread.java:748){noformat}
> This appears to be due to [HADOOP-17728].
> I'm not sure I understand why that change was made? Wasn't it by design that
> the remove should wait until something is queued?
> [~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3094: [YARN-9246][Test][followup] Add test for YARN-9246
hadoop-yetus commented on pull request #3094:
URL: https://github.com/apache/hadoop/pull/3094#issuecomment-858694173
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 15m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 47s | | trunk passed |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 14m 9s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 14m 29s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 17s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3094/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 27
new + 0 unchanged - 0 fixed = 27 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 13m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 27m 39s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 113m 22s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3094/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3094 |
| JIRA Issue | YARN-9246 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux cd81e8aaf24e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b2a3fa3ccc969f643546f7e8e4e6116ac08c9fa |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3094/1/testReport/ |
| Max. process+thread count | 726 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
[jira] [Created] (HADOOP-17758) NPE and excessive warnings after HADOOP-17728
Jim Brennan created HADOOP-17758:
Summary: NPE and excessive warnings after HADOOP-17728
Key: HADOOP-17758
URL: https://issues.apache.org/jira/browse/HADOOP-17758
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 3.4.0
Reporter: Jim Brennan
I'm noticing these warnings and NPE's when just running a simple pi test on a
one node cluster:
{noformat}
2021-06-09 21:51:12,334 WARN
[org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner]
fs.FileSystem (FileSystem.java:run(4025)) - Exception in the cleaner thread but
it will continue to run
java.lang.NullPointerException
at
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4020)
at java.lang.Thread.run(Thread.java:748){noformat}
This appears to be due to [HADOOP-17728].
I'm not sure I understand why that change was made? Wasn't it by design that
the remove should wait until something is queued?
[~kaifeiYi] can you please investigate?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] amahussein opened a new pull request #3095: HDFS-16061. DFTestUtil.waitReplication can produce false positives
amahussein opened a new pull request #3095: URL: https://github.com/apache/hadoop/pull/3095 HDFS-16061 DFTestUtil.waitReplication can produce false positives While checking the intermittent failure in `TestBalancerRPCDelay#testBalancerRPCDelayQpsDefault` described in HDFS-15146, I found that the implementation of waitReplication is incorrect. In the last iteration, when correctReplFactor is false, the thread sleeps for 1 second, then a TimeoutException is thrown without check whether the replication was complete in the last second. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3094: [YARN-9246][Test][followup] Add test for YARN-9246
hadoop-yetus commented on pull request #3094:
URL: https://github.com/apache/hadoop/pull/3094#issuecomment-858684579
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 25s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 45s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 16m 7s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3094/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 27
new + 0 unchanged - 0 fixed = 27 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 14m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 27m 54s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 102m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3094/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3094 |
| JIRA Issue | YARN-9246 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 1a1c0444d675 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17
17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b2a3fa3ccc969f643546f7e8e4e6116ac08c9fa |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3094/2/testReport/ |
| Max. process+thread count | 656 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3063: HDFS-16043. HDFS: Delete performance optimization
hadoop-yetus commented on pull request #3063:
URL: https://github.com/apache/hadoop/pull/3063#issuecomment-858676218
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 25s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 7s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3063/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 6 new + 907 unchanged
- 0 fixed = 913 total (was 907) |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 17m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 252m 13s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3063/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 341m 54s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3063/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3063 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 4bcfe8646e6b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68e87aa2be50994560946e49a8d404943d84f59b |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3063/2/testReport/ |
| Max. process+thread count | 3387 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #3093: [YARN-10817][Client] Avoid NPE when use yarn command like yarn queue -status
hadoop-yetus commented on pull request #3093:
URL: https://github.com/apache/hadoop/pull/3093#issuecomment-858648853
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 36s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 0m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 17m 2s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 17m 19s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 27m 10s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 104m 38s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3093/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3093 |
| JIRA Issue | YARN-10817 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 80682f44 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2b94e1d7c38cf80b51c4bb34c060d8afa6abb349 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3093/1/testReport/ |
| Max. process+thread count | 538 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3093/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
[GitHub] [hadoop] hadoop-yetus commented on pull request #3091: HDFS-16023.Improve blockReportLeaseId acquisition to avoid repeated FBR.
hadoop-yetus commented on pull request #3091:
URL: https://github.com/apache/hadoop/pull/3091#issuecomment-858631882
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 9m 48s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3091/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 9s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 242m 31s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3091/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 305m 20s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockReportLease |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3091/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3091 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 697925ec8657 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ff39e4aba55fb868d9f689ddf75f5a175ddf2fa7 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3091/1/testReport/ |
| Max. process+thread count | 3092 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3091/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
[GitHub] [hadoop] hadoop-yetus commented on pull request #3084: HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE …
hadoop-yetus commented on pull request #3084:
URL: https://github.com/apache/hadoop/pull/3084#issuecomment-858624450
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 54s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 241m 25s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 325m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3084/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3084 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c43175376ddc 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 24c6af4ace4629a316b5fb8bc54efad3cd5d161a |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3084/5/testReport/ |
| Max. process+thread count | 3236 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3084/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail:
[GitHub] [hadoop] yikf opened a new pull request #3094: [YARN-9246][Client][backport] Add test for YARN-9246
yikf opened a new pull request #3094: URL: https://github.com/apache/hadoop/pull/3094 When complete YARN-9246 , I found there are no test for this patch. it's better to add the test to make sure interface stable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] yikf closed pull request #3093: [YARN-10817][Client] Avoid NullPointerException when use yarn shell
yikf closed pull request #3093: URL: https://github.com/apache/hadoop/pull/3093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] yikf commented on pull request #3093: [YARN-10817][Client] Avoid NullPointerException when use yarn shell
yikf commented on pull request #3093: URL: https://github.com/apache/hadoop/pull/3093#issuecomment-858588205 This issue had been fixed in [YARN-9246](https://issues.apache.org/jira/browse/YARN-9246), Will close the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] yikf opened a new pull request #3093: [YARN-10817][Client] Avoid NullPointerException when use yarn shell
yikf opened a new pull request #3093: URL: https://github.com/apache/hadoop/pull/3093 **What changes were proposed in this pull request?** Currently, We will encounter NullPointerException when we use some yarn shell commands, As follow: ``` Missing argument for options usage: queue -help Displays help for all commands. -statusList queue information about given queue. Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.yarn.client.cli.YarnCLI.stop(YarnCLI.java:75) at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:51) ``` Like this, and ``` yarn cluster yarn queue yarn node ``` This is beacuse we dont`t init the `YarnClient` when missing argument for options, This PR will fix the issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17752) Remove lock contention in REGISTRY of Configuration
[ https://issues.apache.org/jira/browse/HADOOP-17752?focusedWorklogId=609661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609661 ] ASF GitHub Bot logged work on HADOOP-17752: --- Author: ASF GitHub Bot Created on: 10/Jun/21 11:49 Start Date: 10/Jun/21 11:49 Worklog Time Spent: 10m Work Description: liangxs commented on pull request #3085: URL: https://github.com/apache/hadoop/pull/3085#issuecomment-858553624 > ... > I think this is ok, because configurations are not very frequently reloaded. > For newly created conf, `Configuration#properties` is null, which will trigger reload operation when access. So I think it is not a problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609661) Time Spent: 1h 10m (was: 1h) > Remove lock contention in REGISTRY of Configuration > --- > > Key: HADOOP-17752 > URL: https://issues.apache.org/jira/browse/HADOOP-17752 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Xuesen Liang >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Every Configuration instance is put into *Configuration#REGISTRY* by its > constructor. This operation is guard by Configuration.class. > REGISTRY is a *WeakHashMap*, which should be replaced by *ConcurrentHashMap*. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liangxs commented on pull request #3085: HADOOP-17752. Remove lock contention in REGISTRY of Configuration
liangxs commented on pull request #3085: URL: https://github.com/apache/hadoop/pull/3085#issuecomment-858553624 > ... > I think this is ok, because configurations are not very frequently reloaded. > For newly created conf, `Configuration#properties` is null, which will trigger reload operation when access. So I think it is not a problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] AlphaGouGe commented on pull request #3065: HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
AlphaGouGe commented on pull request #3065: URL: https://github.com/apache/hadoop/pull/3065#issuecomment-858550741 @aajisaka @ferhui Thanks for comments, i have fixed all mentions, take a look please. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17752) Remove lock contention in REGISTRY of Configuration
[ https://issues.apache.org/jira/browse/HADOOP-17752?focusedWorklogId=609583=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609583 ] ASF GitHub Bot logged work on HADOOP-17752: --- Author: ASF GitHub Bot Created on: 10/Jun/21 09:33 Start Date: 10/Jun/21 09:33 Worklog Time Spent: 10m Work Description: liangxs edited a comment on pull request #3085: URL: https://github.com/apache/hadoop/pull/3085#issuecomment-858455111 @steveloughran Thanks for you comment. The only problem may occur as follows 0. REGISTRY contains two configurations: `conf_b` and `conf_c` 1. Thread t1 is reloading all Configurations of REGISTRY, the state and the map iterator order are [`conf_b`(reloaded) **->** `conf_c`(_reloading_)] 2. Meanwhile, another thread t2 creates `conf_a` and then creates `conf_d`, which are both put into REGISTRY. 3. Thread t1 continues and completes reload operation. 3. Now the state may be [`conf_a`(**not_reload**⚠️) **->** `conf_b`(reloaded) **->**`conf_c`(reloaded) **->** `conf_d`(reloaded)] For thread t2, `conf_a` is happen-before `conf_d`, but `conf_a` is not reloaded and `conf_d` is reloaded. I think this is ok, because configurations are not very frequently reloaded. If this is a concern, I can add a ReentrantReadWriteLock to prevent it. The reload operation can be guard by WriteLock, because configurations are not very frequently reloaded. The put operations can be guard by ReadLock, which gain performance from parallel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609583) Time Spent: 1h (was: 50m) > Remove lock contention in REGISTRY of Configuration > --- > > Key: HADOOP-17752 > URL: https://issues.apache.org/jira/browse/HADOOP-17752 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Xuesen Liang >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Every Configuration instance is put into *Configuration#REGISTRY* by its > constructor. This operation is guard by Configuration.class. > REGISTRY is a *WeakHashMap*, which should be replaced by *ConcurrentHashMap*. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu opened a new pull request #3092: HDFS-16062.When a DataNode hot reload configuration, JMX will block for a long time.
jianghuazhu opened a new pull request #3092: URL: https://github.com/apache/hadoop/pull/3092 …or a long time. ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liangxs edited a comment on pull request #3085: HADOOP-17752. Remove lock contention in REGISTRY of Configuration
liangxs edited a comment on pull request #3085: URL: https://github.com/apache/hadoop/pull/3085#issuecomment-858455111 @steveloughran Thanks for you comment. The only problem may occur as follows 0. REGISTRY contains two configurations: `conf_b` and `conf_c` 1. Thread t1 is reloading all Configurations of REGISTRY, the state and the map iterator order are [`conf_b`(reloaded) **->** `conf_c`(_reloading_)] 2. Meanwhile, another thread t2 creates `conf_a` and then creates `conf_d`, which are both put into REGISTRY. 3. Thread t1 continues and completes reload operation. 3. Now the state may be [`conf_a`(**not_reload**⚠️) **->** `conf_b`(reloaded) **->**`conf_c`(reloaded) **->** `conf_d`(reloaded)] For thread t2, `conf_a` is happen-before `conf_d`, but `conf_a` is not reloaded and `conf_d` is reloaded. I think this is ok, because configurations are not very frequently reloaded. If this is a concern, I can add a ReentrantReadWriteLock to prevent it. The reload operation can be guard by WriteLock, because configurations are not very frequently reloaded. The put operations can be guard by ReadLock, which gain performance from parallel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17752) Remove lock contention in REGISTRY of Configuration
[ https://issues.apache.org/jira/browse/HADOOP-17752?focusedWorklogId=609580=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609580 ] ASF GitHub Bot logged work on HADOOP-17752: --- Author: ASF GitHub Bot Created on: 10/Jun/21 09:21 Start Date: 10/Jun/21 09:21 Worklog Time Spent: 10m Work Description: liangxs edited a comment on pull request #3085: URL: https://github.com/apache/hadoop/pull/3085#issuecomment-858455111 @steveloughran Thanks for you comment. The only problem may occur as follows 0. REGISTRY contains two configurations: `conf_b` and `conf_c` 1. Thread t1 is reloading all Configurations of REGISTRY, the state and the map iterator order are [`conf_b`(reloaded) **->** `conf_c`(_reloading_)] 2. Meanwhile, another thread t2 creates `conf_a` and then creates `conf_d`, which are both put into REGISTRY. 3. Thread t1 continues and completes reload operation. 3. Now the state may be [`conf_a`(**not_reload**⚠️) **->** `conf_b`(reloaded) **->**`conf_c`(reloaded) **->** `conf_d`(reloaded)] For thread t2, `conf_a` is happen-before `conf_d`, but `conf_a` is not reloaded and `conf_d` is reloaded. I think this is ok. If this is a concern, I can add a ReentrantReadWriteLock to prevent it. The reload operation can be guard by WriteLock, because configurations are not very frequently reloaded. The put operations can be guard by ReadLock, which gain performance from parallel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609580) Time Spent: 50m (was: 40m) > Remove lock contention in REGISTRY of Configuration > --- > > Key: HADOOP-17752 > URL: https://issues.apache.org/jira/browse/HADOOP-17752 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Xuesen Liang >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Every Configuration instance is put into *Configuration#REGISTRY* by its > constructor. This operation is guard by Configuration.class. > REGISTRY is a *WeakHashMap*, which should be replaced by *ConcurrentHashMap*. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liangxs edited a comment on pull request #3085: HADOOP-17752. Remove lock contention in REGISTRY of Configuration
liangxs edited a comment on pull request #3085: URL: https://github.com/apache/hadoop/pull/3085#issuecomment-858455111 @steveloughran Thanks for you comment. The only problem may occur as follows 0. REGISTRY contains two configurations: `conf_b` and `conf_c` 1. Thread t1 is reloading all Configurations of REGISTRY, the state and the map iterator order are [`conf_b`(reloaded) **->** `conf_c`(_reloading_)] 2. Meanwhile, another thread t2 creates `conf_a` and then creates `conf_d`, which are both put into REGISTRY. 3. Thread t1 continues and completes reload operation. 3. Now the state may be [`conf_a`(**not_reload**⚠️) **->** `conf_b`(reloaded) **->**`conf_c`(reloaded) **->** `conf_d`(reloaded)] For thread t2, `conf_a` is happen-before `conf_d`, but `conf_a` is not reloaded and `conf_d` is reloaded. I think this is ok. If this is a concern, I can add a ReentrantReadWriteLock to prevent it. The reload operation can be guard by WriteLock, because configurations are not very frequently reloaded. The put operations can be guard by ReadLock, which gain performance from parallel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17752) Remove lock contention in REGISTRY of Configuration
[ https://issues.apache.org/jira/browse/HADOOP-17752?focusedWorklogId=609575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609575 ] ASF GitHub Bot logged work on HADOOP-17752: --- Author: ASF GitHub Bot Created on: 10/Jun/21 09:12 Start Date: 10/Jun/21 09:12 Worklog Time Spent: 10m Work Description: liangxs commented on pull request #3085: URL: https://github.com/apache/hadoop/pull/3085#issuecomment-858455111 @steveloughran Thanks for you comment. The only problem may occur as follows 0. REGISTRY contains two configurations: `conf_b` and `conf_c` 1. Thread t1 is reloading all Configurations of REGISTRY, the state and the map iterator order are [`conf_b`(reloaded) **->** `conf_c`(_reloading_)] 2. Meanwhile, another thread t2 creates `conf_a` and then creates `conf_d`, which are both put into REGISTRY. 3. Thread t1 done reload operation. 3. Now the state may be [`conf_a`(**not_reload**⚠️) **->** `conf_b`(reloaded) **->**`conf_c`(reloaded) **->** `conf_d`(reloaded)] For thread t2, `conf_a` is happen-before `conf_d`, but `conf_a` is not reloaded and `conf_d` is reloaded. I think this is ok. If this is a concern, I can add a ReentrantReadWriteLock to prevent it. The reload operation can be guard by WriteLock, because configurations are not very frequently reloaded. The put operations can be guard by ReadLock, which gain performance from parallel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609575) Time Spent: 40m (was: 0.5h) > Remove lock contention in REGISTRY of Configuration > --- > > Key: HADOOP-17752 > URL: https://issues.apache.org/jira/browse/HADOOP-17752 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Xuesen Liang >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Every Configuration instance is put into *Configuration#REGISTRY* by its > constructor. This operation is guard by Configuration.class. > REGISTRY is a *WeakHashMap*, which should be replaced by *ConcurrentHashMap*. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] liangxs commented on pull request #3085: HADOOP-17752. Remove lock contention in REGISTRY of Configuration
liangxs commented on pull request #3085: URL: https://github.com/apache/hadoop/pull/3085#issuecomment-858455111 @steveloughran Thanks for you comment. The only problem may occur as follows 0. REGISTRY contains two configurations: `conf_b` and `conf_c` 1. Thread t1 is reloading all Configurations of REGISTRY, the state and the map iterator order are [`conf_b`(reloaded) **->** `conf_c`(_reloading_)] 2. Meanwhile, another thread t2 creates `conf_a` and then creates `conf_d`, which are both put into REGISTRY. 3. Thread t1 done reload operation. 3. Now the state may be [`conf_a`(**not_reload**⚠️) **->** `conf_b`(reloaded) **->**`conf_c`(reloaded) **->** `conf_d`(reloaded)] For thread t2, `conf_a` is happen-before `conf_d`, but `conf_a` is not reloaded and `conf_d` is reloaded. I think this is ok. If this is a concern, I can add a ReentrantReadWriteLock to prevent it. The reload operation can be guard by WriteLock, because configurations are not very frequently reloaded. The put operations can be guard by ReadLock, which gain performance from parallel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu opened a new pull request #3091: HDFS-16023.Improve blockReportLeaseId acquisition to avoid repeated FBR.
jianghuazhu opened a new pull request #3091: URL: https://github.com/apache/hadoop/pull/3091 ## NOTICE Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull request which starts with the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.) For more details, please see https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #3086: HDFS-16039. RBF: Some indicators of RBFMetrics count inaccurately
zhuxiangyi commented on a change in pull request #3086:
URL: https://github.com/apache/hadoop/pull/3086#discussion_r648969521
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/metrics/RBFMetrics.java
##
@@ -372,12 +420,69 @@ private static void setStateStoreVersions(
@Override
public long getTotalCapacity() {
-return getNameserviceAggregatedLong(MembershipStats::getTotalSpace);
+return getNameserviceAggregatedLong(
+DatanodeReportType.LIVE, DatanodeInfo::getCapacity);
+ }
+
+ public LoadingCache getDnCache() {
+return dnCache;
+ }
+
+ /**
+ * Get the aggregated value for a DatanodeReportType and
+ * a method for all nameservices.
+ * @param type a DatanodeReportType
+ * @param f Method reference
+ * @return Aggregated long.
+ */
+ public long getNameserviceAggregatedLong(
+ DatanodeReportType type, ToLongFunction f){
+long size = 0;
+try {
+ size = Arrays.stream(dnCache.get(type)).mapToLong(f).sum();
Review comment:
Thank you for your reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17078) hadoop-shaded-protobuf_3_7 depends on the wrong version.
[ https://issues.apache.org/jira/browse/HADOOP-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-17078. Resolution: Duplicate > hadoop-shaded-protobuf_3_7 depends on the wrong version. > > > Key: HADOOP-17078 > URL: https://issues.apache.org/jira/browse/HADOOP-17078 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.1 >Reporter: JiangHua Zhu >Priority: Major > > When using maven to compile hadoop source code, the following exception > message appears: > [*INFO*] > ** > [*INFO*] *BUILD FAILURE* > [*INFO*] > ** > [*INFO*] Total time: 29.546 s > [*INFO*] Finished at: 2020-06-20T23:57:59+08:00 > [*INFO*] > ** > [*ERROR*] Failed to execute goal on project hadoop-common: *Could not resolve > dependencies for project org.apache.hadoop:hadoop-common:jar:3.3.0-SNAPSHOT: > Could not find artifact > org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.0.0-SNAPSHOT in > apache.snapshots.https > (https://repository.apache.org/content/repositories/snapshots)* -> *[Help 1]* > [*ERROR*] > [*ERROR*] To see the full stack trace of the errors, re-run Maven with the > *-e* switch. > [*ERROR*] Re-run Maven using the *-X* switch to enable full debug logging. > [*ERROR*] > [*ERROR*] For more information about the errors and possible solutions, > please read the following articles: > [*ERROR*] *[Help 1]* > http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException > [*ERROR*] > [*ERROR*] After correcting the problems, you can resume the build with the > command > [*ERROR*] *mvn -rf :hadoop-common* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-16988) Remove source code from branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-16988. Resolution: Done > Remove source code from branch-2 > > > Key: HADOOP-16988 > URL: https://issues.apache.org/jira/browse/HADOOP-16988 > Project: Hadoop Common > Issue Type: Task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > > Now, branch-2 is dead and unused. I think we can delete the entire source > code from branch-2 to avoid committing or cherry-picking to the unused branch. > Chen Liang asked ASF INFRA for help but it didn't help for us: INFRA-19581 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-16988) Remove source code from branch-2
[ https://issues.apache.org/jira/browse/HADOOP-16988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HADOOP-16988: > Remove source code from branch-2 > > > Key: HADOOP-16988 > URL: https://issues.apache.org/jira/browse/HADOOP-16988 > Project: Hadoop Common > Issue Type: Task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > > Now, branch-2 is dead and unused. I think we can delete the entire source > code from branch-2 to avoid committing or cherry-picking to the unused branch. > Chen Liang asked ASF INFRA for help but it didn't help for us: INFRA-19581 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17078) hadoop-shaded-protobuf_3_7 depends on the wrong version.
[ https://issues.apache.org/jira/browse/HADOOP-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HADOOP-17078: > hadoop-shaded-protobuf_3_7 depends on the wrong version. > > > Key: HADOOP-17078 > URL: https://issues.apache.org/jira/browse/HADOOP-17078 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.0.1 >Reporter: JiangHua Zhu >Priority: Major > > When using maven to compile hadoop source code, the following exception > message appears: > [*INFO*] > ** > [*INFO*] *BUILD FAILURE* > [*INFO*] > ** > [*INFO*] Total time: 29.546 s > [*INFO*] Finished at: 2020-06-20T23:57:59+08:00 > [*INFO*] > ** > [*ERROR*] Failed to execute goal on project hadoop-common: *Could not resolve > dependencies for project org.apache.hadoop:hadoop-common:jar:3.3.0-SNAPSHOT: > Could not find artifact > org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_7:jar:1.0.0-SNAPSHOT in > apache.snapshots.https > (https://repository.apache.org/content/repositories/snapshots)* -> *[Help 1]* > [*ERROR*] > [*ERROR*] To see the full stack trace of the errors, re-run Maven with the > *-e* switch. > [*ERROR*] Re-run Maven using the *-X* switch to enable full debug logging. > [*ERROR*] > [*ERROR*] For more information about the errors and possible solutions, > please read the following articles: > [*ERROR*] *[Help 1]* > http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException > [*ERROR*] > [*ERROR*] After correcting the problems, you can resume the build with the > command > [*ERROR*] *mvn -rf :hadoop-common* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14936) S3Guard: remove "experimental" from documentation
[ https://issues.apache.org/jira/browse/HADOOP-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-14936: --- Fix Version/s: 3.3.0 > S3Guard: remove "experimental" from documentation > - > > Key: HADOOP-14936 > URL: https://issues.apache.org/jira/browse/HADOOP-14936 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-beta1 >Reporter: Aaron Fabbri >Assignee: Gabor Bota >Priority: Major > Fix For: 3.3.0 > > > I think it is time to remove the "experimental feature" designation in the > site docs for S3Guard. Discuss. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17228) Backport HADOOP-13230 listing changes for preserved directory markers to 3.1.x
[ https://issues.apache.org/jira/browse/HADOOP-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-17228. Resolution: Won't Fix branch-3.1 is EOL, won't fix. > Backport HADOOP-13230 listing changes for preserved directory markers to 3.1.x > -- > > Key: HADOOP-17228 > URL: https://issues.apache.org/jira/browse/HADOOP-17228 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.4 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Backport a small subset of HADOOP-17199 to branch-3.1 > No path capabities, declarative test syntax etc > just > -getFileStatus/list > -markers changes to bucket-info > -startup info message if option is set > -relevant test changes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17228) Backport HADOOP-13230 listing changes for preserved directory markers to 3.1.x
[ https://issues.apache.org/jira/browse/HADOOP-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HADOOP-17228: > Backport HADOOP-13230 listing changes for preserved directory markers to 3.1.x > -- > > Key: HADOOP-17228 > URL: https://issues.apache.org/jira/browse/HADOOP-17228 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.4 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > > Backport a small subset of HADOOP-17199 to branch-3.1 > No path capabities, declarative test syntax etc > just > -getFileStatus/list > -markers changes to bucket-info > -startup info message if option is set > -relevant test changes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17097) start-build-env.sh fails in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-17097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved HADOOP-17097.
Resolution: Won't Fix
branch-3.1 is EoL, won't fix.
> start-build-env.sh fails in branch-3.1
> --
>
> Key: HADOOP-17097
> URL: https://issues.apache.org/jira/browse/HADOOP-17097
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Environment: Ubuntu 20.04
>Reporter: Akira Ajisaka
>Assignee: Masatake Iwasaki
>Priority: Critical
>
> ./start-build-env.sh fails to install ember-cli
> {noformat}
> npm ERR! Linux 5.4.0-37-generic
> npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "ember-cli"
> npm ERR! node v4.2.6
> npm ERR! npm v3.5.2
> npm ERR! code EMISSINGARG
> npm ERR! typeerror Error: Missing required argument #1
> npm ERR! typeerror at andLogAndFinish
> (/usr/share/npm/lib/fetch-package-metadata.js:31:3)
> npm ERR! typeerror at fetchPackageMetadata
> (/usr/share/npm/lib/fetch-package-metadata.js:51:22)
> npm ERR! typeerror at resolveWithNewModule
> (/usr/share/npm/lib/install/deps.js:456:12)
> npm ERR! typeerror at /usr/share/npm/lib/install/deps.js:457:7
> npm ERR! typeerror at /usr/share/npm/node_modules/iferr/index.js:13:50
> npm ERR! typeerror at /usr/share/npm/lib/fetch-package-metadata.js:37:12
> npm ERR! typeerror at addRequestedAndFinish
> (/usr/share/npm/lib/fetch-package-metadata.js:82:5)
> npm ERR! typeerror at returnAndAddMetadata
> (/usr/share/npm/lib/fetch-package-metadata.js:117:7)
> npm ERR! typeerror at pickVersionFromRegistryDocument
> (/usr/share/npm/lib/fetch-package-metadata.js:134:20)
> npm ERR! typeerror at /usr/share/npm/node_modules/iferr/index.js:13:50
> npm ERR! typeerror This is an error with npm itself. Please report this error
> at:
> npm ERR! typeerror
> npm ERR! Please include the following file with any support request:
> npm ERR! /root/npm-debug.log
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17550) property 'ssl.server.keystore.location' has not been set in the ssl configuration file
[ https://issues.apache.org/jira/browse/HADOOP-17550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HADOOP-17550: > property 'ssl.server.keystore.location' has not been set in the ssl > configuration file > -- > > Key: HADOOP-17550 > URL: https://issues.apache.org/jira/browse/HADOOP-17550 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.8.5 >Reporter: hamado dene >Priority: Major > > I trying to install hadoop cluster HA , but datanode does not start properly; > I get this errror: > 2021-02-23 17:13:26,934 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain > java.io.IOException: java.security.GeneralSecurityException: The property > 'ssl.server.keystore.location' has not been set in the ssl configuration file. > at > org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.(DatanodeHttpServer.java:199) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:905) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1303) > at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:481) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2609) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2497) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2544) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2729) > at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2753) > Caused by: java.security.GeneralSecurityException: The property > 'ssl.server.keystore.location' has not been set in the ssl configuration file. > at > org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:152) > at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:148) > at > org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.(DatanodeHttpServer.java:197) > ... 8 more > But in my ssl-server.xml i correctly set this property: > > ssl.server.keystore.location > /data/hadoop/server.jks > Keystore to be used by clients like distcp. Must be > specified. > > > > ssl.server.keystore.password > > Optional. Default value is "". > > > > ssl.server.keystore.keypassword > x > Optional. Default value is "". > > > > ssl.server.keystore.type > jks > Optional. The keystore file format, default value is "jks". > > > Do you have any suggestion to solve this problem? > my hadoop version is: 2.8.5 > java version: 8 > SO: centos 7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17097) start-build-env.sh fails in branch-3.1
[
https://issues.apache.org/jira/browse/HADOOP-17097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reopened HADOOP-17097:
> start-build-env.sh fails in branch-3.1
> --
>
> Key: HADOOP-17097
> URL: https://issues.apache.org/jira/browse/HADOOP-17097
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
> Environment: Ubuntu 20.04
>Reporter: Akira Ajisaka
>Assignee: Masatake Iwasaki
>Priority: Critical
>
> ./start-build-env.sh fails to install ember-cli
> {noformat}
> npm ERR! Linux 5.4.0-37-generic
> npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "ember-cli"
> npm ERR! node v4.2.6
> npm ERR! npm v3.5.2
> npm ERR! code EMISSINGARG
> npm ERR! typeerror Error: Missing required argument #1
> npm ERR! typeerror at andLogAndFinish
> (/usr/share/npm/lib/fetch-package-metadata.js:31:3)
> npm ERR! typeerror at fetchPackageMetadata
> (/usr/share/npm/lib/fetch-package-metadata.js:51:22)
> npm ERR! typeerror at resolveWithNewModule
> (/usr/share/npm/lib/install/deps.js:456:12)
> npm ERR! typeerror at /usr/share/npm/lib/install/deps.js:457:7
> npm ERR! typeerror at /usr/share/npm/node_modules/iferr/index.js:13:50
> npm ERR! typeerror at /usr/share/npm/lib/fetch-package-metadata.js:37:12
> npm ERR! typeerror at addRequestedAndFinish
> (/usr/share/npm/lib/fetch-package-metadata.js:82:5)
> npm ERR! typeerror at returnAndAddMetadata
> (/usr/share/npm/lib/fetch-package-metadata.js:117:7)
> npm ERR! typeerror at pickVersionFromRegistryDocument
> (/usr/share/npm/lib/fetch-package-metadata.js:134:20)
> npm ERR! typeerror at /usr/share/npm/node_modules/iferr/index.js:13:50
> npm ERR! typeerror This is an error with npm itself. Please report this error
> at:
> npm ERR! typeerror
> npm ERR! Please include the following file with any support request:
> npm ERR! /root/npm-debug.log
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16993) Hadoop 3.1.2 download link is broken
[ https://issues.apache.org/jira/browse/HADOOP-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-16993: --- Fix Version/s: asf-site > Hadoop 3.1.2 download link is broken > > > Key: HADOOP-16993 > URL: https://issues.apache.org/jira/browse/HADOOP-16993 > Project: Hadoop Common > Issue Type: Bug > Components: website >Reporter: Arpit Agarwal >Assignee: Akira Ajisaka >Priority: Major > Fix For: asf-site > > > Remove broken Hadoop 3.1.2 download links from the website. > https://hadoop.apache.org/releases.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17550) property 'ssl.server.keystore.location' has not been set in the ssl configuration file
[ https://issues.apache.org/jira/browse/HADOOP-17550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-17550. Resolution: Not A Problem > property 'ssl.server.keystore.location' has not been set in the ssl > configuration file > -- > > Key: HADOOP-17550 > URL: https://issues.apache.org/jira/browse/HADOOP-17550 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.8.5 >Reporter: hamado dene >Priority: Major > > I trying to install hadoop cluster HA , but datanode does not start properly; > I get this errror: > 2021-02-23 17:13:26,934 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain > java.io.IOException: java.security.GeneralSecurityException: The property > 'ssl.server.keystore.location' has not been set in the ssl configuration file. > at > org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.(DatanodeHttpServer.java:199) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startInfoServer(DataNode.java:905) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1303) > at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:481) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2609) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2497) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2544) > at > org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2729) > at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2753) > Caused by: java.security.GeneralSecurityException: The property > 'ssl.server.keystore.location' has not been set in the ssl configuration file. > at > org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:152) > at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:148) > at > org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer.(DatanodeHttpServer.java:197) > ... 8 more > But in my ssl-server.xml i correctly set this property: > > ssl.server.keystore.location > /data/hadoop/server.jks > Keystore to be used by clients like distcp. Must be > specified. > > > > ssl.server.keystore.password > > Optional. Default value is "". > > > > ssl.server.keystore.keypassword > x > Optional. Default value is "". > > > > ssl.server.keystore.type > jks > Optional. The keystore file format, default value is "jks". > > > Do you have any suggestion to solve this problem? > my hadoop version is: 2.8.5 > java version: 8 > SO: centos 7 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17623) Add a publish section to the .asf.yaml file
[ https://issues.apache.org/jira/browse/HADOOP-17623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-17623: --- Fix Version/s: asf-site > Add a publish section to the .asf.yaml file > --- > > Key: HADOOP-17623 > URL: https://issues.apache.org/jira/browse/HADOOP-17623 > Project: Hadoop Common > Issue Type: Task >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Fix For: asf-site > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17651) Backport to branch-3.1 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39
[ https://issues.apache.org/jira/browse/HADOOP-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HADOOP-17651. Resolution: Won't Fix branch-3.1 is EoL. Closing as won't fix. > Backport to branch-3.1 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update > Jetty to 9.4.39 > - > > Key: HADOOP-17651 > URL: https://issues.apache.org/jira/browse/HADOOP-17651 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.2.3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Reopened] (HADOOP-17651) Backport to branch-3.1 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39
[ https://issues.apache.org/jira/browse/HADOOP-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened HADOOP-17651: > Backport to branch-3.1 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update > Jetty to 9.4.39 > - > > Key: HADOOP-17651 > URL: https://issues.apache.org/jira/browse/HADOOP-17651 > Project: Hadoop Common > Issue Type: Task >Affects Versions: 3.2.3 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-17757) Reduce lock contention in properties of Configuration
[ https://issues.apache.org/jira/browse/HADOOP-17757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuesen Liang updated HADOOP-17757: -- Parent: HADOOP-17751 Issue Type: Sub-task (was: Improvement) > Reduce lock contention in properties of Configuration > - > > Key: HADOOP-17757 > URL: https://issues.apache.org/jira/browse/HADOOP-17757 > Project: Hadoop Common > Issue Type: Sub-task > Components: common >Reporter: Xuesen Liang >Priority: Major > > The *properties* field of class *Configuration* is guard by _synchronized_, > and its type java.util.Property is a sub class of java.util.*Hashtable*, > which methods are all synchronized. These are bad for multithread performance. > The properties field should be replaced by a *HashMap* with a > *ReentrantReadWriteLock*, then all _get()_ related operations can be parallel > guard by ReadLock, and _set/load_ operations can be guard by WriteLock. This > is better for performance. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17757) Reduce lock contention in properties of Configuration
Xuesen Liang created HADOOP-17757: - Summary: Reduce lock contention in properties of Configuration Key: HADOOP-17757 URL: https://issues.apache.org/jira/browse/HADOOP-17757 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Xuesen Liang The *properties* field of class *Configuration* is guard by _synchronized_, and its type java.util.Property is a sub class of java.util.*Hashtable*, which methods are all synchronized. These are bad for multithread performance. The properties field should be replaced by a *HashMap* with a *ReentrantReadWriteLock*, then all _get()_ related operations can be parallel guard by ReadLock, and _set/load_ operations can be guard by WriteLock. This is better for performance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17756) Increase precommit job timeout from 20 hours to 24 hours.
[ https://issues.apache.org/jira/browse/HADOOP-17756?focusedWorklogId=609498=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609498 ] ASF GitHub Bot logged work on HADOOP-17756: --- Author: ASF GitHub Bot Created on: 10/Jun/21 06:57 Start Date: 10/Jun/21 06:57 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3090: URL: https://github.com/apache/hadoop/pull/3090#issuecomment-858365682 @virajjasani Thanks for your review. Agreed with you. Updated the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609498) Time Spent: 0.5h (was: 20m) > Increase precommit job timeout from 20 hours to 24 hours. > - > > Key: HADOOP-17756 > URL: https://issues.apache.org/jira/browse/HADOOP-17756 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > If QA runs for the whole project, it may not be finished within 20 hours. I > suggest extending the timeout to 1day. > * https://github.com/apache/hadoop/pull/3049 : 17 hours. (finished) > * https://github.com/apache/hadoop/pull/3087 : 20 hours. (timeout) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tasanuma commented on pull request #3090: HADOOP-17756. Increase precommit job timeout from 20 hours to 24 hours.
tasanuma commented on pull request #3090: URL: https://github.com/apache/hadoop/pull/3090#issuecomment-858365682 @virajjasani Thanks for your review. Agreed with you. Updated the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17753) Keep restrict-imports-enforcer-rule for Guava Lists in hadoop-main pom
[ https://issues.apache.org/jira/browse/HADOOP-17753?focusedWorklogId=609495=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609495 ] ASF GitHub Bot logged work on HADOOP-17753: --- Author: ASF GitHub Bot Created on: 10/Jun/21 06:45 Start Date: 10/Jun/21 06:45 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3087: URL: https://github.com/apache/hadoop/pull/3087#issuecomment-858357924 Sure, Thanks. Exactly, this timeout keeps happening quite often. I was surprised to see full QA getting completed in 17+ hr in the first run on #3049 :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 609495) Time Spent: 50m (was: 40m) > Keep restrict-imports-enforcer-rule for Guava Lists in hadoop-main pom > -- > > Key: HADOOP-17753 > URL: https://issues.apache.org/jira/browse/HADOOP-17753 > Project: Hadoop Common > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #3087: HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom
virajjasani commented on pull request #3087: URL: https://github.com/apache/hadoop/pull/3087#issuecomment-858357924 Sure, Thanks. Exactly, this timeout keeps happening quite often. I was surprised to see full QA getting completed in 17+ hr in the first run on #3049 :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17756) Increase precommit job timeout from 20 hours to 24 hours.
[
https://issues.apache.org/jira/browse/HADOOP-17756?focusedWorklogId=609493=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-609493
]
ASF GitHub Bot logged work on HADOOP-17756:
---
Author: ASF GitHub Bot
Created on: 10/Jun/21 06:42
Start Date: 10/Jun/21 06:42
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3090:
URL: https://github.com/apache/hadoop/pull/3090#issuecomment-858356498
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 20m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | shadedclient | 15m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 15m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 66m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3090/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3090 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs |
| uname | Linux c67562f5c308 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7fe02c28ed0071ef4aeb56c120f9455d08063da9 |
| Max. process+thread count | 594 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3090/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 609493)
Time Spent: 20m (was: 10m)
> Increase precommit job timeout from 20 hours to 24 hours.
> -
>
> Key: HADOOP-17756
> URL: https://issues.apache.org/jira/browse/HADOOP-17756
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If QA runs for the whole project, it may not be finished within 20 hours. I
> suggest extending the timeout to 1day.
> * https://github.com/apache/hadoop/pull/3049 : 17 hours. (finished)
> * https://github.com/apache/hadoop/pull/3087 : 20 hours. (timeout)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3090: HADOOP-17756. Increase precommit job timeout from 20 hours to 24 hours.
hadoop-yetus commented on pull request #3090:
URL: https://github.com/apache/hadoop/pull/3090#issuecomment-858356498
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 20m 25s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 27s | | Maven dependency ordering for branch |
| +1 :green_heart: | shadedclient | 15m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | shadedclient | 15m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 66m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3090/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3090 |
| Optional Tests | dupname asflicense codespell shellcheck shelldocs |
| uname | Linux c67562f5c308 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7fe02c28ed0071ef4aeb56c120f9455d08063da9 |
| Max. process+thread count | 594 (vs. ulimit of 5500) |
| modules | C: U: |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3090/1/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #3084: HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE …
hadoop-yetus commented on pull request #3084:
URL: https://github.com/apache/hadoop/pull/3084#issuecomment-858356272
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 54s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3084/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 38 unchanged -
0 fixed = 40 total (was 38) |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 230m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3084/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 313m 55s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3084/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3084 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 87a3faab9371 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8bb052985577e00b8e6b7656761f2530685bb9a3 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3084/4/testReport/ |
| Max. process+thread count | 3723 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3084/4/console |
| versions |
