[jira] [Created] (HDFS-16309) ViewFileSystem.setVerifyChecksum should not initialize all target filesystems
Abhishek Das created HDFS-16309: --- Summary: ViewFileSystem.setVerifyChecksum should not initialize all target filesystems Key: HDFS-16309 URL: https://issues.apache.org/jira/browse/HDFS-16309 Project: Hadoop HDFS Issue Type: Bug Reporter: Abhishek Das Assignee: Abhishek Das Currently setVerifyChecksum and setWriteChecksum initializes all target file systems which causes delay in hadoop shell copy commands such as get, put, copyFromLocal etc. This also eventually causes OOM. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16308) Enable get command run with multi-thread.
[ https://issues.apache.org/jira/browse/HDFS-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengwei Wang resolved HDFS-16308. -- Resolution: Auto Closed > Enable get command run with multi-thread. > - > > Key: HDFS-16308 > URL: https://issues.apache.org/jira/browse/HDFS-16308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chengwei Wang >Assignee: Chengwei Wang >Priority: Major > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16308) Enable get command run with multi-thread.
[ https://issues.apache.org/jira/browse/HDFS-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440878#comment-17440878 ] Chengwei Wang commented on HDFS-16308: -- Close this issue for wrong hadoop component choosen. Create HADOOP-17998 instead it. > Enable get command run with multi-thread. > - > > Key: HDFS-16308 > URL: https://issues.apache.org/jira/browse/HDFS-16308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chengwei Wang >Assignee: Chengwei Wang >Priority: Major > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16308) Enable get command run with multi-thread.
[ https://issues.apache.org/jira/browse/HDFS-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengwei Wang updated HDFS-16308: - Description: CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and HADOOP-14698， and make put dirs or multiple files faster. So, It's necessary to enable get and copyToLocal command run with multi-thread. was: CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and HADOOP-14698 . It's useful to enable get and copyToLocal command run with multi-thread. > Enable get command run with multi-thread. > - > > Key: HDFS-16308 > URL: https://issues.apache.org/jira/browse/HDFS-16308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chengwei Wang >Assignee: Chengwei Wang >Priority: Major > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698， and make put dirs or multiple files faster. > So, It's necessary to enable get and copyToLocal command run with > multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16308) Enable get command run with multi-thread.
[ https://issues.apache.org/jira/browse/HDFS-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengwei Wang updated HDFS-16308: - Description: CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and HADOOP-14698 . It's useful to enable get and copyToLocal command run with multi-thread. was:It's useful to enable get and copyToLocal command run with multi-thread. > Enable get command run with multi-thread. > - > > Key: HDFS-16308 > URL: https://issues.apache.org/jira/browse/HDFS-16308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chengwei Wang >Assignee: Chengwei Wang >Priority: Major > > CopyFromLocal/Put is enabled to run with multi-thread with HDFS-11786 and > HADOOP-14698 . > It's useful to enable get and copyToLocal command run with multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16308) Enable get command run with multi-thread.
[ https://issues.apache.org/jira/browse/HDFS-16308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengwei Wang updated HDFS-16308: - External issue ID: (was: HDFS-11786) > Enable get command run with multi-thread. > - > > Key: HDFS-16308 > URL: https://issues.apache.org/jira/browse/HDFS-16308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Chengwei Wang >Assignee: Chengwei Wang >Priority: Major > > It's useful to enable get and copyToLocal command run with multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16308) Enable get command run with multi-thread.
Chengwei Wang created HDFS-16308: Summary: Enable get command run with multi-thread. Key: HDFS-16308 URL: https://issues.apache.org/jira/browse/HDFS-16308 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Reporter: Chengwei Wang Assignee: Chengwei Wang It's useful to enable get and copyToLocal command run with multi-thread. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16305) Record the remote NameNode address when the rolling log is triggered
[ https://issues.apache.org/jira/browse/HDFS-16305?focusedWorklogId=678856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678856 ] ASF GitHub Bot logged work on HDFS-16305: - Author: ASF GitHub Bot Created on: 09/Nov/21 02:07 Start Date: 09/Nov/21 02:07 Worklog Time Spent: 10m Work Description: jianghuazhu commented on pull request #3629: URL: https://github.com/apache/hadoop/pull/3629#issuecomment-963746132 It looks like jenkins failed, there are some exceptions. E.g: TestWebHdfsFileSystemContract TestMover TestDataNodeUUID TestAddOverReplicatedStripedBlocks TestDecommission TestLeaseRecovery2 TestBlockReaderLocal It doesn't seem to have much to do with the pr I submitted. @jojochuang @virajjasani @tomscut, would you like to spare some time to help review this pr. Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 678856) Time Spent: 0.5h (was: 20m) > Record the remote NameNode address when the rolling log is triggered > > > Key: HDFS-16305 > URL: https://issues.apache.org/jira/browse/HDFS-16305 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.9.2 >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > When StandbyNN triggers the rolling log, it sends a request to the remote > NameNode. > Here are some log information: > ' > 2021-11-05 10:47:06,312 [3080066887]-INFO [Edit log > tailer:EditLogTailer@316]-Triggering log roll on remote NameNode > 2021-11-05 10:49:14,946 [3080195521]-INFO [Edit log > tailer:EditLogTailer@316]-Triggering log roll on remote NameNode > 2021-11-05 10:51:19,254 [3080319829]-INFO [Edit log > tailer:EditLogTailer@316]-Triggering log roll on remote NameNode > ' > The specific remote NameNode information cannot be seen from here. We can try > to record the specific address. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee resolved HDFS-16304.
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15964) Please update the okhttp version to 4.9.1
[ https://issues.apache.org/jira/browse/HDFS-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-15964: Target Version/s: 3.4.0, 3.3.3 (was: 3.4.0, 3.3.2) > Please update the okhttp version to 4.9.1 > - > > Key: HDFS-15964 > URL: https://issues.apache.org/jira/browse/HDFS-15964 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, dfsclient, security >Affects Versions: 3.3.0 >Reporter: helen huang >Priority: Major > > Currently the okhttp used by the hdfs client is 2.7.5. Our fortify scan > flagged two issues with this version. Please update it to the latest (It is > okhttp3 4.9.1 at this point). Thanks! > > com.squareup.okhttp3 > okhttp > 4.9.1 > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15982) Deleted data using HTTP API should be saved to the trash
[ https://issues.apache.org/jira/browse/HDFS-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-15982: Target Version/s: 3.4.0, 3.3.3 (was: 3.4.0, 3.3.2) > Deleted data using HTTP API should be saved to the trash > > > Key: HDFS-15982 > URL: https://issues.apache.org/jira/browse/HDFS-15982 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs, hdfs-client, httpfs, webhdfs >Reporter: Bhavik Patel >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Attachments: Screenshot 2021-04-23 at 4.19.42 PM.png, Screenshot > 2021-04-23 at 4.36.57 PM.png > > Time Spent: 13h 20m > Remaining Estimate: 0h > > If we delete the data from the Web UI then it should be first moved to > configured/default Trash directory and after the trash interval time, it > should be removed. currently, data directly removed from the system[This > behavior should be the same as CLI cmd] > This can be helpful when the user accidentally deletes data from the Web UI. > Similarly we should provide "Skip Trash" option in HTTP API as well which > should be accessible through Web UI. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15715) ReplicatorMonitor performance degrades, when the storagePolicy of many file are not match with their real datanodestorage
[
https://issues.apache.org/jira/browse/HDFS-15715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HDFS-15715:
Target Version/s: 3.3.3 (was: 3.3.2)
> ReplicatorMonitor performance degrades, when the storagePolicy of many file
> are not match with their real datanodestorage
> --
>
> Key: HDFS-15715
> URL: https://issues.apache.org/jira/browse/HDFS-15715
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.7.3, 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Attachments: HDFS-15715.001.patch, HDFS-15715.002.patch,
> HDFS-15715.002.patch.addendum, image-2021-03-26-12-17-45-500.png
>
>
> One of our Namenode which has 300M files and blocks. In common way, this
> namode shoud not be in heavy load. But we found rpc process time keep high,
> and decommission is very slow.
>
> I search the metrics, I found uderreplicated blocks keep high. Then I jstack
> namenode, found 'InnerNode.getLoc' is hot spot cod. I think maybe
> chooseTarget can't find block, so result to performance degradation. Consider
> with HDFS-10453, I guess maybe some logical trigger to the scene where
> chooseTarget can't find proper block.
> Then I enable some debug. (Of course I revise some code so that only debug
> isGoodTarget, because enable BlockPlacementPolicy's debug log is dangrouse).
> I found "the rack has too many chosen nodes" is called. Then I found some log
> like this
> {code}
> 2020-12-04 12:13:56,345 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 0 to reach 3 (unavailableStorages=[],
> storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK],
> creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For
> more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> 2020-12-04 12:14:03,843 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 0 to reach 3 (unavailableStorages=[],
> storagePolicy=BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE],
> creationFallbacks=[], replicationFallbacks=[]}, newBlock=false) For more
> information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> {code}
> Then through some debug and simulation, I found the reason, and reproduction
> this exception.
> The reason is that some developer use COLD storage policy and mover, but the
> operatiosn of setting storage policy and mover are asynchronous. So some
> file's real datanodestorages are not match with this storagePolicy.
> Let me simualte this proccess. If /tmp/a is create, then have 2 replications
> are DISK. Then set storage policy to COLD. When some logical trigger(For
> example decommission) to copy this block. chooseTarget then use
> chooseStorageTypes to filter real needed block. Here the size of variable
> requiredStorageTypes which chooseStorageTypes returned is 3. But the size of
> result is 2. But 3 means need 3 ARCHIVE storage. 2 means bocks has 2 DISK
> storage. Then will request to choose 3 target. choose first target is right,
> but when choose seconde target, the variable 'counter' is 4 which is larger
> than maxTargetPerRack which is 3 in function isGoodTarget. So skip all
> datanodestorage. Then result to bad performance.
> I think chooseStorageTypes need to consider the result, when the exist
> replication doesn't meet storage policy's demand, we need to remove this from
> result.
> I changed by this way, and test in my unit-test. Then solve it.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15505) Fix NullPointerException when call getAdditionalDatanode method with null extendedBlock parameter
[
https://issues.apache.org/jira/browse/HDFS-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HDFS-15505:
Target Version/s: 3.4.0, 3.3.3 (was: 3.4.0, 3.3.2)
> Fix NullPointerException when call getAdditionalDatanode method with null
> extendedBlock parameter
> -
>
> Key: HDFS-15505
> URL: https://issues.apache.org/jira/browse/HDFS-15505
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 3.0.0, 3.1.0, 3.0.1, 3.0.2, 3.2.0, 3.1.1, 3.0.3, 3.1.2,
> 3.3.0, 3.2.1, 3.1.3
>Reporter: hang chen
>Priority: Major
>
> When client call getAdditionalDatanode method, it will initialize
> GetAdditionalDatanodeRequestProto and send RPC request to Router/namenode.
> However, if we call getAdditionalDatanode method with null extendedBlock
> parameter, it will set GetAdditionalDatanodeRequestProto's blk field with
> null, which will cause NullPointerException. The code show as follow.
> {code:java}
> // code placeholder
> GetAdditionalDatanodeRequestProto req = GetAdditionalDatanodeRequestProto
> .newBuilder()
> .setSrc(src)
> .setFileId(fileId)
> .setBlk(PBHelperClient.convert(blk))
> .addAllExistings(PBHelperClient.convert(existings))
> .addAllExistingStorageUuids(Arrays.asList(existingStorageIDs))
> .addAllExcludes(PBHelperClient.convert(excludes))
> .setNumAdditionalNodes(numAdditionalNodes)
> .setClientName(clientName)
> .build();{code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15289) Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table
[ https://issues.apache.org/jira/browse/HDFS-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-15289: Target Version/s: 3.4.0, 3.2.4, 3.3.3 (was: 3.4.0, 3.3.2, 3.2.4) > Allow viewfs mounts with HDFS/HCFS scheme and centralized mount table > - > > Key: HDFS-15289 > URL: https://issues.apache.org/jira/browse/HDFS-15289 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 3.2.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Major > Attachments: ViewFSOverloadScheme - V1.0.pdf, ViewFSOverloadScheme.png > > > ViewFS provides flexibility to mount different filesystem types with mount > points configuration table. This approach is solving the scalability > problems, but users need to reconfigure the filesystem to ViewFS and to its > scheme. This will be problematic in the case of paths persisted in meta > stores, ex: Hive. In systems like Hive, it will store uris in meta store. So, > changing the file system scheme will create a burden to upgrade/recreate meta > stores. In our experience many users are not ready to change that. > Router based federation is another implementation to provide coordinated > mount points for HDFS federation clusters. Even though this provides > flexibility to handle mount points easily, this will not allow > other(non-HDFS) file systems to mount. So, this does not solve the purpose > when users want to mount external(non-HDFS) filesystems. > So, the problem here is: Even though many users want to adapt to the scalable > fs options available, technical challenges of changing schemes (ex: in meta > stores) in deployments are obstructing them. > So, we propose to allow hdfs scheme in ViewFS like client side mount system > and provision user to create mount links without changing URI paths. > I will upload detailed design doc shortly. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15965) Please upgrade the log4j dependency to log4j2
[ https://issues.apache.org/jira/browse/HDFS-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HDFS-15965: Target Version/s: 3.4.0, 3.3.3 (was: 3.4.0, 3.3.2) > Please upgrade the log4j dependency to log4j2 > - > > Key: HDFS-15965 > URL: https://issues.apache.org/jira/browse/HDFS-15965 > Project: Hadoop HDFS > Issue Type: Improvement > Components: dfsclient >Affects Versions: 3.3.0, 3.2.1, 3.2.2 >Reporter: helen huang >Priority: Major > > The log4j dependency being use by hadoop-common is currently version 1.2.17. > Our fortify scan picked up a couple of issue with this dependency. Please > update it to the latest version of log4j2 dependencies: > > org.apache.logging.log4j > log4j-api > 2.14.1 > > > org.apache.logging.log4j > log4j-core > 2.14.1 > > > The slf4j dependency will need to be updated as well after you upgrade log4j > to log4j2. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16307) Improve HdfsBlockPlacementPolicies docs readability
[
https://issues.apache.org/jira/browse/HDFS-16307?focusedWorklogId=678665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678665
]
ASF GitHub Bot logged work on HDFS-16307:
-
Author: ASF GitHub Bot
Created on: 08/Nov/21 18:50
Start Date: 08/Nov/21 18:50
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3631:
URL: https://github.com/apache/hadoop/pull/3631#issuecomment-963468683
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 49m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 73m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3631/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3631 |
| Optional Tests | dupname asflicense mvnsite codespell markdownlint |
| uname | Linux 05f25b7b5fd0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d5cf473761424c62697f2875f90ef4905da6e33d |
| Max. process+thread count | 548 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3631/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 678665)
Time Spent: 0.5h (was: 20m)
> Improve HdfsBlockPlacementPolicies docs readability
> ---
>
> Key: HDFS-16307
> URL: https://issues.apache.org/jira/browse/HDFS-16307
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.3.1
>Reporter: guo
>Assignee: guo
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> improve HdfsBlockPlacementPolicies docs readability
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16307) Improve HdfsBlockPlacementPolicies docs readability
[ https://issues.apache.org/jira/browse/HDFS-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned HDFS-16307: -- Assignee: guo > Improve HdfsBlockPlacementPolicies docs readability > --- > > Key: HDFS-16307 > URL: https://issues.apache.org/jira/browse/HDFS-16307 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Assignee: guo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16307) Improve HdfsBlockPlacementPolicies docs readability
[ https://issues.apache.org/jira/browse/HDFS-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-16307. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve HdfsBlockPlacementPolicies docs readability > --- > > Key: HDFS-16307 > URL: https://issues.apache.org/jira/browse/HDFS-16307 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16307) Improve HdfsBlockPlacementPolicies docs readability
[ https://issues.apache.org/jira/browse/HDFS-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440650#comment-17440650 ] Íñigo Goiri commented on HDFS-16307: Thanks [~philipse] for the patch. Merged PR 3631 to trunk. > Improve HdfsBlockPlacementPolicies docs readability > --- > > Key: HDFS-16307 > URL: https://issues.apache.org/jira/browse/HDFS-16307 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16307) Improve HdfsBlockPlacementPolicies docs readability
[ https://issues.apache.org/jira/browse/HDFS-16307?focusedWorklogId=678647=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678647 ] ASF GitHub Bot logged work on HDFS-16307: - Author: ASF GitHub Bot Created on: 08/Nov/21 18:21 Start Date: 08/Nov/21 18:21 Worklog Time Spent: 10m Work Description: goiri merged pull request #3631: URL: https://github.com/apache/hadoop/pull/3631 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 678647) Time Spent: 20m (was: 10m) > Improve HdfsBlockPlacementPolicies docs readability > --- > > Key: HDFS-16307 > URL: https://issues.apache.org/jira/browse/HDFS-16307 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16307) Improve HdfsBlockPlacementPolicies docs readability
[ https://issues.apache.org/jira/browse/HDFS-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-16307: --- Summary: Improve HdfsBlockPlacementPolicies docs readability (was: improve HdfsBlockPlacementPolicies docs) > Improve HdfsBlockPlacementPolicies docs readability > --- > > Key: HDFS-16307 > URL: https://issues.apache.org/jira/browse/HDFS-16307 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16307) improve HdfsBlockPlacementPolicies docs
[ https://issues.apache.org/jira/browse/HDFS-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16307: -- Labels: pull-request-available (was: ) > improve HdfsBlockPlacementPolicies docs > --- > > Key: HDFS-16307 > URL: https://issues.apache.org/jira/browse/HDFS-16307 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16307) improve HdfsBlockPlacementPolicies docs
[ https://issues.apache.org/jira/browse/HDFS-16307?focusedWorklogId=678635=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678635 ] ASF GitHub Bot logged work on HDFS-16307: - Author: ASF GitHub Bot Created on: 08/Nov/21 17:47 Start Date: 08/Nov/21 17:47 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3631: URL: https://github.com/apache/hadoop/pull/3631#issuecomment-963407262 > @GuoPhilipse thanks for the work. Can you link it with a JIRA? Thanks @goiri ,have just updated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 678635) Remaining Estimate: 0h Time Spent: 10m > improve HdfsBlockPlacementPolicies docs > --- > > Key: HDFS-16307 > URL: https://issues.apache.org/jira/browse/HDFS-16307 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16307) improve HdfsBlockPlacementPolicies docs
guo created HDFS-16307: -- Summary: improve HdfsBlockPlacementPolicies docs Key: HDFS-16307 URL: https://issues.apache.org/jira/browse/HDFS-16307 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: 3.3.1 Reporter: guo improve HdfsBlockPlacementPolicies docs readability -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440592#comment-17440592
]
Íñigo Goiri commented on HDFS-16304:
Thanks [~sjlee0] for reporting and [~gautham] for the fix.
Merged PR 3625 into trunk; hopefully this will help mitigate the issues.
The thing that concerns me is that we didn't catch this while merging
HDFS-16300.
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?focusedWorklogId=678622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678622
]
ASF GitHub Bot logged work on HDFS-16304:
-
Author: ASF GitHub Bot
Created on: 08/Nov/21 16:58
Start Date: 08/Nov/21 16:58
Worklog Time Spent: 10m
Work Description: goiri merged pull request #3625:
URL: https://github.com/apache/hadoop/pull/3625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 678622)
Time Spent: 1h 40m (was: 1.5h)
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?focusedWorklogId=678608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678608
]
ASF GitHub Bot logged work on HDFS-16304:
-
Author: ASF GitHub Bot
Created on: 08/Nov/21 16:39
Start Date: 08/Nov/21 16:39
Worklog Time Spent: 10m
Work Description: GauthamBanasandra commented on pull request #3625:
URL: https://github.com/apache/hadoop/pull/3625#issuecomment-963348257
> Is the build currently breaking for all PRs without this change?
@sodonnel yes, please see the JIRA of this PR for more details.
> We are getting native compile issues on #3579 too. Seems this fix may be
related.
Yes, this PR fixes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 678608)
Time Spent: 1h 20m (was: 1h 10m)
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?focusedWorklogId=678610=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678610
]
ASF GitHub Bot logged work on HDFS-16304:
-
Author: ASF GitHub Bot
Created on: 08/Nov/21 16:39
Start Date: 08/Nov/21 16:39
Worklog Time Spent: 10m
Work Description: GauthamBanasandra edited a comment on pull request
#3625:
URL: https://github.com/apache/hadoop/pull/3625#issuecomment-963348257
> Is the build currently breaking for all PRs without this change?
@sodonnel yes, please see the JIRA of this PR for more details.
> We are getting native compile issues on #3579 too. Seems this fix may be
related.
Yes, this PR fixes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 678610)
Time Spent: 1.5h (was: 1h 20m)
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440576#comment-17440576
]
István Fajth commented on HDFS-16304:
-
It seems that [this PR|https://github.com/apache/hadoop/pull/3579] is blocked
on the same issue, on the ubuntu images, compilation fails.
I am looking after this fix to be commited, and hope that will solve my problem
as well.
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?focusedWorklogId=678581=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678581
]
ASF GitHub Bot logged work on HDFS-16304:
-
Author: ASF GitHub Bot
Created on: 08/Nov/21 16:08
Start Date: 08/Nov/21 16:08
Worklog Time Spent: 10m
Work Description: sodonnel commented on pull request #3625:
URL: https://github.com/apache/hadoop/pull/3625#issuecomment-963309508
Is the build currently breaking for all PRs without this change? We are
getting native compile issues on #3579 too. Seems this fix may be related.
The last CI run is giving errors like below on the step
hadoop-hdfs-native-client:
```
[WARNING] Cannot find a usable OpenSSL library.
OPENSSL_LIBRARY=OPENSSL_LIBRARY-NOTFOUND, OPENSSL_INCLUDE_DIR=/usr/include,
CUSTOM_OPENSSL_LIB=, CUSTOM_OPENSSL_PREFIX=, CUSTOM_OPENSSL_INCLUDE=
[WARNING] CMake Error at CMakeLists.txt:146 (message):
[WARNING] Terminating build because require.openssl was specified.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 678581)
Time Spent: 1h 10m (was: 1h)
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16305) Record the remote NameNode address when the rolling log is triggered
[
https://issues.apache.org/jira/browse/HDFS-16305?focusedWorklogId=678460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678460
]
ASF GitHub Bot logged work on HDFS-16305:
-
Author: ASF GitHub Bot
Created on: 08/Nov/21 13:07
Start Date: 08/Nov/21 13:07
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3629:
URL: https://github.com/apache/hadoop/pull/3629#issuecomment-963132135
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 370m 54s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3629/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 477m 29s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3629/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3629 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux a74a6a57576d 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 09d0cea925b2451cdae9e7d1b12d6f0b05ab |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Updated] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gautham Banasandra updated HDFS-16304:
--
Attachment: build-log-ubuntu-focal.log
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440422#comment-17440422
]
Gautham Banasandra commented on HDFS-16304:
---
I verified by building with Dockerfile on Ubuntu Focal -
Verifying that the build contains this PR -
{code:bash}
$ git log HEAD...HEAD~4 > git-log-head.log
{code}
Output - [^git-log-head.log]
Building -
{code:bash}
$ mvn package -Dhttps.protocols=TLSv1.2 -DskipTests -Pnative,dist
-Drequire.fuse -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop -Pyarn-ui -Dtar | tee
build-log-ubuntu-focal.log
{code}
Build output - [^build-log-ubuntu-focal.log]
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build-log-ubuntu-focal.log,
> build_failure.txt, git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-16187:
---
Fix Version/s: 3.4.0
(was: 1.3.0)
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
> The side effect of this behavior is : distcp with snapshot diff would fail
> with below error complaining that target cluster has some data changed .
> {code:java}
> WARN tools.DistCp: The target has been modified since snapshot x
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work stopped] (HDFS-16306) Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
[ https://issues.apache.org/jira/browse/HDFS-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-16306 stopped by Haiyang Hu. - > Stale record should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson > > > Key: HDFS-16306 > URL: https://issues.apache.org/jira/browse/HDFS-16306 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=678424=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678424 ] ASF GitHub Bot logged work on HDFS-16287: - Author: ASF GitHub Bot Created on: 08/Nov/21 11:36 Start Date: 08/Nov/21 11:36 Worklog Time Spent: 10m Work Description: ferhui commented on a change in pull request #3596: URL: https://github.com/apache/hadoop/pull/3596#discussion_r744641344 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java ## @@ -260,17 +257,14 @@ final Timer timer = new Timer(); this.slowPeerTracker = dataNodePeerStatsEnabled ? new SlowPeerTracker(conf, timer) : null; -this.excludeSlowNodesEnabled = conf.getBoolean( Review comment: It is used in BlockPlacementPolicyDefault -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 678424) Time Spent: 10h 50m (was: 10h 40m) > Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable > - > > Key: HDFS-16287 > URL: https://issues.apache.org/jira/browse/HDFS-16287 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 10h 50m > Remaining Estimate: 0h > > 1. Consider that make dfs.namenode.avoid.read.slow.datanode reconfigurable > and rapid rollback in case this feature > [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected > things happen in production environment > 2. DatanodeManager#startSlowPeerCollector by parameter > 'dfs.datanode.peer.stats.enabled' to control -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16306) Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
[ https://issues.apache.org/jira/browse/HDFS-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-16306 started by Haiyang Hu. - > Stale record should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson > > > Key: HDFS-16306 > URL: https://issues.apache.org/jira/browse/HDFS-16306 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16306) Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
[ https://issues.apache.org/jira/browse/HDFS-16306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haiyang Hu reassigned HDFS-16306: - Assignee: Haiyang Hu > Stale record should be remove when > DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson > > > Key: HDFS-16306 > URL: https://issues.apache.org/jira/browse/HDFS-16306 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16306) Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson
Haiyang Hu created HDFS-16306: - Summary: Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson Key: HDFS-16306 URL: https://issues.apache.org/jira/browse/HDFS-16306 Project: Hadoop HDFS Issue Type: Bug Reporter: Haiyang Hu -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=678408=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678408
]
ASF GitHub Bot logged work on HDFS-16287:
-
Author: ASF GitHub Bot
Created on: 08/Nov/21 10:28
Start Date: 08/Nov/21 10:28
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-963013131
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 11s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 24s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 330m 15s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 436m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3596 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3388666e1f8e 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 87920fce0c2d28bf28fa2d0ff7589b142e0d3a31 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/5/testReport/ |
| Max. process+thread count | 2491 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This
[jira] [Work logged] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?focusedWorklogId=678399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-678399
]
ASF GitHub Bot logged work on HDFS-16304:
-
Author: ASF GitHub Bot
Created on: 08/Nov/21 10:01
Start Date: 08/Nov/21 10:01
Worklog Time Spent: 10m
Work Description: symious commented on pull request #3625:
URL: https://github.com/apache/hadoop/pull/3625#issuecomment-962989469
+1. The fix works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 678399)
Time Spent: 1h (was: 50m)
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build_failure.txt,
> git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440301#comment-17440301
]
Gautham Banasandra edited comment on HDFS-16304 at 11/8/21, 9:00 AM:
-
I did clean full build with my PR locally with *Dockerfile_debian_10* and
verified that the build goes through.
Verifying that the build contains this PR -
{code:bash}
$ git log HEAD...HEAD~4 > git-log-head.log
{code}
Output - [^git-log-head.log]
Building -
{code:bash}
$ mvn package -Dhttps.protocols=TLSv1.2 -DskipTests -Pnative,dist
-Drequire.fuse -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop -Pyarn-ui -Dtar | tee
build-log-debian-10.log
{code}
Build output - [^build-log-debian-10.log]
I'm going to do a build on Ubuntu Focal and upload the logs shortly.
was (Author: gautham):
I did clean full build with my PR locally with *Dockerfile_debian_10* and
verified that the build goes through.
Verifying that the build contains this PR -
{code:bash}
$ git log HEAD...HEAD~4 > git-log-head.log
{code}
Output - [^git-log-head.log]
Building -
{code:bash}
$ mvn package -Dhttps.protocols=TLSv1.2 -DskipTests -Pnative,dist
-Drequire.fuse -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop -Pyarn-ui -Dtar | tee
build-log.debian-10.log
{code}
Build output - [^build-log-debian-10.log]
I'm going to do a build on Ubuntu Focal and upload the logs shortly.
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build_failure.txt,
> git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440301#comment-17440301
]
Gautham Banasandra commented on HDFS-16304:
---
I did clean full build with my PR locally with Dockerfile_debian_10 and
verified that the build goes through.
Verifying that the build contains this PR -
{code:bash}
$ git log HEAD...HEAD~4 > git-log-head.log
{code}
Output - [^git-log-head.log]
Building -
{code:bash}
$ mvn package -Dhttps.protocols=TLSv1.2 -DskipTests -Pnative,dist
-Drequire.fuse -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop -Pyarn-ui -Dtar | tee
build-log.debian-10.log
{code}
Build output - [^build-log-debian-10.log]
I'm going to do a build on Ubuntu Focal and upload the logs shortly.
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build_failure.txt,
> git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440301#comment-17440301
]
Gautham Banasandra edited comment on HDFS-16304 at 11/8/21, 9:00 AM:
-
I did clean full build with my PR locally with *Dockerfile_debian_10* and
verified that the build goes through.
Verifying that the build contains this PR -
{code:bash}
$ git log HEAD...HEAD~4 > git-log-head.log
{code}
Output - [^git-log-head.log]
Building -
{code:bash}
$ mvn package -Dhttps.protocols=TLSv1.2 -DskipTests -Pnative,dist
-Drequire.fuse -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop -Pyarn-ui -Dtar | tee
build-log.debian-10.log
{code}
Build output - [^build-log-debian-10.log]
I'm going to do a build on Ubuntu Focal and upload the logs shortly.
was (Author: gautham):
I did clean full build with my PR locally with Dockerfile_debian_10 and
verified that the build goes through.
Verifying that the build contains this PR -
{code:bash}
$ git log HEAD...HEAD~4 > git-log-head.log
{code}
Output - [^git-log-head.log]
Building -
{code:bash}
$ mvn package -Dhttps.protocols=TLSv1.2 -DskipTests -Pnative,dist
-Drequire.fuse -Drequire.openssl -Drequire.snappy -Drequire.valgrind
-Drequire.zstd -Drequire.test.libhadoop -Pyarn-ui -Dtar | tee
build-log.debian-10.log
{code}
Build output - [^build-log-debian-10.log]
I'm going to do a build on Ubuntu Focal and upload the logs shortly.
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build_failure.txt,
> git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gautham Banasandra updated HDFS-16304:
--
Attachment: build-log-debian-10.log
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build-log-debian-10.log, build_failure.txt,
> git-log-head.log, screenshot-1.png, screenshot-2.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16304) Locate OpenSSL libs for libhdfspp
[
https://issues.apache.org/jira/browse/HDFS-16304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gautham Banasandra updated HDFS-16304:
--
Attachment: git-log-head.log
> Locate OpenSSL libs for libhdfspp
> -
>
> Key: HDFS-16304
> URL: https://issues.apache.org/jira/browse/HDFS-16304
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, hdfs-client, native
>Affects Versions: 3.4.0
> Environment: Linux 5.13.0-1005-gcp #6-Ubuntu SMP Fri Oct 8 15:14:43
> UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Sangjin Lee
>Assignee: Gautham Banasandra
>Priority: Major
> Labels: pull-request-available
> Attachments: build_failure.txt, git-log-head.log, screenshot-1.png,
> screenshot-2.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The Docker build using {{start-build-env.sh}} is failing since commit
> 1032724. It can be reproduced with a minimal Ubuntu image and by running
> {{mvn clean install -Pnative -DskipTests -DskipShade
> -Dmaven.javadoc.skip=true}} inside the Docker container via
> {{start-build-env.sh}}.
> The regression was introduced by HDFS-16300 where it removed the line
> {{find_library(OPENSSL_LIBRARY NAMES ${OPENSSL_NAME})}}
> I suspect the CI didn't catch it because of the base image the CI runs with?
> I'm not 100% certain how real this build issue is, but according to
> {{BUILDING.txt}}, a base Ubuntu image should be able to build Hadoop using
> {{start-build-env.sh}}, and that's now broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
