[jira] [Updated] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[ https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17381: -- Affects Version/s: 3.4.0 > Distcp of EC files should not be limited to DFS. > > > Key: HDFS-17381 > URL: https://issues.apache.org/jira/browse/HDFS-17381 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Affects Versions: 3.4.0 >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Currently EC file support in distcp is limited to DFS and the code checks if > the given FS instance is DistributedFileSystem, In Ozone, EC is supported > now, and this limitation can be removed and a general contract for any > filesystem that supports EC files should be allowed by implementing few > interfaces/methods. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[ https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17381: -- Fix Version/s: 3.5.0 > Distcp of EC files should not be limited to DFS. > > > Key: HDFS-17381 > URL: https://issues.apache.org/jira/browse/HDFS-17381 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > Currently EC file support in distcp is limited to DFS and the code checks if > the given FS instance is DistributedFileSystem, In Ozone, EC is supported > now, and this limitation can be removed and a general contract for any > filesystem that supports EC files should be allowed by implementing few > interfaces/methods. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17606) Do not require implementing CustomizedCallbackHandler
[
https://issues.apache.org/jira/browse/HDFS-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17874950#comment-17874950
]
Steve Loughran commented on HDFS-17606:
---
please tag with target versions. thanks
> Do not require implementing CustomizedCallbackHandler
> -
>
> Key: HDFS-17606
> URL: https://issues.apache.org/jira/browse/HDFS-17606
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: security
>Reporter: Tsz-wo Sze
>Assignee: Tsz-wo Sze
>Priority: Major
> Labels: pull-request-available
>
> HDFS-17576 added a CustomizedCallbackHandler interface which declares the
> following method:
> {code}
> void handleCallback(List callbacks, String name, char[] password)
> throws UnsupportedCallbackException, IOException;
> {code}
> This Jira is to allow an implementation to define the handleCallback method
> without implementing the CustomizedCallbackHandler interface. It is to avoid
> a security provider depending on the HDFS project.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17463) Support the switch StringTable Split ID feature
[ https://issues.apache.org/jira/browse/HDFS-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841321#comment-17841321 ] Steve Loughran commented on HDFS-17463: --- Sorry, HDFS is not my world. Ask on the hdfs developer list for reviewers. > Support the switch StringTable Split ID feature > --- > > Key: HDFS-17463 > URL: https://issues.apache.org/jira/browse/HDFS-17463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.2.0, 3.3.5, 3.3.3, 3.3.4 >Reporter: wangzhihui >Priority: Major > Labels: pull-request-available > Attachments: Image_struct.png, error.png > > > desc: > * > Hadoop 3.2 introduced optimization features for HDFS StringTable > (b60ca37914b22550e3630fa02742d40697decb3), It resulted in lower versions of > Hadoop upgraded to 3.2 and later versions not supporting downgrade > operations. > !error.png! > * This issue has also been discussed in HDFS-14831, and it is recommended to > revert the feature, but it cannot fundamentally solve the problem。 > * > Therefore, we have added an optimization to support downgrading > > Solution: > * First, we will add the "dfs. image. save. splitId. stringTable" conf > switch "StringTable optimization feature" is enabled > * When the conf value is false, an Image file compatible with lower versions > of HDFS is generated to support downgrading. > * > The difference in HDFS Image file format between Hadoop 3.1.1 and Hadoop 3.2 > is shown in the following figure. > * With the sub-sections feature introduced in HDFS-14617, Protobuf can > support compatible reading. > * > The data structure causing incompatible differences is mainly StringTable. > !Image_struct.png|width=396,height=163! > * In "dfs.image.save.splitId.stringTable = false " the Id generation order > of StringTable starts from 0 to Integer.Max. When true, the Id value range > follows the latest rules. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17382) Add Apache Log4j Extras Library to Hadoop 3.3 for Enhanced Log Rolling Capabilities
[ https://issues.apache.org/jira/browse/HDFS-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820154#comment-17820154 ] Steve Loughran commented on HDFS-17382: --- we are still shipping reload4j with 3.4.x, so perhaps you should target branch-3.4 first > Add Apache Log4j Extras Library to Hadoop 3.3 for Enhanced Log Rolling > Capabilities > --- > > Key: HDFS-17382 > URL: https://issues.apache.org/jira/browse/HDFS-17382 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: woosuk.ro >Priority: Minor > Labels: pull-request-available > > In the current Hadoop 3.3 version, the system relies on Log4j 1.x for logging > purposes. This dependency limits the logging functionality, especially when > it comes to rolling log files such as 'hdfs-audit.log'. Rolling log files is > crucial for managing log size and ensuring logs are rotated out over time to > prevent excessive disk space usage. However, the Log4j 1.x version integrated > within Hadoop lacks the necessary capabilities to efficiently handle log > rolling. > This library extends logging capabilities, including more flexible and > configurable log rolling features. By deploying this library, we can enable > advanced rolling strategies such as time-based rolling, size-based rolling, > and compression of rolled logs, which are not supported by the default Log4j > 1.x setup in Hadoop. > The integration of Apache Log4j Extras into Hadoop will significantly improve > log management by allowing for more sophisticated and configurable log > rotation policies. This enhancement is particularly important for maintaining > system performance and reliability, as well as for compliance with log > retention policies. > Although there are plans to upgrade to Log4j 2 in the forthcoming Hadoop 3.5 > version, which will inherently solve these issues by providing enhanced > logging features, there is an immediate need to enable advanced log rolling > capabilities in the current and previous versions of Hadoop. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17382) Add Apache Log4j Extras Library to Hadoop 3.3 for Enhanced Log Rolling Capabilities
[ https://issues.apache.org/jira/browse/HDFS-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17382: -- Affects Version/s: 3.3.6 (was: 3.3.4) > Add Apache Log4j Extras Library to Hadoop 3.3 for Enhanced Log Rolling > Capabilities > --- > > Key: HDFS-17382 > URL: https://issues.apache.org/jira/browse/HDFS-17382 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.6 >Reporter: woosuk.ro >Priority: Minor > Labels: pull-request-available > > In the current Hadoop 3.3 version, the system relies on Log4j 1.x for logging > purposes. This dependency limits the logging functionality, especially when > it comes to rolling log files such as 'hdfs-audit.log'. Rolling log files is > crucial for managing log size and ensuring logs are rotated out over time to > prevent excessive disk space usage. However, the Log4j 1.x version integrated > within Hadoop lacks the necessary capabilities to efficiently handle log > rolling. > This library extends logging capabilities, including more flexible and > configurable log rolling features. By deploying this library, we can enable > advanced rolling strategies such as time-based rolling, size-based rolling, > and compression of rolled logs, which are not supported by the default Log4j > 1.x setup in Hadoop. > The integration of Apache Log4j Extras into Hadoop will significantly improve > log management by allowing for more sophisticated and configurable log > rotation policies. This enhancement is particularly important for maintaining > system performance and reliability, as well as for compliance with log > retention policies. > Although there are plans to upgrade to Log4j 2 in the forthcoming Hadoop 3.5 > version, which will inherently solve these issues by providing enhanced > logging features, there is an immediate need to enable advanced log rolling > capabilities in the current and previous versions of Hadoop. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17382) Add Apache Log4j Extras Library to Hadoop 3.3 for Enhanced Log Rolling Capabilities
[ https://issues.apache.org/jira/browse/HDFS-17382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17382: -- Target Version/s: (was: 3.3.4) > Add Apache Log4j Extras Library to Hadoop 3.3 for Enhanced Log Rolling > Capabilities > --- > > Key: HDFS-17382 > URL: https://issues.apache.org/jira/browse/HDFS-17382 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: woosuk.ro >Priority: Minor > Labels: pull-request-available > > In the current Hadoop 3.3 version, the system relies on Log4j 1.x for logging > purposes. This dependency limits the logging functionality, especially when > it comes to rolling log files such as 'hdfs-audit.log'. Rolling log files is > crucial for managing log size and ensuring logs are rotated out over time to > prevent excessive disk space usage. However, the Log4j 1.x version integrated > within Hadoop lacks the necessary capabilities to efficiently handle log > rolling. > This library extends logging capabilities, including more flexible and > configurable log rolling features. By deploying this library, we can enable > advanced rolling strategies such as time-based rolling, size-based rolling, > and compression of rolled logs, which are not supported by the default Log4j > 1.x setup in Hadoop. > The integration of Apache Log4j Extras into Hadoop will significantly improve > log management by allowing for more sophisticated and configurable log > rotation policies. This enhancement is particularly important for maintaining > system performance and reliability, as well as for compliance with log > retention policies. > Although there are plans to upgrade to Log4j 2 in the forthcoming Hadoop 3.5 > version, which will inherently solve these issues by providing enhanced > logging features, there is an immediate need to enable advanced log rolling > capabilities in the current and previous versions of Hadoop. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17316) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HDFS-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819271#comment-17819271
]
Steve Loughran commented on HDFS-17316:
---
* I think it should be a hadoop one for more than hdfs
* hdfs/webhdfs work well as unit tests for the functionality
* but can/should also target other stores, with s3a and abfs connectors key
ones for me.
one thing with the contract tests is we need the ability to declare when a
store doesn't quite meet expectations. s3a fs lets you create files under files
if you try hard; some operations raise different exceptions, permissions may be
different. so a design which allows for downgrading is critical
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HDFS-17316
> URL: https://issues.apache.org/jira/browse/HDFS-17316
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17377) Long Standing High Risk CVE in Hadoop
[
https://issues.apache.org/jira/browse/HDFS-17377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816588#comment-17816588
]
Steve Loughran commented on HDFS-17377:
---
[~prathapsagars] thank you very much for doing this against 3.4.0. we've
actually just done a release of the hadoop-third-party jar for the forthcoming
3.4.0 RC2 (this week!), with a goal of a future backport to 3.3.9.
it's list of artifacts is
https://github.com/apache/hadoop-thirdparty/blob/trunk/LICENSE-binary#L205
{code}
com.google.guava:guava:jar:32.0.1-jre
com.google.j2objc:j2objc-annotations:1.3
com.google.errorprone:error_prone_annotations:2.5.1
org.apache.avro:avro:1.11.3
com.google.protobuf:protobuf-java:3.21.12
org.checkerframework:checker-qual:jar:3.8.0
{code}
Regarding others, this is an eternal source of pain; i've tried to document it
before: https://steveloughran.blogspot.com/2022/08/transitive-issues.html
we have (especially pjfanning has, to give credit) put a lot of effort to try
and upgrade things without breaking downstream applications. We've had to do
things with reflection, build our own releases of dependencies to move off
jackson 1.x and more. I also think I'd never to want to use Guava in a project
ever again. We've actually been re-implenting bits of it (Preconditions, sets)
or wrapping it (rate limiting) to isolate it better.
{code}
./share/hadoop/tools/lib/wildfly-openssl-1.1.3.Final.jar
./share/hadoop/tools/lib/kafka-clients-2.8.2.jar
./share/hadoop/tools/lib/ini4j-0.5.4.jar
./share/hadoop/tools/lib/aliyun-java-sdk-kms-2.11.0.jar
./share/hadoop/tools/lib/aliyun-sdk-oss-3.13.2.jar
./share/hadoop/tools/lib/opentracing-api-0.33.0.jar
./share/hadoop/tools/lib/hadoop-federation-balance-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-minicluster-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/azure-data-lake-store-sdk-2.3.9.jar
./share/hadoop/tools/lib/jdk.tools-1.8.jar
./share/hadoop/tools/lib/hadoop-rumen-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-datajoin-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/aliyun-java-sdk-ram-3.1.0.jar
./share/hadoop/tools/lib/hadoop-dynamometer-blockgen-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/azure-keyvault-core-1.0.0.jar
./share/hadoop/tools/lib/junit-4.13.2.jar
./share/hadoop/tools/lib/hadoop-sls-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-resourceestimator-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/opentracing-noop-0.33.0.jar
./share/hadoop/tools/lib/hadoop-archive-logs-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/org.jacoco.agent-0.8.5-runtime.jar
./share/hadoop/tools/lib/hadoop-fs2img-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-gridmix-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/lz4-java-1.7.1.jar
./share/hadoop/tools/lib/hadoop-distcp-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-azure-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-dynamometer-infra-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-archives-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/azure-storage-7.0.1.jar
./share/hadoop/tools/lib/hamcrest-core-1.3.jar
./share/hadoop/tools/lib/bundle-2.23.19.jar
./share/hadoop/tools/lib/hadoop-streaming-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-azure-datalake-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-client-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/opentracing-util-0.33.0.jar
./share/hadoop/tools/lib/hadoop-kafka-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/aliyun-java-sdk-core-4.5.10.jar
./share/hadoop/tools/lib/hadoop-aliyun-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/zstd-jni-1.4.9-1.jar
./share/hadoop/tools/lib/jdom2-2.0.6.1.jar
./share/hadoop/tools/lib/ojalgo-43.0.jar
./share/hadoop/tools/lib/hadoop-extras-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-aws-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/lib/hadoop-dynamometer-workload-3.4.1-SNAPSHOT.jar
./share/hadoop/tools/sources/hadoop-extras-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop/tools/sources/hadoop-federation-balance-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop/tools/sources/hadoop-streaming-3.4.1-SNAPSHOT-sources.jar
./share/hadoop/tools/sources/hadoop-archives-3.4.1-SNAPSHOT-sources.jar
./share/hadoop/tools/sources/hadoop-distcp-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop/tools/sources/hadoop-streaming-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop/tools/sources/hadoop-distcp-3.4.1-SNAPSHOT-sources.jar
./share/hadoop/tools/sources/hadoop-dynamometer-infra-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop/tools/sources/hadoop-dynamometer-workload-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop/tools/sources/hadoop-rumen-3.4.1-SNAPSHOT-sources.jar
./share/hadoop/tools/sources/hadoop-sls-3.4.1-SNAPSHOT-sources.jar
./share/hadoop/tools/sources/hadoop-rumen-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop/tools/sources/hadoop-dynamometer-blockgen-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop/tools/sources/hadoop-gridmix-3.4.1-SNAPSHOT-test-sources.jar
./share/hadoop
[jira] [Commented] (HDFS-17377) Long Standing High Risk CVE in Hadoop
[ https://issues.apache.org/jira/browse/HDFS-17377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816577#comment-17816577 ] Steve Loughran commented on HDFS-17377: --- how is protobuf 2.5 getting in to yarn? i thought I'd purged that. maybe it is coming in through the hbase timeline server > Long Standing High Risk CVE in Hadoop > - > > Key: HDFS-17377 > URL: https://issues.apache.org/jira/browse/HDFS-17377 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Prathap Sagar S >Priority: Major > Attachments: HADOOP_CVE_LIST.xlsx > > > Our ongoing security scans are turning up several long-standing CVEs, even in > the most recent version of Hadoop, which is making it difficult for us to use > Hadoop in our echo system. A comprehensive list of all the long-standing CVEs > and the JARs holding them is attached. I'm asking for community assistance to > address these high-risk vulnerabilities as soon as possible. > > |Vulnerability ID|Severity|Package name|Package version|Package type|Package > path|Package suggested fix| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|12.0.1|java|/hadoop-3.4.0/share/hadoop/yarn/timelineservice/lib/guava-12.0.1.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|27.0-jre|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/guava-27.0-jre.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|27.0-jre|java|/hadoop-3.4.0/share/hadoop/common/lib/guava-27.0-jre.jar|v32.0.0-android| > |CVE-2023-2976|High|com.google.guava:guava|30.1.1-jre|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-guava-1.1.1.jar|v32.0.0-android| > |CVE-2022-25647|High|com.google.code.gson:gson|2.8.5|java|/hadoop-3.4.0/share/hadoop/yarn/timelineservice/lib/hbase-shaded-gson-3.0.0.jar|v2.8.9| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-guava-1.1.1.jar|v3.16.3| > |CVE-2022-3171|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3509|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2022-3510|High|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/avro-1.9.2.jar|v1.11.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v1.11.3| > |CVE-2023-39410|High|org.apache.avro:avro|1.9.2|java|/hadoop-3.4.0/share/hadoop/common/lib/avro-1.9.2.jar|v1.11.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/client/hadoop-client-runtime-3.4.0-SNAPSHOT.jar|v3.16.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|2.5.0|java|/hadoop-3.4.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar|v3.16.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar|v3.16.3| > |CVE-2021-22570|Medium|com.google.protobuf:protobuf-java|3.7.1|java|/hadoop-3.4.0/share/hadoop/common/lib/hadoop-shaded-protobuf_3_7-1.1.1.ja
[jira] [Commented] (HDFS-17316) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HDFS-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814392#comment-17814392
]
Steve Loughran commented on HDFS-17316:
---
bq. However, only one FS instance can be supported per run. This is because
the benchmark should be simple and fast.
that's fine; what I would like is be able to identify a profile to use and have
all the settings picked up there. for the s3a test suites i can choose which
store to run against, but the other options (storage class, fips, aws roles,
...) have to be configured too. Switching from one store to another requires me
to comment out the ones not available. If we could be profile driven from the
start, i'd do a run -Dprofile=google vs -Dprofile=aws-s3express and have the
relevant profiles picked up with my configs for each of them
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HDFS-17316
> URL: https://issues.apache.org/jira/browse/HDFS-17316
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17371) Implement vectored io api
[ https://issues.apache.org/jira/browse/HDFS-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17371: -- Affects Version/s: 3.4.0 > Implement vectored io api > - > > Key: HDFS-17371 > URL: https://issues.apache.org/jira/browse/HDFS-17371 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: Jie Han >Priority: Major > > [HADOOP-18103|https://issues.apache.org/jira/browse/HADOOP-18103] introduced > a vectored io api which can significantly improve read performance, I hope > HDFS also implements it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17371) Implement vectored io api
[ https://issues.apache.org/jira/browse/HDFS-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17371: -- Component/s: dfsclient > Implement vectored io api > - > > Key: HDFS-17371 > URL: https://issues.apache.org/jira/browse/HDFS-17371 > Project: Hadoop HDFS > Issue Type: New Feature > Components: dfsclient >Affects Versions: 3.4.0 >Reporter: Jie Han >Priority: Major > > [HADOOP-18103|https://issues.apache.org/jira/browse/HADOOP-18103] introduced > a vectored io api which can significantly improve read performance, I hope > HDFS also implements it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17316) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HDFS-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811840#comment-17811840
]
Steve Loughran commented on HDFS-17316:
---
good read.
If you look at the hadoop test classes
* we always need an explicit timeout
* on abfs/wasb/s3a tests we want to support parallel test execution, where you
can run multiple JNIT threads in parallel… each path and even temporary
directories must be uniquely allocated to the given thread, which we do by
passing a thread ID down.
* In HADOOP-18508 I'm trying to support having different (local/remote) hadoop
source trees running test against the same S3 store. Goal: I can run a hadoop
public branch from one directory well debugging a different branch locally.
Something like that is needed here, given that it seems intended to run against
live HDFS clusters. Note that also brings authentication into the mix, e.g. the
option to use kinit to log in before running the tests. This is not exclusive
to HDFS either.
It will be good -if not initially supported then at least designed as a future
option- to allow me to provide a list of stores to run the tests against. This
is because four S3A testing I now have to qualify with: amazon s3, amazon s3
express, google gcs and at least one other implementation of the API. All of
which have slightly different capabilities -test process is going to need to
somehow be driven so that for the different implementation it knows which
features to test/results to expect. The current hadoop-aws/contract test design
is not up to this.
Microsoft are pushing hard at windows support. For the shell operations it
might be very good if rather than using bash/sh/zsh that python and pyunit was
the test runner, which could then invoke Windows commands as well as shall
scripts. Pyunit test report can be aggregated displayed in Jenkins, which is
another nice feature of them.
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HDFS-17316
> URL: https://issues.apache.org/jira/browse/HDFS-17316
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
> Attachments: HDFS Compatibility Benchmark Design.pdf
>
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities c
[jira] [Commented] (HDFS-17350) update AWS SDK to support S3 Access Grants in S3A
[ https://issues.apache.org/jira/browse/HDFS-17350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810512#comment-17810512 ] Steve Loughran commented on HDFS-17350: --- you aren't going to get any v2 sdk updates in 3.3 releases; v2 is only for 3.4+ and traumatic: there is no way it will ever be backported into an official 3.3.x release. HADOOP-19046. set update AWS V2 SDK to 2.23. If you need to update it there is a whole section on qualifying a test run, including manual commands and examining test log output for new warning messages. We really need this -but as 2.23 is from earlier this month, I'd be surprised if you needed to upgrade right now. > update AWS SDK to support S3 Access Grants in S3A > - > > Key: HDFS-17350 > URL: https://issues.apache.org/jira/browse/HDFS-17350 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 3.3.6 >Reporter: Jason Han >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.6 > > Original Estimate: 168h > Remaining Estimate: 168h > > In order to support S3 Access > Grants(https://aws.amazon.com/s3/features/access-grants/) in S3A, we need to > update AWS SDK in hadooop package. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17316) Compatibility Benchmark over HCFS Implementations
[
https://issues.apache.org/jira/browse/HDFS-17316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801917#comment-17801917
]
Steve Loughran commented on HDFS-17316:
---
I'd propose decoupling this from the core hadoop/ source tree so it can be
built against 3.3 and
bq. there is no formal suite to do compatibility assessment of a file system
for all such HCFS implementations. Thus, whether the functionality is well
accomplished and meets the core compatible expectations mainly relies on
service provider's own report.
# filesystem contract tests are designed to do this from junit; If your FS
implementation doesn't subclass and run these, you need to start there.
# filesystem API specification is intended to specify the API and document
where problems surface. maintenance there always welcome -and as the contract
tests are derived from it, enhancements in those tests to follow
# there's also terasort to validate commit protocols
# + distcp contract tests for its semantics
# dfsio does a lot, but needs maintenance -it only targets the clusterfs, when
really you should be able to point at cloud storage from your own computer.
extending that to take a specific target fs would be good.
# output must go into the class ant junit xml format so jenkins can present it.
We can create a new hadoop git repo for this. Do you have existing code and any
detailed specification/docs. this also allows you to add dependencies on other
things, e.g. spark.
> Compatibility Benchmark over HCFS Implementations
> -
>
> Key: HDFS-17316
> URL: https://issues.apache.org/jira/browse/HDFS-17316
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Han Liu
>Priority: Major
>
> {*}Background：{*}Hadoop-Compatible File System (HCFS) is a core conception in
> big data storage ecosystem, providing unified interfaces and generally clear
> semantics, and has become the de-factor standard for industry storage systems
> to follow and conform with. There have been a series of HCFS implementations
> in Hadoop, such as S3AFileSystem for Amazon's S3 Object Store, WASB for
> Microsoft's Azure Blob Storage and OSS connector for Alibaba Cloud Object
> Storage, and more from storage service's providers on their own.
> {*}Problems：{*}However, as indicated by introduction.md, there is no formal
> suite to do compatibility assessment of a file system for all such HCFS
> implementations. Thus, whether the functionality is well accomplished and
> meets the core compatible expectations mainly relies on service provider's
> own report. Meanwhile, Hadoop is also developing and new features are
> continuously contributing to HCFS interfaces for existing implementations to
> follow and update, in which case, Hadoop also needs a tool to quickly assess
> if these features are supported or not for a specific HCFS implementation.
> Besides, the known hadoop command line tool or hdfs shell is used to directly
> interact with a HCFS storage system, where most commands correspond to
> specific HCFS interfaces and work well. Still, there are cases that are
> complicated and may not work, like expunge command. To check such commands
> for an HCFS, we also need an approach to figure them out.
> {*}Proposal：{*}Accordingly, we propose to define a formal HCFS compatibility
> benchmark and provide corresponding tool to do the compatibility assessment
> for an HCFS storage system. The benchmark and tool should consider both HCFS
> interfaces and hdfs shell commands. Different scenarios require different
> kinds of compatibilities. For such consideration, we could define different
> suites in the benchmark.
> *Benefits:* We intend the benchmark and tool to be useful for both storage
> providers and storage users. For end users, it can be used to evalute the
> compatibility level and determine if the storage system in question is
> suitable for the required scenarios. For storage providers, it helps to
> quickly generate an objective and reliable report about core functioins of
> the storage service. As an instance, if the HCFS got a 100% on a suite named
> 'tpcds', it is demonstrated that all functions needed by a tpcds program have
> been well achieved. It is also a guide indicating how storage service
> abilities can map to HCFS interfaces, such as storage class on S3.
> Any thoughts? Comments and feedback are mostly welcomed. Thanks in advance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17286) Add UDP as a transfer protocol for HDFS
[ https://issues.apache.org/jira/browse/HDFS-17286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17801587#comment-17801587 ] Steve Loughran commented on HDFS-17286: --- jumbo frames will be critical for this...even then how much data is in the payload of every message? the more ethernet frames in a datagram, more chance of problems > Add UDP as a transfer protocol for HDFS > --- > > Key: HDFS-17286 > URL: https://issues.apache.org/jira/browse/HDFS-17286 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Xing Lin >Priority: Major > Attachments: active.png, observer.png > > > Right now, every connection in HDFS is based on RPC/IPC which is based on > TCP. Connection is re-used based on ConnectionID, which includes RpcTimeout > as part of the key to identify a connection. The consequence is if we want to > use a different rpc timeout between two hosts, this would create different > TCP connections. > A use case which motivated us to consider UDP is getHAServiceState() in > ObserverReadProxyProvider. We'd like getHAServiceState() to time out with a > much smaller timeout threshold and move to probe next Namenode. To support > this, we used an executorService and set a timeout for the task in > HDFS-17030. This implementation can be improved by using UDP to query > HAServiceState. getHAServiceState() does not have to be very reliable, as we > can always fall back to the active. > Another motivation is it seems 5~10% of RPC calls hitting our > active/observers are GetHAServiceState(). If we can move them off to the UDP > server, that can hopefully improve RPC latency. > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16740) Mini cluster test flakiness
[ https://issues.apache.org/jira/browse/HDFS-16740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16740: -- Fix Version/s: 3.3.9 > Mini cluster test flakiness > --- > > Key: HDFS-16740 > URL: https://issues.apache.org/jira/browse/HDFS-16740 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, test >Affects Versions: 3.4.0, 3.3.5 >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Mini clusters used during HDFS unit tests are reporting test failures that do > not appear to be directly related to submitted changes. The failures are the > result of either interactions between tests run in parallel, or tests which > share common disk space for tests. In all cases, the tests can be run > individually serially without any errors. Addressing this issue will > simplify future submissions by eliminating the confusion introduced by these > unrelated test failures. > We can apply lessons recently from TestRollingUpgrade, which was recently > patched to unblock a recent submission. The fixes involved changing the HDFS > configuration to use temporary disk space for each individual tests, and > using try-with-resources to ensure that clusters were shutdown cleanly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17277) Delete invalid code logic
[ https://issues.apache.org/jira/browse/HDFS-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793772#comment-17793772 ] Steve Loughran commented on HDFS-17277: --- # moved to HDFS # can you give the JIRA a title which covers where you are deleting invalid code. We've quite a lot of it, after all.. > Delete invalid code logic > - > > Key: HDFS-17277 > URL: https://issues.apache.org/jira/browse/HDFS-17277 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhangzhanchang >Priority: Minor > Labels: pull-request-available > > There is invalid logical processing in the namenode format process -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-17277) Delete invalid code logic
[ https://issues.apache.org/jira/browse/HDFS-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HADOOP-19002 to HDFS-17277: Key: HDFS-17277 (was: HADOOP-19002) Project: Hadoop HDFS (was: Hadoop Common) > Delete invalid code logic > - > > Key: HDFS-17277 > URL: https://issues.apache.org/jira/browse/HDFS-17277 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: zhangzhanchang >Priority: Minor > Labels: pull-request-available > > There is invalid logical processing in the namenode format process -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16791) Add getEnclosingRoot() API to filesystem interface and implementations
[ https://issues.apache.org/jira/browse/HDFS-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16791: -- Fix Version/s: 3.3.9 > Add getEnclosingRoot() API to filesystem interface and implementations > -- > > Key: HDFS-16791 > URL: https://issues.apache.org/jira/browse/HDFS-16791 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.6 >Reporter: Tom McCormick >Assignee: Tom McCormick >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > At LinkedIn we run many HDFS volumes that are federated by either > ViewFilesystem or Router Based Federation. As our number of hdfs volumes > grows, we have a growing need to migrate data seemlessly across volumes. > Many frameworks have a notion of staging or temp directories, but those > directories often live in random locations. We want an API getEnclosingRoot, > which provides the root path a file or dataset. > In ViewFilesystem / Router Based Federation, the enclosingRoot will be the > mount point. > We will also take into account other restrictions for renames like > encryptions zones. > If there are several paths (a mount point and an encryption zone), we will > return the longer path -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-17249) Fix TestDFSUtil.testIsValidName() unit test failure
[
https://issues.apache.org/jira/browse/HDFS-17249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HDFS-17249.
---
Fix Version/s: 3.4.0
Resolution: Fixed
> Fix TestDFSUtil.testIsValidName() unit test failure
> ---
>
> Key: HDFS-17249
> URL: https://issues.apache.org/jira/browse/HDFS-17249
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuguanghua
>Assignee: liuguanghua
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> TestDFSUtil.testIsValidName runs failed when
> assertFalse(DFSUtil.isValidName("/foo/:/bar")) , fixed it.
> Add test case in TestDFSUtil.testIsValidName.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16791) Add getEnclosingRoot() API to filesystem interface and implementations
[ https://issues.apache.org/jira/browse/HDFS-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16791: -- Component/s: fs > Add getEnclosingRoot() API to filesystem interface and implementations > -- > > Key: HDFS-16791 > URL: https://issues.apache.org/jira/browse/HDFS-16791 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Reporter: Tom McCormick >Assignee: Tom McCormick >Priority: Major > Labels: pull-request-available > > At LinkedIn we run many HDFS volumes that are federated by either > ViewFilesystem or Router Based Federation. As our number of hdfs volumes > grows, we have a growing need to migrate data seemlessly across volumes. > Many frameworks have a notion of staging or temp directories, but those > directories often live in random locations. We want an API getEnclosingRoot, > which provides the root path a file or dataset. > In ViewFilesystem / Router Based Federation, the enclosingRoot will be the > mount point. > We will also take into account other restrictions for renames like > encryptions zones. > If there are several paths (a mount point and an encryption zone), we will > return the longer path -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16791) Add getEnclosingRoot() API to filesystem interface and implementations
[ https://issues.apache.org/jira/browse/HDFS-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HDFS-16791. --- Resolution: Fixed > Add getEnclosingRoot() API to filesystem interface and implementations > -- > > Key: HDFS-16791 > URL: https://issues.apache.org/jira/browse/HDFS-16791 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.6 >Reporter: Tom McCormick >Assignee: Tom McCormick >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > At LinkedIn we run many HDFS volumes that are federated by either > ViewFilesystem or Router Based Federation. As our number of hdfs volumes > grows, we have a growing need to migrate data seemlessly across volumes. > Many frameworks have a notion of staging or temp directories, but those > directories often live in random locations. We want an API getEnclosingRoot, > which provides the root path a file or dataset. > In ViewFilesystem / Router Based Federation, the enclosingRoot will be the > mount point. > We will also take into account other restrictions for renames like > encryptions zones. > If there are several paths (a mount point and an encryption zone), we will > return the longer path -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16791) Add getEnclosingRoot() API to filesystem interface and implementations
[ https://issues.apache.org/jira/browse/HDFS-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16791: -- Affects Version/s: 3.3.6 > Add getEnclosingRoot() API to filesystem interface and implementations > -- > > Key: HDFS-16791 > URL: https://issues.apache.org/jira/browse/HDFS-16791 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.6 >Reporter: Tom McCormick >Assignee: Tom McCormick >Priority: Major > Labels: pull-request-available > > At LinkedIn we run many HDFS volumes that are federated by either > ViewFilesystem or Router Based Federation. As our number of hdfs volumes > grows, we have a growing need to migrate data seemlessly across volumes. > Many frameworks have a notion of staging or temp directories, but those > directories often live in random locations. We want an API getEnclosingRoot, > which provides the root path a file or dataset. > In ViewFilesystem / Router Based Federation, the enclosingRoot will be the > mount point. > We will also take into account other restrictions for renames like > encryptions zones. > If there are several paths (a mount point and an encryption zone), we will > return the longer path -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16791) Add getEnclosingRoot() API to filesystem interface and implementations
[ https://issues.apache.org/jira/browse/HDFS-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16791: -- Summary: Add getEnclosingRoot() API to filesystem interface and implementations (was: New API for enclosing root path for a file) > Add getEnclosingRoot() API to filesystem interface and implementations > -- > > Key: HDFS-16791 > URL: https://issues.apache.org/jira/browse/HDFS-16791 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Tom McCormick >Assignee: Tom McCormick >Priority: Major > Labels: pull-request-available > > At LinkedIn we run many HDFS volumes that are federated by either > ViewFilesystem or Router Based Federation. As our number of hdfs volumes > grows, we have a growing need to migrate data seemlessly across volumes. > Many frameworks have a notion of staging or temp directories, but those > directories often live in random locations. We want an API getEnclosingRoot, > which provides the root path a file or dataset. > In ViewFilesystem / Router Based Federation, the enclosingRoot will be the > mount point. > We will also take into account other restrictions for renames like > encryptions zones. > If there are several paths (a mount point and an encryption zone), we will > return the longer path -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16791) Add getEnclosingRoot() API to filesystem interface and implementations
[ https://issues.apache.org/jira/browse/HDFS-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16791: -- Fix Version/s: 3.4.0 > Add getEnclosingRoot() API to filesystem interface and implementations > -- > > Key: HDFS-16791 > URL: https://issues.apache.org/jira/browse/HDFS-16791 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Affects Versions: 3.3.6 >Reporter: Tom McCormick >Assignee: Tom McCormick >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > At LinkedIn we run many HDFS volumes that are federated by either > ViewFilesystem or Router Based Federation. As our number of hdfs volumes > grows, we have a growing need to migrate data seemlessly across volumes. > Many frameworks have a notion of staging or temp directories, but those > directories often live in random locations. We want an API getEnclosingRoot, > which provides the root path a file or dataset. > In ViewFilesystem / Router Based Federation, the enclosingRoot will be the > mount point. > We will also take into account other restrictions for renames like > encryptions zones. > If there are several paths (a mount point and an encryption zone), we will > return the longer path -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17225) Fix TestNameNodeMXBean#testDecommissioningNodes
[
https://issues.apache.org/jira/browse/HDFS-17225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776332#comment-17776332
]
Steve Loughran commented on HDFS-17225:
---
dont forget to add some error text to the assertion, I do like them -and
modifying an assert is the time to do this
> Fix TestNameNodeMXBean#testDecommissioningNodes
> ---
>
> Key: HDFS-17225
> URL: https://issues.apache.org/jira/browse/HDFS-17225
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ayush Saxena
>Priority: Major
>
> Fails in assertion
> {noformat}
> org.junit.ComparisonFailure: expected:<...commissionDuration":[2]}}> but
> was:<...commissionDuration":[1]}}>
> at org.junit.Assert.assertEquals(Assert.java:117)
> at org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean.testDecommissioningNodes(TestNameNodeMXBean.java:432){noformat}
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6185/1/testReport/org.apache.hadoop.hdfs.server.namenode/TestNameNodeMXBean/testDecommissioningNodes/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17224) TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing intermittently
[ https://issues.apache.org/jira/browse/HDFS-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775028#comment-17775028 ] Steve Loughran commented on HDFS-17224: --- good analysis btw > TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing intermittently > > > Key: HDFS-17224 > URL: https://issues.apache.org/jira/browse/HDFS-17224 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsadmin, test >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing because the > static mbean isn't null. This is inevitably related to the fact that in test > runs, the jvm is reused and so the mbean may be present from a previous test > -maybe one which didn't clean up. > it does not fail standalone -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17224) TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing intermittently
[ https://issues.apache.org/jira/browse/HDFS-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17775027#comment-17775027 ] Steve Loughran commented on HDFS-17224: --- is hdfs doing its tests in the same process? aws module parallel tests have a pool of jvms but then execute each junit suite sequentially within the pool, so contamination is generally limited to cached fs instances or again, some other static state. doing pre-emptive cleanup is good. had to do one with the latest aws uploads test where it didn't clean up pending uploads from the previous run if that stopped partway through, which then broke the next > TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing intermittently > > > Key: HDFS-17224 > URL: https://issues.apache.org/jira/browse/HDFS-17224 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsadmin, test >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing because the > static mbean isn't null. This is inevitably related to the fact that in test > runs, the jvm is reused and so the mbean may be present from a previous test > -maybe one which didn't clean up. > it does not fail standalone -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17224) TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing intermittently
[ https://issues.apache.org/jira/browse/HDFS-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-17224: -- Description: TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing because the static mbean isn't null. This is inevitably related to the fact that in test runs, the jvm is reused and so the mbean may be present from a previous test -maybe one which didn't clean up. it does not fail standalone was:TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing because the static mbean isn't null. This is inevitably related to the fact that in test runs, the jvm is reused and so the mbean may be present from a previous test -maybe one which didn't clean up. Summary: TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing intermittently (was: TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing) > TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing intermittently > > > Key: HDFS-17224 > URL: https://issues.apache.org/jira/browse/HDFS-17224 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsadmin, test >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Priority: Major > > TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing because the > static mbean isn't null. This is inevitably related to the fact that in test > runs, the jvm is reused and so the mbean may be present from a previous test > -maybe one which didn't clean up. > it does not fail standalone -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17224) TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing
[
https://issues.apache.org/jira/browse/HDFS-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17774890#comment-17774890
]
Steve Loughran commented on HDFS-17224:
---
{code}
java.lang.AssertionError: expected null, but
was:
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotNull(Assert.java:756)
at org.junit.Assert.assertNull(Assert.java:738)
at org.junit.Assert.assertNull(Assert.java:748)
at
org.apache.hadoop.hdfs.TestRollingUpgrade.checkMxBeanIsNull(TestRollingUpgrade.java:326)
at
org.apache.hadoop.hdfs.TestRollingUpgrade.testDFSAdminRollingUpgradeCommands(TestRollingUpgrade.java:134)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Standard Output
2023-10-10 04:09:39,478 [main] INFO hdfs.MiniDFSCluster
(MiniDFSCluster.java:(566)) - starting cluster: numNameNodes=1,
numDataNodes=0
2023-10-10 04:09:39,482 [main] INFO namenode.NameNode
(NameNode.java:format(1377)) - Formatting using clusterid: testClusterID
2023-10-10 04:09:39,482 [main] INFO namenode.FSEditLog
(FSEditLog.java:newInstance(238)) - Edit logging is async:true
2023-10-10 04:09:39,490 [main] INFO namenode.FSNamesystem
(FSNamesystem.java:(869)) - KeyProvider: null
2023-10-10 04:09:39,490 [main] INFO namenode.FSNamesystem
(FSNamesystemLock.java:(142)) - fsLock is fair: true
2023-10-10 04:09:39,490 [main] INFO namenode.FSNamesystem
(FSNamesystemLock.java:(160)) - Detailed lock hold time metrics enabled:
false
2023-10-10 04:09:39,490 [main] INFO namenode.FSNamesystem
(FSNamesystem.java:(904)) - fsOwner= jenkins (auth:SIMPLE)
2023-10-10 04:09:39,490 [main] INFO namenode.FSNamesystem
(FSNamesystem.java:(905)) - supergroup = supergroup
2023-10-10 04:09:39,491 [main] INFO namenode.FSNamesystem
(FSNamesystem.java:(906)) - isPermissionEnabled= true
2023-10-10 04:09:39,491 [main] INFO namenode.FSNamesystem
(FSNamesystem.java:(907)) - isStoragePolicyEnabled = true
2023-10-10 04:09:39,491 [main] INFO namenode.FSNamesystem
(FSNamesystem.java:(918)) - HA Enabled: false
2023-10-10 04:09:39,491 [main] INFO common.Util
(Util.java:isDiskStatsEnabled(428)) -
dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO
profiling
2023-10-10 04:09:39,491 [main] INFO blockmanagement.DatanodeManager
(DatanodeManager.java:stopSlowPeerCollector(412)) - Slow peers collection
thread shutdown
2023-10-10 04:09:39,492 [main] IN
[jira] [Created] (HDFS-17224) TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing
Steve Loughran created HDFS-17224: - Summary: TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing Key: HDFS-17224 URL: https://issues.apache.org/jira/browse/HDFS-17224 Project: Hadoop HDFS Issue Type: Bug Components: dfsadmin, test Affects Versions: 3.4.0 Reporter: Steve Loughran TestRollingUpgrade.testDFSAdminRollingUpgradeCommands failing because the static mbean isn't null. This is inevitably related to the fact that in test runs, the jvm is reused and so the mbean may be present from a previous test -maybe one which didn't clean up. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-17212) use http simple auth only for Web UIs and not for webhdfs
[ https://issues.apache.org/jira/browse/HDFS-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HADOOP-18914 to HDFS-17212: Component/s: webhdfs (was: auth) Key: HDFS-17212 (was: HADOOP-18914) Affects Version/s: 3.3.6 (was: 3.3.6) Project: Hadoop HDFS (was: Hadoop Common) > use http simple auth only for Web UIs and not for webhdfs > - > > Key: HDFS-17212 > URL: https://issues.apache.org/jira/browse/HDFS-17212 > Project: Hadoop HDFS > Issue Type: Wish > Components: webhdfs >Affects Versions: 3.3.6 >Reporter: Anup Ahire >Priority: Minor > > I see that in hadoop 3.3.6, setting hadoop.http.authentication.type to simple > sets simple auth to webhdfs unlike previous verison like 3.1 > > I have a use case where I only want to skip spnego for web ui access and not > for webhdfs. > > Is this something we can achieve? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17202) TestDFSAdmin.testAllDatanodesReconfig assertion failing (again)
Steve Loughran created HDFS-17202: - Summary: TestDFSAdmin.testAllDatanodesReconfig assertion failing (again) Key: HDFS-17202 URL: https://issues.apache.org/jira/browse/HDFS-17202 Project: Hadoop HDFS Issue Type: Bug Components: dfsadmin, test Affects Versions: 3.3.9 Reporter: Steve Loughran surfacing in the test run for HADOOP-18895 pr https://github.com/apache/hadoop/pull/6073 ``` Expecting: <["started at Thu Sep 14 23:14:07 GMT 2023SUCCESS: Changed property dfs.datanode.peer.stats.enabled", " From: "false"", " To: "true"", " and finished at Thu Sep 14 23:14:07 GMT 2023."]> to contain subsequence: <["SUCCESS: Changed property dfs.datanode.peer.stats.enabled", " From: "false"", " To: "true""]> ``` looks like some logging race condition again as the "started at" output is on the same line as "SUCCESS". maybe something needs to add a \n after the started message, or before SUCCESS> -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-17183) HDFS defaults tls cipher to "no encryption" when keystore key is unset or empty
[ https://issues.apache.org/jira/browse/HDFS-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HADOOP-18882 to HDFS-17183: Component/s: security (was: security) Key: HDFS-17183 (was: HADOOP-18882) Affects Version/s: 3.3.4 (was: 3.3.4) Project: Hadoop HDFS (was: Hadoop Common) > HDFS defaults tls cipher to "no encryption" when keystore key is unset or > empty > --- > > Key: HDFS-17183 > URL: https://issues.apache.org/jira/browse/HDFS-17183 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 3.3.4 > Environment: We saw this issue when running in a Kubernetes > environment. > Hadoop was deployed using the [Stackable Operator for Apache > Hadoop|[https://github.com/stackabletech/hdfs-operator|http://example.com/]]. > The binaries contained in the deployed images are taken from the ASF mirrors, > not self-compiled. >Reporter: Sönke Liebau >Priority: Major > > It looks like some hdfs servers default the cipher suite to not encrypt > traffic when the keystore password is not set or set to an empty string. > Historically this has probably not often been an issue as java `keytool` > refuses to create a keystore with less than 6 characters, so usually people > would need to set passwords on the keystores (and hence in the config). > When using keystores without a password, we noticed that HDFS refuses to load > keys from this keystore when `ssl.server.keystore.password` is unset or set > to an empty string - and instead of erroring out sets the cipher suite for > rpc connections to `TLS_NULL_WITH_NULL_NULL` which is basically TLS but > without any encryption. > The impact varies depending on which communication channel we looked at, what > we saw was: > * JournalNodes seem to happily go along with this and NameNodes equally > happily connect to the JournalNodes without any warnings - we do have tls > enabled after all :) > * NameNodes refuse connections with a handshake exception, so the real world > impact of this should hopefully be small, but it does seem like less than > ideal behavior. > > So effectively, HDFS cannot use keystores without passwords, as connections > cannot be established successfully. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13916) Distcp SnapshotDiff to support WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-13916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-13916: -- Fix Version/s: 3.3.9 > Distcp SnapshotDiff to support WebHDFS > -- > > Key: HDFS-13916 > URL: https://issues.apache.org/jira/browse/HDFS-13916 > Project: Hadoop HDFS > Issue Type: New Feature > Components: distcp, webhdfs >Affects Versions: 3.0.1, 3.1.1 >Reporter: Xun REN >Assignee: Xun REN >Priority: Major > Labels: easyfix, newbie, patch, pull-request-available > Fix For: 3.4.0, 3.3.9 > > Attachments: HDFS-13916.002.patch, HDFS-13916.003.patch, > HDFS-13916.004.patch, HDFS-13916.005.patch, HDFS-13916.006.patch, > HDFS-13916.007.patch, HDFS-13916.patch > > > [~ljain] has worked on the JIRA: HDFS-13052 to provide the possibility to > make DistCP of SnapshotDiff with WebHDFSFileSystem. However, in the patch, > there is no modification for the real java class which is used by launching > the command "hadoop distcp ..." > > You can check in the latest version here: > [https://github.com/apache/hadoop/blob/branch-3.1.1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpSync.java#L96-L100] > In the method "preSyncCheck" of the class "DistCpSync", we still check if the > file system is DFS. > So I propose to change the class DistCpSync in order to take into > consideration what was committed by Lokesh Jain. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17039) CSV output support for NNThroughputBenchmark
[ https://issues.apache.org/jira/browse/HDFS-17039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729684#comment-17729684 ] Steve Loughran commented on HDFS-17039: --- there's a CSV writer in hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/Csvout.java > CSV output support for NNThroughputBenchmark > > > Key: HDFS-17039 > URL: https://issues.apache.org/jira/browse/HDFS-17039 > Project: Hadoop HDFS > Issue Type: Improvement > Components: benchmarks, test >Reporter: Mark Bukhner >Priority: Minor > > Now the NNThroughputBenchmark has only console result output, i use this > benchmark a lot and it's more convenient to have ability to write benchmark > results in csv file. > This change add BenchmarkOutputWriter class, which prints any benchmark > results in csv file, this entity could work with any benchmark. To implement > this feature in benchmark, @Parameter annotation was added in benchmark > fields. > I think this option very helpful for community -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17024) Potential data race introduced by HDFS-15865
[
https://issues.apache.org/jira/browse/HDFS-17024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17725723#comment-17725723
]
Steve Loughran commented on HDFS-17024:
---
+version, component...
> Potential data race introduced by HDFS-15865
>
>
> Key: HDFS-17024
> URL: https://issues.apache.org/jira/browse/HDFS-17024
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Labels: pull-request-available
>
> After HDFS-15865, we found client aborted due to an NPE.
> {noformat}
> 2023-04-10 16:07:43,409 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: * ABORTING region
> server kqhdp36,16020,1678077077562: Replay of WAL required. Forcing server
> shutdown *
> org.apache.hadoop.hbase.DroppedSnapshotException: region: WAFER_ALL,16|CM
> RIE.MA1|CP1114561.18|PROC|,1625899466315.0fbdf0f1810efa9e68af831247e6555f.
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2870)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2539)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2511)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2401)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:613)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:582)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:362)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.DataStreamer.waitForAckedSeqno(DataStreamer.java:880)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:781)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:898)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:850)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:76)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:105)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishClose(HFileWriterImpl.java:859)
> at
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:687)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:393)
> at
> org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:69)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:78)
> at
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:1047)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2349)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2806)
> {noformat}
> This is only possible if a data race happened. File this jira to improve the
> data and eliminate the data race.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-17006) Compute correct checksum type when file is empty/ file length is 0
[ https://issues.apache.org/jira/browse/HDFS-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HADOOP-18735 to HDFS-17006: Component/s: dfsclient (was: common) Key: HDFS-17006 (was: HADOOP-18735) Project: Hadoop HDFS (was: Hadoop Common) > Compute correct checksum type when file is empty/ file length is 0 > -- > > Key: HDFS-17006 > URL: https://issues.apache.org/jira/browse/HDFS-17006 > Project: Hadoop HDFS > Issue Type: Bug > Components: dfsclient >Reporter: Chayanika Bhandary >Priority: Trivial > Labels: pull-request-available > > When the file is empty or the file size is 0, the checksum returned is always > MD5MD5CRC type even when the selected checksum type is COMPOSITE_CRC. > This is misleading and can create confusion while debugging. > > Contributing a fix to this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-16997) Set the locale to avoid printing useless logs.
[ https://issues.apache.org/jira/browse/HDFS-16997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HADOOP-18726 to HDFS-16997: Key: HDFS-16997 (was: HADOOP-18726) Project: Hadoop HDFS (was: Hadoop Common) > Set the locale to avoid printing useless logs. > -- > > Key: HDFS-16997 > URL: https://issues.apache.org/jira/browse/HDFS-16997 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > > In our production environment, if the hadoop process is started in a > non-English environment, many unexpected error logs will be printed. The > following is the error message printed by datanode. > ``` > 2023-05-01 09:10:50,299 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,299 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242) > at > org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258) > at java.lang.Thread.run(Thread.java:745) > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,302 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242) > at > org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258) > at java.lang.Thread.run(Thread.java:745) > 2023-05-01 09:10:50,303 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,303 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > at > sun.nio.ch.FileChan
[jira] [Updated] (HDFS-16997) Set the locale to avoid printing useless logs in BlockSender
[ https://issues.apache.org/jira/browse/HDFS-16997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16997: -- Summary: Set the locale to avoid printing useless logs in BlockSender (was: Set the locale to avoid printing useless logs.) > Set the locale to avoid printing useless logs in BlockSender > > > Key: HDFS-16997 > URL: https://issues.apache.org/jira/browse/HDFS-16997 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > > In our production environment, if the hadoop process is started in a > non-English environment, many unexpected error logs will be printed. The > following is the error message printed by datanode. > ``` > 2023-05-01 09:10:50,299 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,299 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242) > at > org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258) > at java.lang.Thread.run(Thread.java:745) > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,298 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,302 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > at > sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) > at > org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:242) > at > org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:260) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:559) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:801) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:755) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:580) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:116) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:258) > at java.lang.Thread.run(Thread.java:745) > 2023-05-01 09:10:50,303 ERROR > org.apache.hadoop.hdfs.server.datanode.FileIoProvider: error in op > transferToSocketFully : 断开的管道 > 2023-05-01 09:10:50,303 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() > exception: > java.io.IOException: 断开的管道 > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > at > sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428
[jira] [Updated] (HDFS-16936) Add baseDir option in NNThroughputBenchmark
[
https://issues.apache.org/jira/browse/HDFS-16936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16936:
--
Fix Version/s: (was: 3.4.0)
(was: 3.3.5)
> Add baseDir option in NNThroughputBenchmark
> ---
>
> Key: HDFS-16936
> URL: https://issues.apache.org/jira/browse/HDFS-16936
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Mark Bukhner
>Priority: Trivial
> Labels: pull-request-available
>
> Now it's impossible to configure directory path using by
> {*}NNThroughputBenchmark{*}.
> This improvement is helpful in *RBF* features testing.
> Example of usage:
> {code:java}
> sudo bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark
> -fs ... -op create -threads 16 -baseDir /cluster1{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16939) Fix the thread safety bug in LowRedundancyBlocks
[ https://issues.apache.org/jira/browse/HDFS-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698879#comment-17698879 ] Steve Loughran commented on HDFS-16939: --- thanks for the update > Fix the thread safety bug in LowRedundancyBlocks > > > Key: HDFS-16939 > URL: https://issues.apache.org/jira/browse/HDFS-16939 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The remove method in LowRedundancyBlocks is not protected by synchronized. > This method is private and is called by BlockManager. As a result, > priorityQueues has the risk of being accessed concurrently by multiple > threads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16939) Fix the thread safety bug in LowRedundancyBlocks
[ https://issues.apache.org/jira/browse/HDFS-16939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698344#comment-17698344 ] Steve Loughran commented on HDFS-16939: --- i'm planning to do the next 3.3.5 RC over the weekend. if you think this needs to go in, put patches on github up for branch-3.3 and 3.3.5, once they are through yetus they can both get merged in > Fix the thread safety bug in LowRedundancyBlocks > > > Key: HDFS-16939 > URL: https://issues.apache.org/jira/browse/HDFS-16939 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The remove method in LowRedundancyBlocks is not protected by synchronized. > This method is private and is called by BlockManager. As a result, > priorityQueues has the risk of being accessed concurrently by multiple > threads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16742) Test TestDirectoryScanner.testThrottling failing
[
https://issues.apache.org/jira/browse/HDFS-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17697953#comment-17697953
]
Steve Loughran commented on HDFS-16742:
---
seems to be a flaky test
> Test TestDirectoryScanner.testThrottling failing
> -
>
> Key: HDFS-16742
> URL: https://issues.apache.org/jira/browse/HDFS-16742
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Reporter: Samrat Deb
>Assignee: Samrat Deb
>Priority: Major
>
> Running unit test on testThrottling failing with assertion error
> java.lang.AssertionError: Throttle is too permissive
>
>
> {code:java}
> @Test
> public void testThrottling() throws Exception
> {
> {code}
>
>
>
>
>
> {code:java}
>
> 2022-08-25 10:43:54,593
> [org.apache.hadoop.hdfs.server.blockmanagement.PendingReconstructionBlocks$PendingReconstructionMonitor@210366b4]
> DEBUG blockmanagement.BlockManager
> (PendingReconstructionBlocks.java:run(261)) - PendingReconstructionMonitor
> thread is interrupted. java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method) at
> org.apache.hadoop.hdfs.server.blockmanagement.PendingReconstructionBlocks$PendingReconstructionMonitor.run(PendingReconstructionBlocks.java:259)
> at java.lang.Thread.run(Thread.java:750) 2022-08-25 10:43:54,596
> [Listener at localhost/55920] INFO namenode.FSNamesystem
> (FSNamesystem.java:stopActiveServices(1499)) - Stopping services started for
> active state 2022-08-25 10:43:54,596 [Listener at localhost/55920] INFO
> namenode.FSNamesystem (FSNamesystem.java:stopStandbyServices(1603)) -
> Stopping services started for standby state 2022-08-25 10:43:54,597 [Listener
> at localhost/55920] INFO handler.ContextHandler
> (ContextHandler.java:doStop(1159)) - Stopped
> o.e.j.w.WebAppContext@5f9d02cb\{hdfs,/,null,STOPPED}
> {file:/CODE/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs}
> 2022-08-25 10:43:54,598 [Listener at localhost/55920] INFO
> server.AbstractConnector (AbstractConnector.java:doStop(383)) - Stopped
> ServerConnector@544fe44c{HTTP/1.1, (http/1.1)}
> {localhost:0}
> 2022-08-25 10:43:54,598 [Listener at localhost/55920] INFO server.session
> (HouseKeeper.java:stopScavenging(149)) - node0 Stopped scavenging
> 2022-08-25 10:43:54,598 [Listener at localhost/55920] INFO
> handler.ContextHandler (ContextHandler.java:doStop(1159)) - Stopped
> o.e.j.s.ServletContextHandler@6e1ec318{static,/static,file:///CODE/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/,STOPPED}
> 2022-08-25 10:43:54,598 [Listener at localhost/55920] INFO
> handler.ContextHandler (ContextHandler.java:doStop(1159)) - Stopped
> o.e.j.s.ServletContextHandler@dc24521{logs,/logs,file:///CODE/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/log/,STOPPED}
> 2022-08-25 10:43:54,599 [Listener at localhost/55920] INFO
> impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping DataNode
> metrics system...
> 2022-08-25 10:43:54,599 [Listener at localhost/55920] INFO
> impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - DataNode metrics
> system stopped.
> 2022-08-25 10:43:54,599 [Listener at localhost/55920] INFO
> impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(611)) - DataNode
> metrics system shutdown complete.
> java.lang.AssertionError: Throttle is too permissive6.0234113
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16934) org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression
[
https://issues.apache.org/jira/browse/HDFS-16934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HDFS-16934.
---
Fix Version/s: 3.4.0
3.3.5
Resolution: Fixed
fixed; ran new test on 3.3.5 to verify the backport/conflict resolution was ok.
> org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression
> -
>
> Key: HDFS-16934
> URL: https://issues.apache.org/jira/browse/HDFS-16934
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsadmin, test
>Affects Versions: 3.4.0, 3.3.5, 3.3.9
>Reporter: Steve Loughran
>Assignee: Shilun Fan
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5
>
>
> jenkins test failure as the logged output is in the wrong order for the
> assertions. HDFS-16624 flipped the order...without that this would have
> worked.
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1149)
> {code}
> Here the code is asserting about the contents of the output,
> {code}
> assertTrue(outs.get(0).startsWith("Reconfiguring status for node"));
> assertTrue("SUCCESS: Changed property
> dfs.datanode.peer.stats.enabled".equals(outs.get(2))
> || "SUCCESS: Changed property
> dfs.datanode.peer.stats.enabled".equals(outs.get(1))); // here
> assertTrue("\tFrom: \"false\"".equals(outs.get(3)) || "\tFrom:
> \"false\"".equals(outs.get(2)));
> assertTrue("\tTo: \"true\"".equals(outs.get(4)) || "\tTo:
> \"true\"".equals(outs.get(3)))
> {code}
> If you look at the log, the actual line is appearing in that list, just in a
> different place. race condition
> {code}
> 2023-02-24 01:02:06,275 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin (TestDFSAdmin.java:testAllDatanodesReconfig(1146)) -
> dfsadmin -status -livenodes output:
> 2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring
> status for node [127.0.0.1:41795]: started at Fri Feb 24 01:02:03 GMT 2023
> and finished at Fri Feb 24 01:02:03 GMT 2023.
> 2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring
> status for node [127.0.0.1:34007]: started at Fri Feb 24 01:02:03 GMT
> 2023SUCCESS: Changed property dfs.datanode.peer.stats.enabled
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - From: "false"
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - To: "true"
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - and finished
> at Fri Feb 24 01:02:03 GMT 2023.
> 2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO
> tools.TestDFSAdmin
> (TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - SUCCESS:
> Changed property dfs.datanode.peer.stats.enabled
> {code}
> we have a race condition in output generation and the assertions are clearly
> too brittle
> for the 3.3.5 release I'm not going to make this a blocker. What i will do is
> propose that the asserts move to assertJ with an assertion that the
> collection "containsExactlyInAnyOrder" all the strings.
> That will
> 1. not be brittle.
> 2. give nice errors on failure
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16941) Path.suffix raises NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17696248#comment-17696248
]
Steve Loughran commented on HDFS-16941:
---
patch with test welcome...
> Path.suffix raises NullPointerException
> ---
>
> Key: HDFS-16941
> URL: https://issues.apache.org/jira/browse/HDFS-16941
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hadoop-client, hdfs
>Affects Versions: 3.3.2
>Reporter: Patrick Grandjean
>Priority: Minor
>
> Calling the Path.suffix method on root raises a NullPointerException. Tested
> with hadoop-client-api 3.3.2
> Scenario:
> {code:java}
> import org.apache.hadoop.fs.*
> Path root = new Path("/")
> root.getParent == null // true
> root.suffix("bar") // NPE is raised
> {code}
> Stack:
> {code:none}
> 23/03/03 15:13:18 ERROR Uncaught throwable from user code:
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:104)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at org.apache.hadoop.fs.Path.suffix(Path.java:361)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16941) Path.suffix raises NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16941:
--
Priority: Minor (was: Major)
> Path.suffix raises NullPointerException
> ---
>
> Key: HDFS-16941
> URL: https://issues.apache.org/jira/browse/HDFS-16941
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hadoop-client, hdfs
>Affects Versions: 3.3.2
>Reporter: Patrick Grandjean
>Priority: Minor
>
> Calling the Path.suffix method on root raises a NullPointerException. Tested
> with hadoop-client-api 3.3.2
> Scenario:
> {code:java}
> import org.apache.hadoop.fs.*
> Path root = new Path("/")
> root.getParent == null // true
> root.suffix("bar") // NPE is raised
> {code}
> Stack:
> {code:none}
> 23/03/03 15:13:18 ERROR Uncaught throwable from user code:
> java.lang.NullPointerException
> at org.apache.hadoop.fs.Path.(Path.java:104)
> at org.apache.hadoop.fs.Path.(Path.java:93)
> at org.apache.hadoop.fs.Path.suffix(Path.java:361)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16935) TestFsDatasetImpl.testReportBadBlocks brittle
[
https://issues.apache.org/jira/browse/HDFS-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HDFS-16935.
---
Resolution: Fixed
> TestFsDatasetImpl.testReportBadBlocks brittle
> -
>
> Key: HDFS-16935
> URL: https://issues.apache.org/jira/browse/HDFS-16935
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0, 3.3.5, 3.3.9
>Reporter: Steve Loughran
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> jenkins failure as sleep() time not long enough
> {code}
> Failing for the past 1 build (Since #4 )
> Took 7.4 sec.
> Error Message
> expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:633)
> {code}
> assert is after a 3s sleep waiting for reports coming in.
> {code}
> dataNode.reportBadBlocks(block, dataNode.getFSDataset()
> .getFsVolumeReferences().get(0));
> Thread.sleep(3000); // 3s
> sleep
> BlockManagerTestUtil.updateState(cluster.getNamesystem()
> .getBlockManager());
> // Verify the bad block has been reported to namenode
> Assert.assertEquals(1,
> cluster.getNamesystem().getCorruptReplicaBlocks()); // here
> {code}
> LambdaTestUtils.eventually() should be used around this assert, maybe with an
> even shorter initial delay so on faster systems, test is faster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16935) TestFsDatasetImpl.testReportBadBlocks brittle
[
https://issues.apache.org/jira/browse/HDFS-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16935:
--
Fix Version/s: 3.4.0
3.3.9
> TestFsDatasetImpl.testReportBadBlocks brittle
> -
>
> Key: HDFS-16935
> URL: https://issues.apache.org/jira/browse/HDFS-16935
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0, 3.3.5, 3.3.9
>Reporter: Steve Loughran
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
>
> jenkins failure as sleep() time not long enough
> {code}
> Failing for the past 1 build (Since #4 )
> Took 7.4 sec.
> Error Message
> expected:<1> but was:<0>
> Stacktrace
> java.lang.AssertionError: expected:<1> but was:<0>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:633)
> {code}
> assert is after a 3s sleep waiting for reports coming in.
> {code}
> dataNode.reportBadBlocks(block, dataNode.getFSDataset()
> .getFsVolumeReferences().get(0));
> Thread.sleep(3000); // 3s
> sleep
> BlockManagerTestUtil.updateState(cluster.getNamesystem()
> .getBlockManager());
> // Verify the bad block has been reported to namenode
> Assert.assertEquals(1,
> cluster.getNamesystem().getCorruptReplicaBlocks()); // here
> {code}
> LambdaTestUtils.eventually() should be used around this assert, maybe with an
> even shorter initial delay so on faster systems, test is faster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16935) TestFsDatasetImpl.testReportBadBlocks brittle
Steve Loughran created HDFS-16935:
-
Summary: TestFsDatasetImpl.testReportBadBlocks brittle
Key: HDFS-16935
URL: https://issues.apache.org/jira/browse/HDFS-16935
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 3.4.0, 3.3.5, 3.3.9
Reporter: Steve Loughran
jenkins failure as sleep() time not long enough
{code}
Failing for the past 1 build (Since #4 )
Took 7.4 sec.
Error Message
expected:<1> but was:<0>
Stacktrace
java.lang.AssertionError: expected:<1> but was:<0>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
{code}
assert is after a 3s sleep waiting for reports coming in.
{code}
dataNode.reportBadBlocks(block, dataNode.getFSDataset()
.getFsVolumeReferences().get(0));
Thread.sleep(3000); // 3s sleep
BlockManagerTestUtil.updateState(cluster.getNamesystem()
.getBlockManager());
// Verify the bad block has been reported to namenode
Assert.assertEquals(1,
cluster.getNamesystem().getCorruptReplicaBlocks()); // here
{code}
LambdaTestUtils.eventually() should be used around this assert, maybe with an
even shorter initial delay so on faster systems, test is faster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16624) Fix org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig ERROR
[ https://issues.apache.org/jira/browse/HDFS-16624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693179#comment-17693179 ] Steve Loughran commented on HDFS-16624: --- this reordering is breaking some yetus runs, implying the output ordering is brittle. # Created HDFS-16934 # proposed: use assertJ to assert on the exact contents of the log *in any order*. [~slfan1989] you wan't to take that up? > Fix org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig ERROR > > > Key: HDFS-16624 > URL: https://issues.apache.org/jira/browse/HDFS-16624 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Attachments: testAllDatanodesReconfig.png > > Time Spent: 3h > Remaining Estimate: 0h > > HDFS-16619 found an error message during Junit unit testing, as follows: > expected:<[SUCCESS: Changed property dfs.datanode.peer.stats.enabled]> but > was:<[ From: "false"]> > After code debugging, it was found that there was an error in the selection > outs.get(2) of the assertion(1208), index should be equal to 1. > Please refer to the attachment for debugging pictures > !testAllDatanodesReconfig.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16934) org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression
Steve Loughran created HDFS-16934:
-
Summary:
org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression
Key: HDFS-16934
URL: https://issues.apache.org/jira/browse/HDFS-16934
Project: Hadoop HDFS
Issue Type: Bug
Components: dfsadmin, test
Affects Versions: 3.4.0, 3.3.5, 3.3.9
Reporter: Steve Loughran
jenkins test failure as the logged output is in the wrong order for the
assertions. HDFS-16624 flipped the order...without that this would have worked.
{code}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at
org.apache.hadoop.hdfs.tools.TestDFSAdmin.testAllDatanodesReconfig(TestDFSAdmin.java:1149)
{code}
Here the code is asserting about the contents of the output,
{code}
assertTrue(outs.get(0).startsWith("Reconfiguring status for node"));
assertTrue("SUCCESS: Changed property
dfs.datanode.peer.stats.enabled".equals(outs.get(2))
|| "SUCCESS: Changed property
dfs.datanode.peer.stats.enabled".equals(outs.get(1))); // here
assertTrue("\tFrom: \"false\"".equals(outs.get(3)) || "\tFrom:
\"false\"".equals(outs.get(2)));
assertTrue("\tTo: \"true\"".equals(outs.get(4)) || "\tTo:
\"true\"".equals(outs.get(3)))
{code}
If you look at the log, the actual line is appearing in that list, just in a
different place. race condition
{code}
2023-02-24 01:02:06,275 [Listener at localhost/41795] INFO tools.TestDFSAdmin
(TestDFSAdmin.java:testAllDatanodesReconfig(1146)) - dfsadmin -status
-livenodes output:
2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO tools.TestDFSAdmin
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring
status for node [127.0.0.1:41795]: started at Fri Feb 24 01:02:03 GMT 2023 and
finished at Fri Feb 24 01:02:03 GMT 2023.
2023-02-24 01:02:06,276 [Listener at localhost/41795] INFO tools.TestDFSAdmin
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - Reconfiguring
status for node [127.0.0.1:34007]: started at Fri Feb 24 01:02:03 GMT
2023SUCCESS: Changed property dfs.datanode.peer.stats.enabled
2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO tools.TestDFSAdmin
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) -From: "false"
2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO tools.TestDFSAdmin
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) -To: "true"
2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO tools.TestDFSAdmin
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - and finished at
Fri Feb 24 01:02:03 GMT 2023.
2023-02-24 01:02:06,277 [Listener at localhost/41795] INFO tools.TestDFSAdmin
(TestDFSAdmin.java:lambda$testAllDatanodesReconfig$0(1147)) - SUCCESS: Changed
property dfs.datanode.peer.stats.enabled
{code}
we have a race condition in output generation and the assertions are clearly
too brittle
for the 3.3.5 release I'm not going to make this a blocker. What i will do is
propose that the asserts move to assertJ with an assertion that the collection
"containsExactlyInAnyOrder" all the strings.
That will
1. not be brittle.
2. give nice errors on failure
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16453) Upgrade okhttp from 2.7.5 to 4.9.3
[
https://issues.apache.org/jira/browse/HDFS-16453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692865#comment-17692865
]
Steve Loughran commented on HDFS-16453:
---
[~chengpan] its license update too. but no, not aFAIK
> Upgrade okhttp from 2.7.5 to 4.9.3
> --
>
> Key: HDFS-16453
> URL: https://issues.apache.org/jira/browse/HDFS-16453
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.3.1
>Reporter: Ivan Viaznikov
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.4
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> {{org.apache.hadoop:hadoop-hdfs-client}} comes with
> {{com.squareup.okhttp:okhttp:2.7.5}} as a dependency, which is vulnerable to
> an information disclosure issue due to how the contents of sensitive headers,
> such as the {{Authorization}} header, can be logged when an
> {{IllegalArgumentException}} is thrown.
> This issue could allow an attacker or malicious user who has access to the
> logs to obtain the sensitive contents of the affected headers which could
> facilitate further attacks.
> Fixed in {{5.0.0-alpha3}} by
> [this|https://github.com/square/okhttp/commit/dcc6483b7dc6d9c0b8e03ff7c30c13f3c75264a5]
> commit. The fix was cherry-picked and backported into {{4.9.2}} with
> [this|https://github.com/square/okhttp/commit/1fd7c0afdc2cee9ba982b07d49662af7f60e1518]
> commit.
> Requesting you to clarify if this dependency will be updated to a fixed
> version in the following releases
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16853) The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed because HADOOP-18324
[
https://issues.apache.org/jira/browse/HDFS-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HDFS-16853.
---
Fix Version/s: 3.3.5
Resolution: Fixed
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
> because HADOOP-18324
> ---
>
> Key: HDFS-16853
> URL: https://issues.apache.org/jira/browse/HDFS-16853
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.5
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 3.3.5
>
>
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
> with error message: Waiting for cluster to become active. And the blocking
> jstack as bellows:
> {code:java}
> "BP-1618793397-192.168.3.4-1669198559828 heartbeating to
> localhost/127.0.0.1:54673" #260 daemon prio=5 os_prio=31 tid=0x
> 7fc1108fa000 nid=0x19303 waiting on condition [0x700017884000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007430a9ec0> (a
> java.util.concurrent.SynchronousQueue$TransferQueue)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.SynchronousQueue$TransferQueue.awaitFulfill(SynchronousQueue.java:762)
> at
> java.util.concurrent.SynchronousQueue$TransferQueue.transfer(SynchronousQueue.java:695)
> at
> java.util.concurrent.SynchronousQueue.put(SynchronousQueue.java:877)
> at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1186)
> at org.apache.hadoop.ipc.Client.call(Client.java:1482)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
> at com.sun.proxy.$Proxy23.sendHeartbeat(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClient
> SideTranslatorPB.java:168)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:570)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:714)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:915)
> at java.lang.Thread.run(Thread.java:748) {code}
> After looking into the code and found that this bug is imported by
> HADOOP-18324. Because RpcRequestSender exited without cleaning up the
> rpcRequestQueue, then caused BPServiceActor was blocked in sending request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16906) CryptoOutputStream::close leak when encrypted zones + quota exceptions
[
https://issues.apache.org/jira/browse/HDFS-16906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17683820#comment-17683820
]
Steve Loughran commented on HDFS-16906:
---
hello, thank you for your first submission.
* it's an HDFS bug so moved projects; someone there needs to review it
* can you submit as a PR against apache/hadoop, which adds CI testing and where
we do our reviews/merge.
* regarding level, i'd put "major" rather than critical, and it is not a
loss-of-data class of problem, and we do need to leave space for them in the
hierarchy
thanks
> CryptoOutputStream::close leak when encrypted zones + quota exceptions
> --
>
> Key: HDFS-16906
> URL: https://issues.apache.org/jira/browse/HDFS-16906
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 3.3.1, 3.3.2, 3.3.3, 3.3.4
>Reporter: Colm Dougan
>Priority: Critical
> Attachments: hadoop_cryto_stream_close_try_finally.diff
>
>
> {color:#172b4d}I would like to report an issue with a resource leak
> ({color}DFSOutputStream objects) when using the (java) hadoop-hdfs-client
> And specifically (at least in my case) when there is a combination of:
> * encrypted zones
> * quota space exceptions (DSQuotaExceededException)
> As you know, when encrypted zones are in play, when calling fs.create(path)
> in the hadoop-hdfs-client it will return a HdfsDataOutputStream stream object
> which wraps a CryptoOutputStream object which then wraps a DFSOutputStream
> object.
> Even though my code is correctly calling stream.close() on the above I can
> see from debugging that the underlying DFSOutputStream objects are being
> leaked.
> Specifically I see the DFSOutputStream objects being leaked in the
> filesBeingWritten map in DFSClient. (i.e. the DFSOutputStream objects remain
> in the map even though I've called close() on the stream object).
> I suspect this is due to a bug in CryptoOutputStream::close
> {code:java}
> @Override
>
> public synchronized void close() throws IOException {
>
> if (closed) {
>
> return;
>
> }
>
> try {
>
> flush();
>
> if (closeOutputStream) {
>
> super.close();
>
> codec.close();
>
> }
>
> freeBuffers();
>
> } finally {
>
> closed = true;
>
> }
>
> }{code}
> ... whereby if flush() throws (observed in my case when a
> DSQuotaExceededException exception is thrown due to quota exceeded) then the
> super.close() on the underlying DFSOutputStream is skipped.
> In my case I had a space quota set up on a given directory which is also in
> an encrypted zone and so each attempt to create and write to a file failed
> and leaked as above.
> I have attached a speculative patch
> ([^hadoop_cryto_stream_close_try_finally.diff]) which simply wraps the
> flush() in a try .. finally. The patch resolves the problem in my testing.
> Thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16906) CryptoOutputStream::close leak when encrypted zones + quota exceptions
[
https://issues.apache.org/jira/browse/HDFS-16906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HDFS-16906:
-
Assignee: Colm Dougan
> CryptoOutputStream::close leak when encrypted zones + quota exceptions
> --
>
> Key: HDFS-16906
> URL: https://issues.apache.org/jira/browse/HDFS-16906
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 3.3.1, 3.3.2, 3.3.3, 3.3.4
>Reporter: Colm Dougan
>Assignee: Colm Dougan
>Priority: Critical
> Attachments: hadoop_cryto_stream_close_try_finally.diff
>
>
> {color:#172b4d}I would like to report an issue with a resource leak
> ({color}DFSOutputStream objects) when using the (java) hadoop-hdfs-client
> And specifically (at least in my case) when there is a combination of:
> * encrypted zones
> * quota space exceptions (DSQuotaExceededException)
> As you know, when encrypted zones are in play, when calling fs.create(path)
> in the hadoop-hdfs-client it will return a HdfsDataOutputStream stream object
> which wraps a CryptoOutputStream object which then wraps a DFSOutputStream
> object.
> Even though my code is correctly calling stream.close() on the above I can
> see from debugging that the underlying DFSOutputStream objects are being
> leaked.
> Specifically I see the DFSOutputStream objects being leaked in the
> filesBeingWritten map in DFSClient. (i.e. the DFSOutputStream objects remain
> in the map even though I've called close() on the stream object).
> I suspect this is due to a bug in CryptoOutputStream::close
> {code:java}
> @Override
>
> public synchronized void close() throws IOException {
>
> if (closed) {
>
> return;
>
> }
>
> try {
>
> flush();
>
> if (closeOutputStream) {
>
> super.close();
>
> codec.close();
>
> }
>
> freeBuffers();
>
> } finally {
>
> closed = true;
>
> }
>
> }{code}
> ... whereby if flush() throws (observed in my case when a
> DSQuotaExceededException exception is thrown due to quota exceeded) then the
> super.close() on the underlying DFSOutputStream is skipped.
> In my case I had a space quota set up on a given directory which is also in
> an encrypted zone and so each attempt to create and write to a file failed
> and leaked as above.
> I have attached a speculative patch
> ([^hadoop_cryto_stream_close_try_finally.diff]) which simply wraps the
> flush() in a try .. finally. The patch resolves the problem in my testing.
> Thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-16906) CryptoOutputStream::close leak when encrypted zones + quota exceptions
[
https://issues.apache.org/jira/browse/HDFS-16906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran moved HADOOP-18615 to HDFS-16906:
Component/s: dfsclient
(was: common)
Key: HDFS-16906 (was: HADOOP-18615)
Affects Version/s: 3.3.4
3.3.3
3.3.2
3.3.1
(was: 3.3.1)
(was: 3.3.2)
(was: 3.3.3)
(was: 3.3.4)
Project: Hadoop HDFS (was: Hadoop Common)
> CryptoOutputStream::close leak when encrypted zones + quota exceptions
> --
>
> Key: HDFS-16906
> URL: https://issues.apache.org/jira/browse/HDFS-16906
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsclient
>Affects Versions: 3.3.4, 3.3.3, 3.3.2, 3.3.1
>Reporter: Colm Dougan
>Priority: Critical
> Attachments: hadoop_cryto_stream_close_try_finally.diff
>
>
> {color:#172b4d}I would like to report an issue with a resource leak
> ({color}DFSOutputStream objects) when using the (java) hadoop-hdfs-client
> And specifically (at least in my case) when there is a combination of:
> * encrypted zones
> * quota space exceptions (DSQuotaExceededException)
> As you know, when encrypted zones are in play, when calling fs.create(path)
> in the hadoop-hdfs-client it will return a HdfsDataOutputStream stream object
> which wraps a CryptoOutputStream object which then wraps a DFSOutputStream
> object.
> Even though my code is correctly calling stream.close() on the above I can
> see from debugging that the underlying DFSOutputStream objects are being
> leaked.
> Specifically I see the DFSOutputStream objects being leaked in the
> filesBeingWritten map in DFSClient. (i.e. the DFSOutputStream objects remain
> in the map even though I've called close() on the stream object).
> I suspect this is due to a bug in CryptoOutputStream::close
> {code:java}
> @Override
>
> public synchronized void close() throws IOException {
>
> if (closed) {
>
> return;
>
> }
>
> try {
>
> flush();
>
> if (closeOutputStream) {
>
> super.close();
>
> codec.close();
>
> }
>
> freeBuffers();
>
> } finally {
>
> closed = true;
>
> }
>
> }{code}
> ... whereby if flush() throws (observed in my case when a
> DSQuotaExceededException exception is thrown due to quota exceeded) then the
> super.close() on the underlying DFSOutputStream is skipped.
> In my case I had a space quota set up on a given directory which is also in
> an encrypted zone and so each attempt to create and write to a file failed
> and leaked as above.
> I have attached a speculative patch
> ([^hadoop_cryto_stream_close_try_finally.diff]) which simply wraps the
> flush() in a try .. finally. The patch resolves the problem in my testing.
> Thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16853) The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed because HADOOP-18324
[
https://issues.apache.org/jira/browse/HDFS-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677849#comment-17677849
]
Steve Loughran commented on HDFS-16853:
---
marking asa a blocker for 3.3.5. [~omalley] could you look at this?
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
> because HADOOP-18324
> ---
>
> Key: HDFS-16853
> URL: https://issues.apache.org/jira/browse/HDFS-16853
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.5
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Blocker
> Labels: pull-request-available
>
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
> with error message: Waiting for cluster to become active. And the blocking
> jstack as bellows:
> {code:java}
> "BP-1618793397-192.168.3.4-1669198559828 heartbeating to
> localhost/127.0.0.1:54673" #260 daemon prio=5 os_prio=31 tid=0x
> 7fc1108fa000 nid=0x19303 waiting on condition [0x700017884000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007430a9ec0> (a
> java.util.concurrent.SynchronousQueue$TransferQueue)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.SynchronousQueue$TransferQueue.awaitFulfill(SynchronousQueue.java:762)
> at
> java.util.concurrent.SynchronousQueue$TransferQueue.transfer(SynchronousQueue.java:695)
> at
> java.util.concurrent.SynchronousQueue.put(SynchronousQueue.java:877)
> at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1186)
> at org.apache.hadoop.ipc.Client.call(Client.java:1482)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
> at com.sun.proxy.$Proxy23.sendHeartbeat(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClient
> SideTranslatorPB.java:168)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:570)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:714)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:915)
> at java.lang.Thread.run(Thread.java:748) {code}
> After looking into the code and found that this bug is imported by
> HADOOP-18324. Because RpcRequestSender exited without cleaning up the
> rpcRequestQueue, then caused BPServiceActor was blocked in sending request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16853) The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed because HADOOP-18324
[
https://issues.apache.org/jira/browse/HDFS-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16853:
--
Affects Version/s: 3.3.5
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
> because HADOOP-18324
> ---
>
> Key: HDFS-16853
> URL: https://issues.apache.org/jira/browse/HDFS-16853
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.5
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Blocker
> Labels: pull-request-available
>
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
> with error message: Waiting for cluster to become active. And the blocking
> jstack as bellows:
> {code:java}
> "BP-1618793397-192.168.3.4-1669198559828 heartbeating to
> localhost/127.0.0.1:54673" #260 daemon prio=5 os_prio=31 tid=0x
> 7fc1108fa000 nid=0x19303 waiting on condition [0x700017884000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007430a9ec0> (a
> java.util.concurrent.SynchronousQueue$TransferQueue)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.SynchronousQueue$TransferQueue.awaitFulfill(SynchronousQueue.java:762)
> at
> java.util.concurrent.SynchronousQueue$TransferQueue.transfer(SynchronousQueue.java:695)
> at
> java.util.concurrent.SynchronousQueue.put(SynchronousQueue.java:877)
> at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1186)
> at org.apache.hadoop.ipc.Client.call(Client.java:1482)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
> at com.sun.proxy.$Proxy23.sendHeartbeat(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClient
> SideTranslatorPB.java:168)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:570)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:714)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:915)
> at java.lang.Thread.run(Thread.java:748) {code}
> After looking into the code and found that this bug is imported by
> HADOOP-18324. Because RpcRequestSender exited without cleaning up the
> rpcRequestQueue, then caused BPServiceActor was blocked in sending request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16853) The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed because HADOOP-18324
[
https://issues.apache.org/jira/browse/HDFS-16853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16853:
--
Priority: Blocker (was: Major)
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
> because HADOOP-18324
> ---
>
> Key: HDFS-16853
> URL: https://issues.apache.org/jira/browse/HDFS-16853
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: ZanderXu
>Assignee: ZanderXu
>Priority: Blocker
> Labels: pull-request-available
>
> The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed
> with error message: Waiting for cluster to become active. And the blocking
> jstack as bellows:
> {code:java}
> "BP-1618793397-192.168.3.4-1669198559828 heartbeating to
> localhost/127.0.0.1:54673" #260 daemon prio=5 os_prio=31 tid=0x
> 7fc1108fa000 nid=0x19303 waiting on condition [0x700017884000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x0007430a9ec0> (a
> java.util.concurrent.SynchronousQueue$TransferQueue)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.SynchronousQueue$TransferQueue.awaitFulfill(SynchronousQueue.java:762)
> at
> java.util.concurrent.SynchronousQueue$TransferQueue.transfer(SynchronousQueue.java:695)
> at
> java.util.concurrent.SynchronousQueue.put(SynchronousQueue.java:877)
> at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1186)
> at org.apache.hadoop.ipc.Client.call(Client.java:1482)
> at org.apache.hadoop.ipc.Client.call(Client.java:1429)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
> at com.sun.proxy.$Proxy23.sendHeartbeat(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClient
> SideTranslatorPB.java:168)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:570)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:714)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:915)
> at java.lang.Thread.run(Thread.java:748) {code}
> After looking into the code and found that this bug is imported by
> HADOOP-18324. Because RpcRequestSender exited without cleaning up the
> rpcRequestQueue, then caused BPServiceActor was blocked in sending request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17646092#comment-17646092
]
Steve Loughran commented on HDFS-16864:
---
although i want to do rc0 out this week, as xmas is coming it will take a while
for testing to finish and we are all having our own vacations. if others in
the hdfs team are happy it could go in now for that testing coverage
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16860) Upgrade moment.min.js to 2.29.4
[ https://issues.apache.org/jira/browse/HDFS-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645631#comment-17645631 ] Steve Loughran commented on HDFS-16860: --- [~hemanthboyina] should this be backported? > Upgrade moment.min.js to 2.29.4 > --- > > Key: HDFS-16860 > URL: https://issues.apache.org/jira/browse/HDFS-16860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, ui >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: Anurag Parvatikar >Priority: Major > Labels: pull-request-available, transitive-cve > Fix For: 3.4.0 > > > Upgrade moment.min.js to 2.29.4 to resolve > https://nvd.nist.gov/vuln/detail/CVE-2022-31129 > "Users may notice a noticeable slowdown is observed with inputs above 10k > characters. Users who pass user-provided strings without sanity length checks > to moment constructor are vulnerable to (Re)DoS attacks. The problem is > patched in 2.29.4" > this only appears to affect the UI, not the yarn services, so it is a > self-harm DoS rather than anything important. "if you pass in big strings the > ui slows down" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16864) HDFS advisory caching should drop cache behind block when block closed
[
https://issues.apache.org/jira/browse/HDFS-16864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645620#comment-17645620
]
Steve Loughran commented on HDFS-16864:
---
wanted to kick out an rc 0 next week (been testing the staging process while
getting an abfs blocker in)
* how critical is this
* how risky is it in terms of "if it plays up, what will go wrong"
> HDFS advisory caching should drop cache behind block when block closed
> --
>
> Key: HDFS-16864
> URL: https://issues.apache.org/jira/browse/HDFS-16864
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.4
>Reporter: Dave Marion
>Priority: Minor
> Labels: pull-request-available
>
> One of the comments in HDFS-4817 describes the behavior in
> BlockReceiver.manageWriterOsCache:
> "The general idea is that there isn't much point in calling
> {{sync_file_pages}} twice on the same offsets, since the sync process has
> presumably already begun. On the other hand, calling
> {{fadvise(FADV_DONTNEED)}} again and again will tend to purge more and more
> bytes from the cache. The reason is because dirty pages (those containing
> un-written-out-data) cannot be purged using {{{}FADV_DONTNEED{}}}. And we
> can't know exactly when the pages we wrote will be flushed to disk. But we do
> know that calling {{FADV_DONTNEED}} on very recently written bytes is a
> waste, since they will almost certainly not have been written out to disk.
> That is why it purges between 0 and {{{}lastCacheManagementOffset -
> CACHE_WINDOW_SIZE{}}}, rather than simply 0 to pos."
> Looking at the code, I'm wondering if at least the last 8MB (size of
> CACHE_WINDOW_SIZE) of a block might be left without an associated
> FADVISE_DONT_NEED call. We're having a
> [discussion|https://the-asf.slack.com/archives/CERNB8NDC/p1669399302264189]
> in #accumulo about the file caching feature and I found some interesting
> [results|https://gist.github.com/dlmarion/1835f387b0fa8fb9dbf849a0c87b6d04]
> in a test that we wrote. Specifically, that for a multi-block file using
> setDropBehind with either hsync or CreateFlag.SYNC_BLOCK, parts of each block
> remained in the cache instead of parts of the last block.
> I'm wondering if there is a reason not to call fadvise(FADV_DONTNEED) on the
> entire block in close
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java#L371]
> when dropCacheBehindWrites is true.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16860) Upgrade moment.min.js to 2.29.4
[ https://issues.apache.org/jira/browse/HDFS-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16860: -- Description: Upgrade moment.min.js to 2.29.4 to resolve https://nvd.nist.gov/vuln/detail/CVE-2022-31129 "Users may notice a noticeable slowdown is observed with inputs above 10k characters. Users who pass user-provided strings without sanity length checks to moment constructor are vulnerable to (Re)DoS attacks. The problem is patched in 2.29.4" this only appears to affect the UI, not the yarn services, so it is a self-harm DoS rather than anything important. "if you pass in big strings the ui slows down" was:Upgrade moment.min.js to 2.29.4 to resolve https://nvd.nist.gov/vuln/detail/CVE-2022-31129 > Upgrade moment.min.js to 2.29.4 > --- > > Key: HDFS-16860 > URL: https://issues.apache.org/jira/browse/HDFS-16860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, ui >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Labels: transitive-cve > > Upgrade moment.min.js to 2.29.4 to resolve > https://nvd.nist.gov/vuln/detail/CVE-2022-31129 > "Users may notice a noticeable slowdown is observed with inputs above 10k > characters. Users who pass user-provided strings without sanity length checks > to moment constructor are vulnerable to (Re)DoS attacks. The problem is > patched in 2.29.4" > this only appears to affect the UI, not the yarn services, so it is a > self-harm DoS rather than anything important. "if you pass in big strings the > ui slows down" -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16860) Upgrade moment.min.js to 2.29.4
[ https://issues.apache.org/jira/browse/HDFS-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16860: -- Component/s: build > Upgrade moment.min.js to 2.29.4 > --- > > Key: HDFS-16860 > URL: https://issues.apache.org/jira/browse/HDFS-16860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, ui >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Labels: transitive-cve > > Upgrade moment.min.js to 2.29.4 to resolve > https://nvd.nist.gov/vuln/detail/CVE-2022-31129 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16860) Upgrade moment.min.js to 2.29.4
[ https://issues.apache.org/jira/browse/HDFS-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16860: -- Labels: transitive-cve (was: ) > Upgrade moment.min.js to 2.29.4 > --- > > Key: HDFS-16860 > URL: https://issues.apache.org/jira/browse/HDFS-16860 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Labels: transitive-cve > > Upgrade moment.min.js to 2.29.4 to resolve > https://nvd.nist.gov/vuln/detail/CVE-2022-31129 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16860) Upgrade moment.min.js to 2.29.4
[ https://issues.apache.org/jira/browse/HDFS-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17643416#comment-17643416 ] Steve Loughran commented on HDFS-16860: --- when you file jiras like this, can you add the component to the Components field, and which affected versions. thanks > Upgrade moment.min.js to 2.29.4 > --- > > Key: HDFS-16860 > URL: https://issues.apache.org/jira/browse/HDFS-16860 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > > Upgrade moment.min.js to 2.29.4 to resolve > https://nvd.nist.gov/vuln/detail/CVE-2022-31129 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16766) hdfs ec command loads (administrator provided) erasure code policy files without disabling xml entity expansion
[
https://issues.apache.org/jira/browse/HDFS-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16766:
--
Description:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec}} command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
bq. hdfs cluster administrators who receive XML erasure coding policies from
untrusted sources (email etc) must sanitize the file by removing all &entity;
references before using the "hdfs ec" command. otherwise the tool will fail
before it has a chance to apply whatever the malicious EC policy was.
Alternatively: do not configure your hadoop cluster from XML files you haven't
written yourself.
was:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
bq. hdfs cluster administrators who receive XML erasure coding policies from
untrusted sources (email etc) must sanitize the file by removing all &entity;
references before using the "hdfs ec" command. otherwise the tool will fail
before it has a chance to apply whatever the malicious EC policy was.
Alternatively: do not configure your hadoop cluster from XML files you haven't
written yourself.
> hdfs ec command loads (administrator provided) erasure code policy files
> without disabling xml entity expansion
> ---
>
> Key: HDFS-16766
> URL: https://issues.apache.org/jira/browse/HDFS-16766
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 3.3.4
>Reporter: Jing
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.2.5
>
>
> XML External Entity (XXE) attacks can occur when an XML parser supports XML
> entities while processing XML received from an untrusted source. The attack
> resides in XML input containing references to an external entity an is parsed
> by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
> h3. sonatype-2022-5732
> If anyone is landing on this page following the sonatype-2022-5732 alert
> # the xml expansion only happens on the command line of the {{hdfs ec}}
> command
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
> # the outcome of entity expansion will be the command failing/running out of
> memory
> # if a cluster admin is loading erasure policies from untrusted sources,
> there are fundamental process issues to worry about beyond xml references
> bq. hdfs cluster administrators who receive XML erasure coding policies from
> untrusted sources (email etc) must sanitize the file by removing all &entity;
> references before using the "hdfs ec" command. otherwise the tool will fail
> before it has a chance to apply whatever the malicious EC policy was.
> Alternatively:
[jira] [Updated] (HDFS-16800) Upgrade Huawei OBS client to 3.22.3.1
[ https://issues.apache.org/jira/browse/HDFS-16800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16800: -- Component/s: (was: hdfs) > Upgrade Huawei OBS client to 3.22.3.1 > - > > Key: HDFS-16800 > URL: https://issues.apache.org/jira/browse/HDFS-16800 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Cheng Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16777) datatables@1.10.17 sonatype-2020-0988 vulnerability
[ https://issues.apache.org/jira/browse/HDFS-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17622175#comment-17622175 ] Steve Loughran commented on HDFS-16777: --- if this is low risk, can we get backports for the 3.3 and 3.3.5 branches > datatables@1.10.17 sonatype-2020-0988 vulnerability > > > Key: HDFS-16777 > URL: https://issues.apache.org/jira/browse/HDFS-16777 > Project: Hadoop HDFS > Issue Type: Bug > Components: ui >Affects Versions: 3.3.4 >Reporter: Eugene Shinn (Truveta) >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Our static analysis security tool detected that HDFS's UI currently includes > a vulnerable version of datatables detected by Sonatype (sonatype-2020-0988). > From the vulnerability description: > _"The `datatables.net` package is vulnerable to Prototype Pollution. The > `setData` function in `jquery.dataTables.js` fails to protect prototype > attributes when objects are created during the application's execution. A > remote attacker can exploit this to modify the behavior of object prototypes > which, depending on their use in the application, may result in a Denial of > Service (DoS), Remote Code Execution (RCE), or other unexpected execution > flow."_ > This issue was addressed in v 1.11.5 (ref: [Fix: Protect developers from > inadvertantely introducing prototype pol… · > DataTables/Dist-DataTables@e2e19ea > (github.com)).|https://github.com/DataTables/Dist-DataTables/commit/e2e19eac7e5a6f140d7eefca5c7deba165b357eb#diff-e7d8309f017dd2ef6385fa8cdc1539a2R2765] > N.B. this issue was also detected within the YARN UI as well. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16795) Use secure XML parser utils in hdfs classes
[ https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HDFS-16795. --- Fix Version/s: 3.3.5 Resolution: Fixed > Use secure XML parser utils in hdfs classes > --- > > Key: HDFS-16795 > URL: https://issues.apache.org/jira/browse/HDFS-16795 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5, 3.3.9 > > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16795) Use secure XML parser utils in hdfs classes
[ https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HDFS-16795: - Assignee: PJ Fanning > Use secure XML parser utils in hdfs classes > --- > > Key: HDFS-16795 > URL: https://issues.apache.org/jira/browse/HDFS-16795 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Assignee: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16795) Use secure XML parser utils in hdfs classes
[ https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16795: -- Fix Version/s: 3.4.0 3.3.9 > Use secure XML parser utils in hdfs classes > --- > > Key: HDFS-16795 > URL: https://issues.apache.org/jira/browse/HDFS-16795 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16766) hdfs ec command loads (administrator provided) erasure code policy files without disabling xml entity expansion
[
https://issues.apache.org/jira/browse/HDFS-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16766:
--
Description:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
bq. hdfs cluster administrators who receive XML erasure coding policies from
untrusted sources (email etc) must sanitize the file by removing all &entity;
references before using the "hdfs ec" command. otherwise the tool will fail
before it has a chance to apply whatever the malicious EC policy was.
Alternatively: do not configure your hadoop cluster from XML files you haven't
written yourself.
was:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
bq. hdfs cluster administrators who receive XML erasure coding policies from
untrusted sources (email etc) must sanitize the file by removing all &entity;
references before using the "hdfs ec" command. otherwise the tool will fail
before it has a chance to apply whatever the malicious EC policy was.
> hdfs ec command loads (administrator provided) erasure code policy files
> without disabling xml entity expansion
> ---
>
> Key: HDFS-16766
> URL: https://issues.apache.org/jira/browse/HDFS-16766
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 3.3.4
>Reporter: Jing
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.2.5
>
>
> XML External Entity (XXE) attacks can occur when an XML parser supports XML
> entities while processing XML received from an untrusted source. The attack
> resides in XML input containing references to an external entity an is parsed
> by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
> h3. sonatype-2022-5732
> If anyone is landing on this page following the sonatype-2022-5732 alert
> # the xml expansion only happens on the command line of the {{hdfs ec))
> command
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
> # the outcome of entity expansion will be the command failing/running out of
> memory
> # if a cluster admin is loading erasure policies from untrusted sources,
> there are fundamental process issues to worry about beyond xml references
> bq. hdfs cluster administrators who receive XML erasure coding policies from
> untrusted sources (email etc) must sanitize the file by removing all &entity;
> references before using the "hdfs ec" command. otherwise the tool will fail
> before it has a chance to apply whatever the malicious EC policy was.
> Alternatively: do not configure your hadoop cluster from XML files you
> haven't written yourself.
--
This me
[jira] [Updated] (HDFS-16766) hdfs ec command loads (administrator provided) erasure code policy files without disabling xml entity expansion
[
https://issues.apache.org/jira/browse/HDFS-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16766:
--
Description:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
bq. hdfs cluster administrators who receive XML erasure coding policies from
untrusted sources (email etc) must sanitize the file by removing all &entity;
references before using the "hdfs ec" command. otherwise the tool will fail
before it has a chance to apply whatever the malicious EC policy was.
was:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
bq.: hdfs cluster administrators who receive XML erasure coding policies from
untrusted sources (email etc) must sanitize the file by removing all &entity;
references before using the "hdfs ec" command. otherwise the tool will fail
before it has a chance to apply whatever the malicious EC policy was.
> hdfs ec command loads (administrator provided) erasure code policy files
> without disabling xml entity expansion
> ---
>
> Key: HDFS-16766
> URL: https://issues.apache.org/jira/browse/HDFS-16766
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 3.3.4
>Reporter: Jing
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.2.5
>
>
> XML External Entity (XXE) attacks can occur when an XML parser supports XML
> entities while processing XML received from an untrusted source. The attack
> resides in XML input containing references to an external entity an is parsed
> by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
> h3. sonatype-2022-5732
> If anyone is landing on this page following the sonatype-2022-5732 alert
> # the xml expansion only happens on the command line of the {{hdfs ec))
> command
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
> # the outcome of entity expansion will be the command failing/running out of
> memory
> # if a cluster admin is loading erasure policies from untrusted sources,
> there are fundamental process issues to worry about beyond xml references
> bq. hdfs cluster administrators who receive XML erasure coding policies from
> untrusted sources (email etc) must sanitize the file by removing all &entity;
> references before using the "hdfs ec" command. otherwise the tool will fail
> before it has a chance to apply whatever the malicious EC policy was.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additiona
[jira] [Updated] (HDFS-16766) hdfs ec command loads (administrator provided) erasure code policy files without disabling xml entity expansion
[
https://issues.apache.org/jira/browse/HDFS-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16766:
--
Description:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
bq.: hdfs cluster administrators who receive XML erasure coding policies from
untrusted sources (email etc) must sanitize the file by removing all &entity;
references before using the "hdfs ec" command. otherwise the tool will fail
before it has a chance to apply whatever the malicious EC policy was.
was:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
> hdfs ec command loads (administrator provided) erasure code policy files
> without disabling xml entity expansion
> ---
>
> Key: HDFS-16766
> URL: https://issues.apache.org/jira/browse/HDFS-16766
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 3.3.4
>Reporter: Jing
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.2.5
>
>
> XML External Entity (XXE) attacks can occur when an XML parser supports XML
> entities while processing XML received from an untrusted source. The attack
> resides in XML input containing references to an external entity an is parsed
> by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
> h3. sonatype-2022-5732
> If anyone is landing on this page following the sonatype-2022-5732 alert
> # the xml expansion only happens on the command line of the {{hdfs ec))
> command
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
> # the outcome of entity expansion will be the command failing/running out of
> memory
> # if a cluster admin is loading erasure policies from untrusted sources,
> there are fundamental process issues to worry about beyond xml references
> bq.: hdfs cluster administrators who receive XML erasure coding policies from
> untrusted sources (email etc) must sanitize the file by removing all &entity;
> references before using the "hdfs ec" command. otherwise the tool will fail
> before it has a chance to apply whatever the malicious EC policy was.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16766) hdfs ec command loads (administrator provided) erasure code policy files without disabling xml entity expansion
[
https://issues.apache.org/jira/browse/HDFS-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16766:
--
Description:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if a cluster admin is loading erasure policies from untrusted sources, there
are fundamental process issues to worry about beyond xml references
was:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if you cluster admin is loading erasure policies from untrusted sources,
there are fundamental process issues to worry about beyond xml references
> hdfs ec command loads (administrator provided) erasure code policy files
> without disabling xml entity expansion
> ---
>
> Key: HDFS-16766
> URL: https://issues.apache.org/jira/browse/HDFS-16766
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 3.3.4
>Reporter: Jing
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.2.5
>
>
> XML External Entity (XXE) attacks can occur when an XML parser supports XML
> entities while processing XML received from an untrusted source. The attack
> resides in XML input containing references to an external entity an is parsed
> by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
> h3. sonatype-2022-5732
> If anyone is landing on this page following the sonatype-2022-5732 alert
> # the xml expansion only happens on the command line of the {{hdfs ec))
> command
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
> # the outcome of entity expansion will be the command failing/running out of
> memory
> # if a cluster admin is loading erasure policies from untrusted sources,
> there are fundamental process issues to worry about beyond xml references
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16766) hdfs ec command loads (administrator provided) erasure code policy files without disabling xml entity expansion
[
https://issues.apache.org/jira/browse/HDFS-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16766:
--
Summary: hdfs ec command loads (administrator provided) erasure code policy
files without disabling xml entity expansion (was: XML External Entity (XXE)
attacks can occur while processing XML received from an untrusted source)
> hdfs ec command loads (administrator provided) erasure code policy files
> without disabling xml entity expansion
> ---
>
> Key: HDFS-16766
> URL: https://issues.apache.org/jira/browse/HDFS-16766
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 3.3.4
>Reporter: Jing
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.2.5
>
>
> XML External Entity (XXE) attacks can occur when an XML parser supports XML
> entities while processing XML received from an untrusted source. The attack
> resides in XML input containing references to an external entity an is parsed
> by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
> h3. sonatype-2022-5732
> If anyone is landing on this page following the sonatype-2022-5732 alert
> # the xml expansion only happens on the command line of the {{hdfs ec))
> command
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
> # the outcome of entity expansion will be the command failing/running out of
> memory
> # if you cluster admin is loading erasure policies from untrusted sources,
> there are fundamental process issues to worry about beyond xml references
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16766) XML External Entity (XXE) attacks can occur while processing XML received from an untrusted source
[
https://issues.apache.org/jira/browse/HDFS-16766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16766:
--
Description:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
h3. sonatype-2022-5732
If anyone is landing on this page following the sonatype-2022-5732 alert
# the xml expansion only happens on the command line of the {{hdfs ec)) command
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
# the outcome of entity expansion will be the command failing/running out of
memory
# if you cluster admin is loading erasure policies from untrusted sources,
there are fundamental process issues to worry about beyond xml references
was:
XML External Entity (XXE) attacks can occur when an XML parser supports XML
entities while processing XML received from an untrusted source. The attack
resides in XML input containing references to an external entity an is parsed
by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
> XML External Entity (XXE) attacks can occur while processing XML received
> from an untrusted source
> --
>
> Key: HDFS-16766
> URL: https://issues.apache.org/jira/browse/HDFS-16766
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 3.3.4
>Reporter: Jing
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.5, 3.2.5
>
>
> XML External Entity (XXE) attacks can occur when an XML parser supports XML
> entities while processing XML received from an untrusted source. The attack
> resides in XML input containing references to an external entity an is parsed
> by the weakly configured javax.xml.parsers.DocumentBuilder XML parser.
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/util/ECPolicyLoader.java#L93
> h3. sonatype-2022-5732
> If anyone is landing on this page following the sonatype-2022-5732 alert
> # the xml expansion only happens on the command line of the {{hdfs ec))
> command
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html#Administrative_commands
> # the outcome of entity expansion will be the command failing/running out of
> memory
> # if you cluster admin is loading erasure policies from untrusted sources,
> there are fundamental process issues to worry about beyond xml references
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16024) RBF: Rename data to the Trash should be based on src locations
[ https://issues.apache.org/jira/browse/HDFS-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17614984#comment-17614984 ] Steve Loughran commented on HDFS-16024: --- not too late to get into 3.3.5 if you think it is low risk and put the cherrypick pr through a github/jenkins build > RBF: Rename data to the Trash should be based on src locations > -- > > Key: HDFS-16024 > URL: https://issues.apache.org/jira/browse/HDFS-16024 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Affects Versions: 3.4.0 >Reporter: Xiangyi Zhu >Assignee: Xiangyi Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > Time Spent: 7h 20m > Remaining Estimate: 0h > > 1.When deleting data to the Trash without configuring a mount point for the > Trash, the Router should recognize and move the data to the Trash > 2.When the user’s trash can is configured with a mount point and is different > from the NS of the deleted directory, the router should identify and move the > data to the trash can of the current user of src > The same is true for using ViewFs mount points, I think we should be > consistent with it -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16795) Use secure XML parser utils in hdfs classes
[ https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16795: -- Summary: Use secure XML parser utils in hdfs classes (was: Use secure XML parser utils) > Use secure XML parser utils in hdfs classes > --- > > Key: HDFS-16795 > URL: https://issues.apache.org/jira/browse/HDFS-16795 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16795) Use secure XML parser utils in hdfs classes
[ https://issues.apache.org/jira/browse/HDFS-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16795: -- Affects Version/s: 3.3.4 > Use secure XML parser utils in hdfs classes > --- > > Key: HDFS-16795 > URL: https://issues.apache.org/jira/browse/HDFS-16795 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.4 >Reporter: PJ Fanning >Priority: Major > Labels: pull-request-available > > Uptakes HADOOP-18469 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16142) TestObservernode#testMkdirsRaceWithObserverRead is flaky
[
https://issues.apache.org/jira/browse/HDFS-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613707#comment-17613707
]
Steve Loughran commented on HDFS-16142:
---
seen this again
{code}
Error Message
Client #3 lastSeenStateId=-9223372036854775808 activStateId=37
null
Stacktrace
java.lang.AssertionError:
Client #3 lastSeenStateId=-9223372036854775808 activStateId=37
null
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestObserverNode.testMkdirsRaceWithObserverRead(TestObserverNode.java:607)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
{code}
> TestObservernode#testMkdirsRaceWithObserverRead is flaky
>
>
> Key: HDFS-16142
> URL: https://issues.apache.org/jira/browse/HDFS-16142
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Akira Ajisaka
>Priority: Major
> Attachments: patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
>
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3227/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
> {quote}
> [ERROR] Tests run: 21, Failures: 1, Errors: 4, Skipped: 0, Time elapsed:
> 741.856 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.namenode.ha.TestObserverNode [ERROR]
> testMkdirsRaceWithObserverRead(org.apache.hadoop.hdfs.server.namenode.ha.TestObserverNode)
> Time elapsed: 2.697 s <<< FAILURE! java.lang.AssertionError: Client #2
> lastSeenStateId=-9223372036854775808 activStateId=37 null at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.assertTrue(Assert.java:42) at
> org.apache.hadoop.hdfs.server.namenode.ha.TestObserverNode.testMkdirsRaceWithObserverRead(TestObserverNode.java:557)
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16791) New API for enclosing root path for a file
[ https://issues.apache.org/jira/browse/HDFS-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612620#comment-17612620 ] Steve Loughran commented on HDFS-16791: --- usual expectations of anything going near the filesystem apis, as covered in FileSystem.java javadocs. I'd prefer a new interface which AbstractFileSystem and FileSystem can both implement; makes it easier to write contract tests for both. > New API for enclosing root path for a file > -- > > Key: HDFS-16791 > URL: https://issues.apache.org/jira/browse/HDFS-16791 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Tom McCormick >Priority: Major > > At LinkedIn we run many HDFS volumes that are federated by either > ViewFilesystem or Router Based Federation. As our number of hdfs volumes > grows, we have a growing need to migrate data seemlessly across volumes. > Many frameworks have a notion of staging or temp directories, but those > directories often live in random locations. We want an API getEnclosingRoot, > which provides the root path a file or dataset. > In ViewFilesystem / Router Based Federation, the enclosingRoot will be the > mount point. > We will also take into account other restrictions for renames like > encryptions zones. > If there are several paths (a mount point and an encryption zone), we will > return the longer path -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16789) Update FileSystem class to read disk usage from actual usage value instead of file's length
[ https://issues.apache.org/jira/browse/HDFS-16789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612228#comment-17612228 ] Steve Loughran commented on HDFS-16789: --- bq. retrieve the disk usage value directly from file's length instead of actual usage value. bq. We need to update FileSystem & related classes to read disk usage from actual usage value. what do you mean here? number of blocks consumed on hdd? as that info isn't available to most filesystems. GetContentSummary is used in places in hive and spark to estimate workload; for it the summary is exactly what's needed. its unfortunate there that it is so inefficient on cloud storage (single thread treewalk). HADOOP-13704 did some speedup for s3a, but it is still an expensive operation. and the other clouds are still doing that treewalk. # hive/spark should stop using it # a multithreaded implementation would be nice > Update FileSystem class to read disk usage from actual usage value instead of > file's length > --- > > Key: HDFS-16789 > URL: https://issues.apache.org/jira/browse/HDFS-16789 > Project: Hadoop HDFS > Issue Type: Improvement > Components: dfs >Reporter: Dave Teng >Assignee: Dave Teng >Priority: Major > > Currently FileSystem class retrieve the disk usage value directly from > [file's > length|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L1895] > instead of actual usage value. > We need to update FileSystem & related classes to read disk usage from actual > usage value. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14478) Add libhdfs APIs for openFile
[
https://issues.apache.org/jira/browse/HDFS-14478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HDFS-14478:
-
Assignee: Sahil Takiar (was: Steve Loughran)
> Add libhdfs APIs for openFile
> -
>
> Key: HDFS-14478
> URL: https://issues.apache.org/jira/browse/HDFS-14478
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.3.5
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> HADOOP-15229 added a "FileSystem builder-based openFile() API" that allows
> specifying configuration values for opening files (similar to HADOOP-14365).
> Support for {{openFile}} will be a little tricky as it is asynchronous and
> {{FutureDataInputStreamBuilder#build}} returns a {{CompletableFuture}}.
> At a high level, the API for {{openFile}} could look something like this:
> {code:java}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags,
> int bufferSize, short replication, tSize blocksize);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderAlloc(hdfsFS fs,
> const char *path);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderMust(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderOpt(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileFuture *hdfsOpenFileBuilderBuild(hdfsOpenFileBuilder *builder);
> void hdfsOpenFileBuilderFree(hdfsOpenFileBuilder *builder);
> hdfsFile hdfsOpenFileFutureGet(hdfsOpenFileFuture *future);
> hdfsFile hdfsOpenFileFutureGetWithTimeout(hdfsOpenFileFuture *future,
> int64_t timeout, javaConcurrentTimeUnit timeUnit);
> int hdfsOpenFileFutureCancel(hdfsOpenFileFuture *future,
> int mayInterruptIfRunning);
> void hdfsOpenFileFutureFree(hdfsOpenFileFuture *future);
> {code}
> Instead of exposing all the functionality of {{CompleteableFuture}} libhdfs
> would just expose the functionality of {{Future}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-2139) Fast copy for HDFS.
[
https://issues.apache.org/jira/browse/HDFS-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600719#comment-17600719
]
Steve Loughran commented on HDFS-2139:
--
As you are going to are you making changes in the public/stable filesystem
APIs, I'd like to keep an eye on that.
Anything that goes in
* should have its own HADOOP- JIRA, even if all the work is in the hdfs branch.
* needs to be able to work well in cloud infrastructure is which implement
similar capabilities but different latencies etc.
*
Whatever copy command goes in shouldn't be a copy(src, dest) -> boolean call
but instead return a builder subclass of FSDataOutputStreamBuilder which can
allow for extra options, and return a Future which the caller can block on; etc
etc
* needs to have something in the filesystem markdown spec and a matching
contract test
* and a PathCapabilities probe which can check for the API being available
under a path.
* and fail by throwing exceptions, not returning true/false. A return value is
needed for the future; something which implements IOStatisticsSource is useful.
Any new API should work identically with azure storage as/when it adds the
needed operation.; S3's file-by-file COPY call would also be supported. It is
not going to be as fast as anything in HFS, but as it doesn't use any network
IO outside the S3 store it is higher bandwidth and scales better than this CP
would normally do. (The Hive team have asked for S3 copying before, but it gets
complex once you start to think about encryption; s3a support might need to add
extra source files
{code}
Future r = fs.copy(src, dest)
.withFileStatus(srcStatus) // as with openFile
.withProgress(progressable)
.must("fs.option.copy.atomic", true) // example of a builder option,
here one requiring atomic file/dir copy.
.build())
r.get(); // block for result
{code}
I'd also propose it as a new interface which both FileContext and FileSystem
implement.
Also, fs shell could be good simple place for this to be used too...easier to
get working/stabilise there.
> Fast copy for HDFS.
> ---
>
> Key: HDFS-2139
> URL: https://issues.apache.org/jira/browse/HDFS-2139
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Pritam Damania
>Assignee: Rituraj
>Priority: Major
> Attachments: HDFS-2139-For-2.7.1.patch, HDFS-2139.patch,
> HDFS-2139.patch, image-2022-08-11-11-48-17-994.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> There is a need to perform fast file copy on HDFS. The fast copy mechanism
> for a file works as
> follows :
> 1) Query metadata for all blocks of the source file.
> 2) For each block 'b' of the file, find out its datanode locations.
> 3) For each block of the file, add an empty block to the namesystem for
> the destination file.
> 4) For each location of the block, instruct the datanode to make a local
> copy of that block.
> 5) Once each datanode has copied over its respective blocks, they
> report to the namenode about it.
> 6) Wait for all blocks to be copied and exit.
> This would speed up the copying process considerably by removing top of
> the rack data transfers.
> Note : An extra improvement, would be to instruct the datanode to create a
> hardlink of the block file if we are copying a block on the same datanode
> [~xuzq_zander]Provided a design doc
> https://docs.google.com/document/d/1OHdUpQmKD3TZ3xdmQsXNmlXJetn2QFPinMH31Q4BqkI/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16755) TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution
[ https://issues.apache.org/jira/browse/HDFS-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599440#comment-17599440 ] Steve Loughran commented on HDFS-16755: --- no problem. if it's a big change i likelthose prs just for a yetus test run; here i manually ran the single test before pushing up > TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host > resolution > --- > > Key: HDFS-16755 > URL: https://issues.apache.org/jira/browse/HDFS-16755 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.4.0, 3.3.9 > Environment: Running using both Maven Surefire and an IDE results in > a test failure. Switching the name to "bogus.invalid" results in the > expected behavior, which depends on an UnknownHostException. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Tests that want to use an unresolvable address may actually resolve in some > environments. Replacing host names like "bogus" with a IETF RFC 2606 domain > name avoids the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16755) TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution
[ https://issues.apache.org/jira/browse/HDFS-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17599090#comment-17599090 ] Steve Loughran commented on HDFS-16755: --- it's already in branch-3.3, why it is tagged as the fix version. > TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host > resolution > --- > > Key: HDFS-16755 > URL: https://issues.apache.org/jira/browse/HDFS-16755 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.4.0, 3.3.9 > Environment: Running using both Maven Surefire and an IDE results in > a test failure. Switching the name to "bogus.invalid" results in the > expected behavior, which depends on an UnknownHostException. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Tests that want to use an unresolvable address may actually resolve in some > environments. Replacing host names like "bogus" with a IETF RFC 2606 domain > name avoids the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16755) TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution
[ https://issues.apache.org/jira/browse/HDFS-16755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HDFS-16755. --- Fix Version/s: 3.3.9 Resolution: Fixed > TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host > resolution > --- > > Key: HDFS-16755 > URL: https://issues.apache.org/jira/browse/HDFS-16755 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.4.0, 3.3.9 > Environment: Running using both Maven Surefire and an IDE results in > a test failure. Switching the name to "bogus.invalid" results in the > expected behavior, which depends on an UnknownHostException. >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > > Tests that want to use an unresolvable address may actually resolve in some > environments. Replacing host names like "bogus" with a IETF RFC 2606 domain > name avoids the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16747) some hdfs unit tests failing
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584949#comment-17584949 ] Steve Loughran commented on HDFS-16747: --- ok, changed the title. probably something related to your setup, or they are intermittent ones which have surfaced. afraid you are going to have to start debugging those tests to work out what they expect and why your deployment doesn't work > some hdfs unit tests failing > > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.4.0 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16747) List of unit test failing in 3.3.3
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16747: -- Affects Version/s: 3.4.0 (was: 3.3.3) > List of unit test failing in 3.3.3 > -- > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.4.0 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16747) some hdfs unit tests failing
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-16747: -- Summary: some hdfs unit tests failing (was: List of unit test failing in 3.3.3) > some hdfs unit tests failing > > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.4.0 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16747) List of unit test failing in 3.3.3
[ https://issues.apache.org/jira/browse/HDFS-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17584729#comment-17584729 ] Steve Loughran commented on HDFS-16747: --- samrat, are these tests still failing on hadoop trunk/branch-3.3? as those are the branches where all fixed will go in. 3.3.3/3.3.4 are already released > List of unit test failing in 3.3.3 > -- > > Key: HDFS-16747 > URL: https://issues.apache.org/jira/browse/HDFS-16747 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.3.3 >Reporter: Samrat Deb >Assignee: Samrat Deb >Priority: Major > > Effected Test class > * org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > * org.apache.hadoop.hdfs.TestMultipleNNPortQOP > * org.apache.hadoop.hdfs.TestBlockTokenWrappingQOP > * -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16711) Empty hadoop-client-api artifacts on maven central
[ https://issues.apache.org/jira/browse/HDFS-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17575358#comment-17575358 ] Steve Loughran commented on HDFS-16711: --- given the artifacts have been published, i don't think we can fix this. what matters more is that we include validation of those artifacts in the release process. > Empty hadoop-client-api artifacts on maven central > -- > > Key: HDFS-16711 > URL: https://issues.apache.org/jira/browse/HDFS-16711 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 3.2.3 >Reporter: Robin Wolters >Priority: Major > > I observed that for at least version 3.2.3 the artifacts on maven central for > the shaded jars of both > [hadoop-client-api|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.2.3/] > and > [hadoop-client-runtime|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.2.3/] > are empty, i.e. of ~45KB size and do not contain any class files if > extracted (or listed with "jar tf"). > I've come across this [e-mail > thread|https://www.mail-archive.com/common-dev@hadoop.apache.org/msg37261.html] > suggesting that there was the same problem with version 3.3.3, which appears > to be fixed. Version 3.2.3 is mentioned as well, could it be that this > version simply wasn't re-released? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16711) Empty hadoop-client-api artifacts on maven central
[ https://issues.apache.org/jira/browse/HDFS-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HDFS-16711. --- Resolution: Won't Fix closing as "cantfix". sorry > Empty hadoop-client-api artifacts on maven central > -- > > Key: HDFS-16711 > URL: https://issues.apache.org/jira/browse/HDFS-16711 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 3.2.3 >Reporter: Robin Wolters >Priority: Major > > I observed that for at least version 3.2.3 the artifacts on maven central for > the shaded jars of both > [hadoop-client-api|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.2.3/] > and > [hadoop-client-runtime|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.2.3/] > are empty, i.e. of ~45KB size and do not contain any class files if > extracted (or listed with "jar tf"). > I've come across this [e-mail > thread|https://www.mail-archive.com/common-dev@hadoop.apache.org/msg37261.html] > suggesting that there was the same problem with version 3.3.3, which appears > to be fixed. Version 3.2.3 is mentioned as well, could it be that this > version simply wasn't re-released? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16711) Empty hadoop-client-api artifacts on maven central
[ https://issues.apache.org/jira/browse/HDFS-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17574342#comment-17574342 ] Steve Loughran commented on HDFS-16711: --- hmmm. the 3.3.4 RC1 is up to play with; looking at its staging https://repository.apache.org/content/repositories/orgapachehadoop-1358/org/apache/hadoop/hadoop-client-api/3.3.4/ looking at mvnrepo, yes, looks like it is missing in 3.2.3. but it is found in 3.2.4, to which you should upgrade if for some reason you can't go to 3.3.4 https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client-api/3.2.3 > Empty hadoop-client-api artifacts on maven central > -- > > Key: HDFS-16711 > URL: https://issues.apache.org/jira/browse/HDFS-16711 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 3.2.3 >Reporter: Robin Wolters >Priority: Major > > I observed that for at least version 3.2.3 the artifacts on maven central for > the shaded jars of both > [hadoop-client-api|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-api/3.2.3/] > and > [hadoop-client-runtime|https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.2.3/] > are empty, i.e. of ~45KB size and do not contain any class files if > extracted (or listed with "jar tf"). > I've come across this [e-mail > thread|https://www.mail-archive.com/common-dev@hadoop.apache.org/msg37261.html] > suggesting that there was the same problem with version 3.3.3, which appears > to be fixed. Version 3.2.3 is mentioned as well, could it be that this > version simply wasn't re-released? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11440) Error compiling native libraries error: ^ ^ _mm_set_pd ^ ^ : target specific option mismatch
[ https://issues.apache.org/jira/browse/HDFS-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572656#comment-17572656 ] Steve Loughran commented on HDFS-11440: --- is this still an issue or can we close it? > Error compiling native libraries error: ^ ^ _mm_set_pd ^ ^ : target specific > option mismatch > > > Key: HDFS-11440 > URL: https://issues.apache.org/jira/browse/HDFS-11440 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 2.7.3 > Environment: Linux 3.16.0-4-686-pae #1 SMP Debian 3.16.39-1 > (2016-12-30) i686 GNU/Linux > Machine: System: Hewlett-Packard product: HP Compaq dc7100 SFF(PC922A) > serial: MXJ512018N >Mobo: Hewlett-Packard model: 097Ch serial: MXJ512018N >Bios: Hewlett-Packard v: 786C1 v01.05 date: 06/16/2004 >Reporter: Dario Castaño >Priority: Blocker > Labels: build > > When i try to build native libraries for my system, i'm getting this error: > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 7:48.369s > [INFO] Finished at: Wed Feb 22 13:14:21 EST 2017 > [INFO] Final Memory: 69M/247M > [INFO] > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project > hadoop-hdfs: An Ant BuildException has occured: exec returned: 1 > [ERROR] around Ant part ... dir="/home/hadoop/hadoop-2.7.3/src/hadoop-2.7.3-src/hadoop-hdfs-project/hadoop-hdfs/target/native" > executable="cmake">... @ 5:146 in > /home/hadoop/hadoop-2.7.3/src/hadoop-2.7.3-src/hadoop-hdfs-project/hadoop-hdfs/target/antrun/build-main.xml > [ERROR] -> [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project > hadoop-hdfs: An Ant BuildException has occured: exec returned: 1 > around Ant part ... dir="/home/hadoop/hadoop-2.7.3/src/hadoop-2.7.3-src/hadoop-hdfs-project/hadoop-hdfs/target/native" > executable="cmake">... @ 5:146 in > /home/hadoop/hadoop-2.7.3/src/hadoop-2.7.3-src/hadoop-hdfs-project/hadoop-hdfs/target/antrun/build-main.xml > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) > at > org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) > at > org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) > at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) > at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) > at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) > at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) > at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) > at > org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) > at > org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) > at > org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) > Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant > BuildException has occured: exec returned: 1 > around Ant part ... dir="/home/hadoop/hadoop-2.7.3/src/hadoop-2.7.3-src/hadoop-hdfs-project/hadoop-hdfs/target/native" > executable="cmake">... @ 5:146 in > /home/hadoop/hadoop-2.7.3/src/hadoop-2.7.3-src/hadoop-hdfs-project/hadoop-hdfs/target/antrun/build-main.xml > at > org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355) > at > org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) > at > org.apache.maven.lifecycle.intern
