Re: HBase lz4 UnsatisfiedLinkError

Hi Ayush, Upgrading to 2.6.0-hadoop3 worked, thanks so much! On 2024-05-25 20:15, Ayush Saxena wrote: Multiple things, the output of checknative only contains these stuff only, not everything. From the code [1], So looking at your command output everything is sorted there barring OpenSSL &

Re: HBase lz4 UnsatisfiedLinkError

Multiple things, the output of checknative only contains these stuff only, not everything. From the code [1], So looking at your command output everything is sorted there barring OpenSSL & PMDK which you explicitly didn't ask for in your maven command & I believe you don't need them either, in

Re: HBase lz4 UnsatisfiedLinkError

Hey Ayush, thanks for the advice! Building 3.3.6 from an EL9.4 machine resulted in the following: [root@localhost bin]# JAVA_HOME=/etc/alternatives/java_sdk_openjdk/ ./hadoop checknative -a 2024-05-25 19:05:56,068 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library

Re: bootstrap standby namenode failure

Hi Anup, Did you explore: -skipSharedEditsCheck, Check this ticket once [1], if your use case is similar, little bit description can be found here [2], search for skipSharedEditsCheck, the jira does mention another solution as well, in case you don't like this or if it doesn't work -Ayush [1]

Re: HBase lz4 UnsatisfiedLinkError

Hi, We can't help with the HBase thing, for that you need to chase the HBase user ML. For the `hadoop checknative -a` showing false, maybe the native libraries that are pre-built & published aren't compatible with the OS you are using, In that case you need to build them on the "same" OS, the

Re: How is HDFS Erasure Coding Phase II now？

Hi, > Or is it just not developed to this point? It isn't developed & I don't think there is any effort going on in that direction > I learned that continuous layout can ensure the locality of file blocks How? Erasure Coding will have BlockGroups not just one Block, whether you write in a

Re: How to contribute code for the first time

Hi Jim, Directly create a PR against the trunk branch in Hadoop repo, if it is accepted then add the link to the PR and resubmit your request for Jira account, it will get approved -Ayush > On 17 Apr 2024, at 10:02 AM, Jim Chen wrote: > >  > Hi all, I want to optimize a script in

Re: Recommended way of using hadoop-minicluster für unit testing?

Hi Ayush, thanks for your time investigating! I followed your recommendation and it seems to work (also for some of our consumer projects), so thanks a lot for your time! Gruß Richard Am Samstag, dem 13.04.2024 um 03:35 +0530 schrieb Ayush Saxena: > Hi Richard, > Thanx for sharing the steps

Re: Recommended way of using hadoop-minicluster für unit testing?

Hi Richard, Thanx for sharing the steps to reproduce the issue. I cloned the Apache Storm repo and was able to repro the issue. The build was indeed failing due to missing classes. Spent some time to debug the issue, might not be very right (no experience with Storm), There are Two ways to get

Re: Recommended way of using hadoop-minicluster für unit testing?

Hi, thanks for the fast reply. The PR is here [1]. It works, if I exclude the client-api and client-api-runtime from being scanned in surefire, which is a hacky workaround for the actual issue. The hadoop-commons jar is a transient dependency of the minicluster, which is used for testing.

Re: [ANNOUNCE] Apache Hadoop 3.4.0 release

Xiaoqiao He and Shilun Fan Awesome! Thanks for leading the effort to release the Hadoop 3.4.0 ! Bests, Sammi On Tue, 19 Mar 2024 at 21:12, slfan1989 wrote: > On behalf of the Apache Hadoop Project Management Committee, We are > pleased to announce the release of Apache Hadoop 3.4.0. > > This

Re: Recommended way of using hadoop-minicluster für unit testing?

Hi Richard, I am not able to decode the issue properly here, It would have been better if you shared the PR or the failure trace as well. QQ: Why are you having hadoop-common as an explicit dependency? Those hadoop-common stuff should be there in hadoop-client-api I quickly checked once on the

Re: ContainerId starts with 1 ?

Dear Hadoop/Yarn community, I still beg your help for the question above. Additionally, I might have other questions. The target is to get the driver container id of a Spark app, from Yarn Aggregation Log. I would like to call LogAggregationIndexedFileController#readAggregatedLogsMeta()

Re: 为什么org.apache.hadoop.fs.FileSystem.Cache.Key的构造方法需要一个conf参数

hi 黄晟， 这是历史遗留代码造成的。过去获取ugi要使用conf，后来改变了ugi的获取方式，但漏删了参数。可参考 https://issues.apache.org/jira/browse/HADOOP-6299 黄晟 于2024年3月18日周一 19:24写道： > > > 为什么org.apache.hadoop.fs.FileSystem.Cache.Key的构造方法需要一个conf参数 > ？但是传进来的这个conf却没有地方使用 > > > > | | > 黄晟 > | > | > huangshen...@163.com > | > >

Re: NM status during RM failover

> We observe a drop of NumActiveNodes metric when fails over on a new RM. Is that normal? Yes, this does not seem unusual - the NMs will try to connect to the old RM for some time before they fail over to the new RM. If this time exceeds the heartbeat interval, the NMs may show up as disconnected

Re: NM status during RM failover

Hi, All: How to reduce RM fail over because it introduces disturbances to current workload. The failover is mainly because of JVM pause (around 6 seconds)and high CPU usage. Thanks. Have a nice day! On Sat, Feb 24, 2024 at 8:06 PM Dong Ye wrote: > Hi, All: > > I have a question, in the

Re: subscribe

Please drop mail to "user-unsubscr...@hadoop.apache.org" as mentioned in the footer mail. From: Shuyan Zhang Date: Thursday, February 1, 2024 at 09:00 To: user@hadoop.apache.org Subject: subscribe subscribe

Re: unsubscribe

Please drop mail to "user-unsubscr...@hadoop.apache.org" as mentioned in the footer mail. On Fri, Feb 9, 2024 at 2:32 PM Henning Blohm wrote: > unsubscribe > > > - > To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >

Re: observer namenode and Router-based Federation

RBF does support observer reads, it was added as part of https://issues.apache.org/jira/browse/HDFS-16767 you need to go through it, there are different configs and stuff you might need to setup to get RBF & Observer NN work together. -Ayush On Fri, 26 Jan 2024 at 13:44, 尉雁磊 wrote: > Can't

Re: Data Remanence in HDFS

Hi Daniel, In short you can’t create a HDFS block with unallocated data. You can create a zero length block, which will result in a zero byte file being created on the data node, but you can’t create a sparse file in HDFS. While HDFS has a block size e.g. 128MB if you create a small file then

Re: Data Remanence in HDFS

Thank Jim, The scenario I have in mind is something like: 1) Ask HDFS to create a file that is 32k in length. 2) Attempt to read the contents of the file. Can I even attempt to read the contents of a file that has not yet been written? If so, what data would get sent? For example, I asked a

Re: Data Remanence in HDFS

Hi Danny, This does depend on a number of circumstances, mostly based on file permissions. If for example a file is deleted without the -skipTrash option then it will be moved to the .Trash directory. From here it could be read, but the original file permissions will be preserved. Therefore if

Re: I don't want to set quotas through the router

Hi, Your question is not very clear. So, I am answering whatever I understand. 1. You don't want Router to manage Quotas? Ans: Then you can use this config: dfs.federation.router.quota.enable and set it to false 2. You have default NS as Router but you want to set Quota individually to NS? Ans.

Re: JSON in Kafka -> ORC in HDFS - Thoughts on different tools?

ts list. Whether it can't be done easier. If not, Flink would be my choice as I had good experience with it and you can really hack anything inside. But prepare yourself, that the requirement list is hard, even if you get pipeline up in 2 weeks, you surely will re-iterate the decision afte

Re: JSON in Kafka -> ORC in HDFS - Thoughts on different tools?

even if you get pipeline up in 2 weeks, you surely will re-iterate the decision after some incidents in next 6 months. If you loosen requirements a bit, it becomes easier and easier. Your current solution sounds very reasonable to me. You picked something that works out of the box (Kafka Conne

Re: INFRA-25203

unsubscribe On Mon, 27 Nov 2023, 11:26 pm Drew Foulks, wrote: > Redirect test. > > -- > Cheers, > > Drew Foulks > ASF Infra > > >

Re: Details about cluster balancing

Thanks Ayush! > On 15-Nov-2023, at 10:59 PM, Ayush Saxena wrote: > > Hi Akash, > You can read about balancer here: > https://apache.github.io/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer > HADOOP-1652(https://issues.apache.org/jira/browse/HADOOP-1652) has > some details

Re: Details about cluster balancing

Hi Akash, You can read about balancer here: https://apache.github.io/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer HADOOP-1652(https://issues.apache.org/jira/browse/HADOOP-1652) has some details around it as well, it has some docs attached to it, you can read them... For the

Re: 关于hadoop-3.3.1版本使用libhdfs3.so 访问hdfs联邦模式router节点rpc端口有20分钟延时问题

Add hdfs-dev@h.a.o and user@h.a.o On Thu, Oct 26, 2023 at 7:07 PM 王继泽 wrote: > 最近在使用hadoop的过程中，发现了一个情况。 > 当我使用c > api向hdfs联邦模式router节点rpc端口发送请求时，比如说写文件，客户端发送完成请求后，hadoop端需要20分钟延时文件才有字节大小，延时期间不能对文件进行操作。 > > 客户端这边运行结束之后，hadoop端日志大致过程： > 1.namenode接收到客户端的请求，FSEditLog打印日志。 >

Re: 关于hadoop-3.3.1版本使用libhdfs3.so 访问hdfs联邦模式router节点rpc端口有20分钟延时问题

Add hdfs-dev@h.a.o and user@h.a.o On Thu, Oct 26, 2023 at 7:07 PM 王继泽 wrote: > 最近在使用hadoop的过程中，发现了一个情况。 > 当我使用c > api向hdfs联邦模式router节点rpc端口发送请求时，比如说写文件，客户端发送完成请求后，hadoop端需要20分钟延时文件才有字节大小，延时期间不能对文件进行操作。 > > 客户端这边运行结束之后，hadoop端日志大致过程： > 1.namenode接收到客户端的请求，FSEditLog打印日志。 >

Re: Namenode Connection Refused

It is not an HA cluster, I gave up on that due to separate problems. And I am doing this query from the same host as the namenode. including the netstat -tulapn that shows the namenode is not exposing the port On Tuesday, October 24, 2023 at 09:40:09 AM PDT, Wei-Chiu Chuang wrote:

Re: Namenode Connection Refused

If it's an HA cluster, is it possible the client doesn't have the proper HA configuration so it doesn't know what host name to connect to? Otherwise, the usual suspect is the firewall configuration between the client and the NameNode. On Tue, Oct 24, 2023 at 9:05 AM Harry Jamison wrote: > I

RE: MODERATE for hdfs-iss...@hadoop.apache.org

take 3-4 calendar days to cover all missing days. - only one partition is done so far, no loss appeared yet. Thank you! -Original Message- From: Ayush Saxena Sent: 18 October, 2023 2:25 To: Sergey Onuchin Cc: Hdfs-dev ; Xiaoqiao He ; user.hadoop Subject: Re: MODERATE for hdfs-is

Re: How to clear EXPIRED routers?

dfs.federation.router.store.router.expiration.deletion is the configuration value for that purpose. https://apache.github.io/hadoop/hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml - Takanobu 2023年10月19日(木) 0:41 杨光 : > Hi everyone! > > I'm using hadoop 3.3.4, and started 5 hdfs routers

Re: DistCP from Hadoop 2.X to 3.X - where to compute

Hi PA, We just did the same work recently, copying data from hadoop 2 to hadoop 3, to be precise, src hadoop version was CDH hadoop-2.6 (5 hdfs nameservices federation), dst hadoop version was hadoop 3.3.4. Both clusters are protected with Kerberos, and of course, two realms have been trusted

Re: MODERATE for hdfs-iss...@hadoop.apache.org

+ user@hadoop This sounds pretty strange, do you have any background job in your cluster running, like for compaction kind of stuff, which plays with the files? Any traces in the Namenode Logs, what happens to the blocks associated with those files, If they get deleted before a FBR, that ain't a

Re: Unsubscribe

Unsubscribe Thanks Niketh > On Oct 5, 2023, at 7:56 AM, Viral Mehta wrote: > >  - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org

Re: HDFS HA standby

First of all, could you please explain how you installed Hadoop? It's possible that you may have already disclosed this information in a previous thread, but please understand that I haven't gone through all of them and don't have all the details memorized. I haven't actually tried it, but I

Re: HDFS HA standby

@Kiyoshi Mizumaru How would I do that?I tried changing /hadoop/etc/hadoop/hadoop-env.sh export HADOOP_ROOT_LOGGER=TRACE,console But that did not seem to work, I still only get INFO.On Tuesday, October 3, 2023 at 09:13:13 PM PDT, Harry Jamison wrote: I am not sure exactly what the

Re: HDFS HA standby

Why don't you try to change the logging level? DEBUG or TRACE would be helpful. On Wed, Oct 4, 2023 at 1:13 PM Harry Jamison wrote: > I am not sure exactly what the problem is now. > > My namenode (and I think journal node are getting shut down. > Is there a way to tell Why it is getting the

Re: HDFS HA namenode issue

Thanks guys, I figured out what my issue was.I did not setup the ssh key correctly, it was for my user but I started the service as root. Now it is working except none of the namenodes are transitioning to active on startup, and the datanodes are not starting automatically (I think because no

Re: HDFS HA namenode issue

Why you have set this again in hdfs-site.xml at the end. dfs.namenode.rpc-address nn1:8020 Remove this and start name node again. Regards Susheel Kumar On Tue, 3 Oct 2023, 10:09 pm Harry Jamison, wrote: > OK here is where I am at now. > > When I start the namenodes, they work, but

Re: Locating frequent data blocks

Hello. I would appreciate any help on this matter. Thanks On Wed, Sep 13, 2023 at 1:30 PM Mohammad Aghanabi wrote: > Hello. > > I read in a few articles like [1] that we can obtain data block stats from > "historical data access recorder from the NameNode log file" or in another > paper it's

Re: HDFS HA namenode issue

OK here is where I am at now. When I start the namenodes, they work, but they are all in standby mode.When I start my first datanode it seems to kill one of the namenodes (the active one I assume) I am getting 2 different warnings in the namenode [2023-10-03 09:03:52,162] WARN Unable to

Re: HDFS HA namenode issue

Harry, Great question. I would say the same configurations in core-site.xml and hdfs-site.xml will be overwriting each other in some way. Glad you found the root cause. Keep going. On Tue, Oct 3, 2023 at 10:27 AM Harry Jamison wrote: > Liming > > After looking at my config, I think that

Re: HDFS HA namenode issue

The core-site.xml configuration settings will be overridden by hdfs-site.xml, mapred-site.xml, yarn-site.xml. This was like that but don't know if it is changed now. Look at your shared.edits.dir configuration. You have not set it correct across name nodes. Regards On Tue, 3 Oct 2023, 1:59 pm

Re: HDFS HA namenode issue

> Or do I just have it there mistakenly? Yes, It should be in core-site.xml It is there in the HA doc ``` fs.defaultFS - the default path prefix used by the Hadoop FS client when none is given Optionally, you may now configure the default path for Hadoop clients to use the new HA-enabled

Re: HDFS HA namenode issue

Liming  After looking at my config, I think that maybe my problem is because my  fs.defaultFS is inconsistent between hdfs-site.xml and core-site.xmlWhat does hdfs-site.xml vs core-site.xml do why is the same setting in 2 different places?Or do I just have it there mistakenly? this is what I

Re: HDFS HA namenode issue

Can you show us the configuration files? Maybe I can help you with some suggestions. On Tue, Oct 3, 2023 at 9:05 AM Harry Jamison wrote: > I am trying to setup a HA HDFS cluster, and I am running into a problem > > I am not sure what I am doing wrong, I thought I followed the HA namenode >

Re: Compare hadoop and ytsaurus

Why still investing in these old technologies? Any reasons except for not able to migrate to cloud because of non-availabilty and data residency requirements. How much is Hadoop data compatibility (parquet and HBase data), code compatibility of UDFs, megastore migration etc.. Thanks Susheel

Re: Compare hadoop and ytsaurus

On Thu, Sep 28, 2023 at 7:31 PM Kirill wrote: > > Hi everyone! > > Have you seen this platform https://ytsaurus.tech/platform-overview ? Yes ;-) I was pretty involved in a few Open Source projects that came out of Yandex recently. > What do you think? Has somebody tried it? Their CLA is weird

Re: Compare hadoop and ytsaurus

Hey Kirill, Thanks for sharing! I wasn't aware of this project. According to the blog post https://medium.com/yandex/ytsaurus-exabyte-scale-storage-and-processing-system-is-now-open-source-42e7f5fa5fc6 It was released in public earlier this year by Yandex. It was inspired by Google's MapReduce,

Re: Deploy multi-node Hadoop with Docker

Hi Nikos, I think you are talking about the documentation in the overview section of the docker image: https://hub.docker.com/r/apache/hadoop I just wrote that 2-3 Months back particularly for dev purposes not for any prod use case, you should change those values accordingly. The docker-compose

Re: Deploy multi-node Hadoop with Docker

The Hadoop's docker image is not for production use. That's why But we should update that if people are thinking to use it for production. Not familiar with docker compose but contributions welcomed: https://github.com/apache/hadoop/blob/docker-hadoop-3/docker-compose.yaml On Fri, Sep 22, 2023

Re:

Well sending this unsubscribe won’t do anything, send a mail to: user-unsubscr...@hadoop.apache.org And for any other individual, if you want to unsubscribe, the above mail id does that. Not this one!!! It is mentioned over here as well: https://hadoop.apache.org/mailing_lists.html -Ayush On

Re: Hadoop execution failure

What is the bug here? Connection reset by peer, mostly n/w issue or the client aborted the connection.What were you executing? Is this intermittent? What is the state of the task that you ran? Is it happening for all operations or few?Mostly this ain’t a bug but some issue with your

Re: Question about getSchema method in SFTPFileSystem

SFTPFileSystem was introduced in HADOOP-5732. I don't see any discussion there about the getScheme() implementation, so this might not have been an intentional design choice. I think it's a bug. Are you interested in contributing a patch? Chris Nauroth On Thu, Apr 20, 2023 at 6:00 AM Wenqi Ma

Re: Why is this node shutting down?

Thanks! On Wed, 1 Mar 2023 at 16:54, Ayush Saxena wrote: > Not related to hadoop, reach out to hbase ML > > -Ayush > > On 02-Mar-2023, at 4:17 AM, Douglas A. Whitfield > wrote: > >  > > I can see a call and response between the regionserver and the > central node, but I don't know why there

Re: Why is this node shutting down?

Not related to hadoop, reach out to hbase ML-AyushOn 02-Mar-2023, at 4:17 AM, Douglas A. Whitfield wrote:I can see a call and response between the regionserver and the central node, but I don't know why there is a shutdown happening. Do I need to raise the log

Re: Monitoring HDFS filesystem changes

Many thanks, Wei-Chiu. - Mail original - De: "Wei-Chiu Chuang" À: phi...@free.fr Cc: user@hadoop.apache.org Envoyé: Mercredi 15 Février 2023 16:50:44 Objet: Re: Monitoring HDFS filesystem changes Use the inotify api https://dev-listener.medium.com/watch-for-chang

Re: Monitoring HDFS filesystem changes

Use the inotify api https://dev-listener.medium.com/watch-for-changes-in-hdfs-800c6fb5481f https://github.com/onefoursix/hdfs-inotify-example/blob/master/src/main/java/com/onefoursix/HdfsINotifyExample.java On Wed, Feb 15, 2023 at 1:12 AM wrote: > Hello, > is there an efficient way to

Re: request open hadoop issues to create Jira tickets

Hi Liangrui, Please offer information as mentioned at link[1]. Thanks. [1] https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-RequestingforaJiraaccount Best Regards, - He Xiaoqiao On Wed, Feb 15, 2023 at 4:41 PM liang...@yy.com wrote: > > hello > Can you

Re: Monitoring HDFS filesystem changes

Hey, The best I know you can check in the HDFS Audit logs. Just copying a sample entry, 2023-02-15 14:47:30,679 [IPC Server handler 1 on default port 59514] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(8852)) - allowed=true ugi=ayushsaxena (auth:SIMPLE) ip=localhost/127.0.0.1

Re: Query on HDFS version 3.3.4

We had to revert it since it broke a lot of downstream stuff, the upgrade patch had issues.At present we know it requires Jersey upgrade as well for sure, which is in blocked state as well, and not sure what else comes up post that.So, short answer: it isn’t there in the upcoming release, nor

Re: unsubscribe

Hello Please stop spamming all of us. If you want to unsubscribe, your software folks, Google the instructions and follow it. *Instructions for this group, to unsubscribe is to send an email to : user-unsubscr...@hadoop.apache.org * You do not need to also email all of us by marking this group.

Re: unsubscribe

unsubscribe On Mon, 30 Jan, 2023, 4:58 pm Lake Chang, wrote: > unsubscribe >

Re: consistency of yarn exclude file

Yes, I expect that will work (for both yarn.resourcemanager.nodes.exclude-path and yarn.resourcemanager.nodes.include-path), using the "s3a://..." scheme to specify a file in an S3 bucket. Chris Nauroth On Tue, Jan 3, 2023 at 11:50 PM Dong Ye wrote: > Hi, All: > > For resource

Re: consistency of yarn exclude file

You can do this by pushing the same file to all Resource Managers at the same time. This is either done by (1) admins / ops via something like scp / rsync with the source file in something like git, or (b) by an installer application that keeps the source in a DB and pushes to all the nodes.

Re: stale_status_of_NM_from_standby_RM

You can only run "yarn rmadmin -refreshNodes" against the active ResourceManager instance. In an HA deployment, a standby instance would return a "not active" error if it received this call, and then the client would failover to the other instance to retry. The ResourceManagers do not synchronize

Re: Block missing due to power failure

Agree. For some reason, if you would not like to use more than one datanode (let alone datanodes across multiple racks for fault tolerance) for some non-critical usecase, it's still recommended to use hsync over the output stream for on-disk persistence (unless the single DN setup is being used

Re: Block missing due to power failure

The file was in progress? In that case this is possible, once the data gets persisted on the disk of the datanode then the data loss ain’t possible.If someone did a hflush and not hsync while writing and the power loss happens immediately after that, so in that case also I feel there is a

Re: stale_status_of_NM_from_standby_RM

Hi, Chris: Thank you very much! Yes, I am also concerned with the decommissioning of nodemanager in a Resource Manager High Availability scenario. In order to decommission a node manager, Can I add the node manager address to a standby RM exclude.xml and run "yarn refreshnodes"? Or I can

Re: stale_status_of_NM_from_standby_RM

Every NodeManager registers and heartbeats to the active ResourceManager instance, which acts as the source of truth for cluster node status. If the active ResourceManager terminates, then another becomes active, and every NodeManager will start a new connection to register and heartbeat with that

Re: Unsubscribe

Unsubscribe On Mon, Dec 19, 2022 at 7:26 PM Gabriel James wrote: > > > -- > > *Gabriel James, PhD* > > Director > > > > *Heliase * > > > > This correspondence is for the named person’s use only. It may contain > information that is confidential, proprietary or the subject of legal > privilege.

Re: unsubscribe

unsubscribe On Sat, 17 Dec 2022 at 17:22, Agron Cela wrote: > unsubscribe > -- *Gabriel James, PhD* Director *Heliase * This correspondence is for the named person’s use only. It may contain information that is confidential, proprietary or the subject of legal privilege. No

Re: Hadoop 2 to Hadoop 3 Rolling Upgrade feasibility

FYI, We are trying to upgrade from 2.10 to 3.3. On Fri, Dec 16, 2022 at 10:20 AM Nishtha Shah wrote: > Hi team, > > While I am checking on feasible upgrade plans for this major upgrade, A > quick check if someone was able to perform a successful rolling upgrade > from Hadoop 2 to Hadoop 3. > >

Re: dfs.namenode.blockreport.queue Full of frequently，It may be related to the datanode Capacity

Hi, Is it happening regularly? kind of with regular FBR's, in that case you need to configure your Datanode's block report interval high enough and in a way that all of them don't bombard the namenode at same time and there is enough gap between FBR's from the datanodes. If it is happening with

Re: subscribe

Send a mail to user-subscr...@hadoop.apache.org-AyushOn 01-Dec-2022, at 11:31 AM, fanyuping [范育萍] wrote:Hi Community,    I’d like to subscribe to this mailing list.Best RegardsYuping FanB‹CB•È[œÝXœØÜšX™KK[XZ[ˆ

RE: Vulnerability query on Hadoop

Thank you Ayush Regards, Deepti Sharma PMP® & ITIL From: Ayush Saxena Sent: 29 November 2022 16:27 To: Deepti Sharma S Cc: user@hadoop.apache.org Subject: Re: Vulnerability query on Hadoop Hi Deepti, The OkHttp one I think got sorted as part of HDFS-16453, It is there in Hadoop-3

Re: Vulnerability query on Hadoop

Hi Deepti, The OkHttp one I think got sorted as part of HDFS-16453, It is there in Hadoop-3.3.4(Released), Second, netty is also upgraded as part of HADOOP-18079 and is also there in Hadoop-3.3.4, I tried to grep on the dependency tree of 3.3.4 and didn't find 4.1.42. If you still see it let me

Re: hdfs dfsadmin -printTopology The target of the information may be abnormal

What you are trying to achieve via that extra parameter can easily be done using GenericOptions, use the -fs and specify the namenode and port for which you want to get the results[1] check the overview [2] here to see how to use them. the second point doesn't make sense, fetch from all return

Re: hdfs dfsadmin -printTopology The target of the information may be abnormal

I agree with you, and I wonder if there is anything that can be done to help managers look at possible problems in this area I have two ideas: 1. Add a namenodeIp parameter to hdfs dfsadmin-printTopology to obtain rack information about the specified namenode. 2. Add debug

Re: Re: hdfs dfsadmin -printTopology The target of the information may be abnormal

If some sort of debugging is going on which doubts topological misconfiguration, you anyway need to check all the namenodes, if one namenode is misconfigured and if another is not. Maybe the issue won't surface if the properly configured namenode is the Active namenode at that time, but one

Re: hdfs dfsadmin -printTopology The target of the information may be abnormal

In a stable cluster, usually all the datanodes report to all the namenodes and mostly the information would be more or less same in all namenodes. This isn't data which goes stale you might land up in some mess, and moreover these aren't user commands but Admin commands, it is pre assumed that

Re: HDFS space quota exception

Is this cluster using snapshots? I'm not sure if this completely explains what you're seeing, but there were several bugs in accounting of space consumption by snapshots prior to 2.8.0, for example: https://issues.apache.org/jira/browse/HDFS-7728 https://issues.apache.org/jira/browse/HDFS-9063

Re: Unsubscribe

Unsubscribe On Friday, November 4, 2022 at 12:33:44 AM EDT, rajachivuk...@yahoo.com.INVALID wrote: Unsubscribe Sent from Yahoo Mail on Android On Thu, 3 Nov 2022 at 16:34, rajila2008 wrote: Unsubscribe On Sun, 16 Oct, 2022, 4:47 AM Manish Verma, wrote: Please unsubscribe

Re: Unsubscribe

Unsubscribe Sent from Yahoo Mail on Android On Thu, 3 Nov 2022 at 16:34, rajila2008 wrote: Unsubscribe On Sun, 16 Oct, 2022, 4:47 AM Manish Verma, wrote: Please unsubscribe me from this account.

Re: issue when enable gpu isolation

Also when we directly use container-executor command to put something into devices.deny, it report unexpected operation code. test@ip:/opt/hadoop-3.3.0$ sudo -U yarn /opt/hadoop-3.3.0/bin/container-executor --module-gpu --container_id container_e57_1667177358230_0650_01_01 -excluded_gpus

Re: unsubscribe

https://hadoop.apache.org/mailing_lists.html As described here,you can unsubscribe by sending an email to user-unsubscr...@hadoop.apache.org. (That's a general pattern for all ASF mailing lists.) Chris Nauroth On Sat, Oct 29, 2022 at 1:14 AM Vara Prasad Beerakam < mr.b.varapra...@gmail.com>

RE: CVE-2022-42889

Thank you for sharing the link, however when is the plan to release version 3.3.5 which has the fix of this CVE? Regards, Deepti Sharma PMP® & ITIL From: Wei-Chiu Chuang Sent: 27 October 2022 21:21 Cc: user@hadoop.apache.org Subject: Re: CVE-2022-42889 1. HADOOP-18497&l

Re: CVE-2022-42889

1. HADOOP-18497 On Thu, Oct 27, 2022 at 4:45 AM Deepti Sharma S wrote: > Hello Team, > > > > As we have received the vulnerability “CVE-2022-42889”. We are using > Apache Hadoop common 3pp version 3.3.3 which has transitive dependency of

Re: HDFS DataNode unavailable

Hello, I think broadly there could be 2 potential root cause explanations: 1. Logs are routed to a volume that is too small to hold the expected logging. You can review configuration settings in log4j.properties related to the rolling file appender. This determines how large logs can get and how

Re: Make hadoop not listen on public network interface

Hey Pratyush, If you're talking specifically about YARN, have you tried modifying the yarn.resourcemanager.hostname property in yarn-default.xml (at least in version 2.10.x)? Cheers, Malcolm McFarland Cavulus Malcolm McFarland Cavulus This correspondence is from HealthPlanCRM, LLC, d/b/a

Re: Performance with large no of files

Do you have security enabled? We did some preliminary benchmarks around webhdfs (i really want to revisit it again) and with security enabled, a lot of overhead is between client and KDC (SPENGO). Try run webhdfs using delegation tokens should help remove that bottleneck. On Sat, Oct 8, 2022 at

Re: Performance with large no of files

Not sure, what's your backup approach. One option can be archiving[1] the files which were done for yarn logs[2]. To Speed on this, you can write one mapreduce job for archiving the files. Please refer to the Document for sample mapreduce[3].

Re: Performance with large no of files

Using DistCp is the only option AFAIK. Distcp does support webhdfs, then try playing with the number of mappers and so to tune it for better performance -Ayush > On 09-Oct-2022, at 8:56 AM, Abhishek wrote: > >  > Hi, > We want to backup large no of hadoop small files (~1mn) with webhdfs API

Re: Communicating between yarn and tasks after delegation token renewal

There’s no way to do that. Once YARN launches containers, it doesn’t communicate with them for anything after that. The tasks / containers can obviously always reach out to YARN services. But even that in this case is not helpful because YARN never exposes through APIs what it is doing with

Re: Communicating between yarn and tasks after delegation token renewal

Hi Vinod, Thank you for the quick response. To give you more context, I would like to reduce the load on the remote target service. My idea was that, when YARN renews the token, the remote service would also return an updated piece of data to YARN. Then YARN would somehow share that piece of

Re: hdfs commands available in Spark?

I should further clarify. First, here is a correction to the link in my last email: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tools/GetGroupsBase.java Additionally, there is a nuance in that "hdfs groups" is really contacting

Re: hdfs commands available in Spark?

I expect you'd be able to fork a separate process and run any hadoop/hdfs commands that you'd like. If you're coding in Java, then I expect you can code to APIs to accomplish this goal, basically doing the equivalent of what the HDFS CLI does now:

  1   2   3   4   5   6   7   8   9   10   >