Fwd: Hadoop 3.2.2 Compilation Error

Hi, This issue is resolved (missed some library in my machine). Thanks -- Forwarded message - From: Ascot Moss Date: Tue, Apr 13, 2021 at 2:13 PM Subject: Hadoop 3.2.2 Compilation Error To: Hi, I got compilation error when building Hadoop 3.2.2 with -Pnative Please help

Hadoop 3.2.2 Compilation Error

Hi, I got compilation error when building Hadoop 3.2.2 with -Pnative Please help! [*INFO*] ** [*INFO*] *Reactor Summary for Apache Hadoop Main 3.2.2:* [*INFO*] [*INFO*] Apache Hadoop Main

yarn.app.mapreduce.am.staging-dir, to use local file system

Hi, Any idea how to set *yarn.app.mapreduce.am.staging-dir*to have it to use local shared folder (non-hdfs)? Regards

Re: YARN on non-hdfs Error

Hi, Can "yarn.app.mapreduce.am.staging-dir" be set to use Linux shared mount point? if yes, is the following correct? mapred-site.xml yarn.app.mapreduce.am.staging-dir file:/share_mnt/tmp where "share_mnt" is the shared folder that can be accessed by all nodes. Please help!

Re: YARN on non-hdfs Error

at 5:39 PM, Ascot Moss wrote: > Hi, > > I found that yarn can be used on non-hdfs file system. like AWS s3: or gfs > > I am trying yarn on localFS, *I use "yarn" user to start yarn on localFS > (which is non-HDFS), and have other users to submit t

YARN on non-hdfs Error

Hi, I found that yarn can be used on non-hdfs file system. like AWS s3: or gfs I am trying yarn on localFS, *I use "yarn" user to start yarn on localFS (which is non-HDFS), and have other users to submit their own jobs, I got error* on *yarn.app.mapreduce.am.staging-dir permission:*

How to fix "HDFS Missing replicas"

Hi, After running 'hdfs fsck /blocks' to check the cluster, I got 'Missing replicas: 441 (0.24602923 %)" How to fix HDFS missing replicas? Regards (detailed output) Status: HEALTHY Total size:3375617914739 B (Total open files size: 68183613174 B) Total dirs:2338

Hadoop KMS, security module

Hi, I am studying Hadoop KMS and encryption, I understand that Hadoop KMS is proxy of security module, have some questions and need help: Q1. Is there a reference list about Hardware Security Modules which support Hadoop KMS? Q2. Any suggestion about (open source) software security modules that

Apache Hadoop Statistics

Hi, Is there any information or links about how many install-base of Apache Hadoop? e.g. daily download number, number of installed clusters of Apache Hadoop (not MapR, not Hortnworks, not Cloudera editions) Regards

MapReduce Job State: PREP over 8 hours, state no change

Hi, I have submitted a mapreduce job, and can find it from job list, however I find its STATE is PREP over last 8 hours, any idea why it takes so long to "PREP"? regards (mapred job -list) JobId State StartTime UserName Queue Priority UsedContainers

Re: HDFS2 vs MaprFS

d in turn all the data for the volume > (e.g. under the directory) is stored on the storage hardware assigned to > the topology. > > These topological labels provide the same benefits as dfs.storage.policy > as well as enabling additional types of use cases. > > On Mon,

Re: HDFS2 vs MaprFS

ored in MapRFS) to the topology, and hence the data in > that subset would be served by whatever hardware was mapped into the > topology. > > >no to mention that Mapr-FS loses Data-Locality. > > This statement is false. > > > > On Mon, Jun 6, 2016 at 8:32 AM, Asco

Re: HDFS2 vs MaprFS

, 2016 at 11:26 PM, Ascot Moss <ascot.m...@gmail.com> wrote: > I don't think HDFS2 needs SAN, use the QuorumJournal approach is much > better than using Shared edits directory SAN approach. > > > > > On Monday, June 6, 2016, Peyman Mohajerian <mohaj...@gmail.com&

Re: HDFS2 vs MaprFS

t setting up a data infrastructure, I would say consider >> alternatives before you pick HDFS. If you run in AWS, S3 is a good >> alternative. If you run in some other cloud, it's probably worth >> considering whatever their equivalent storage system is. >> >> >> On Sat,

Re: HDFS2 vs MaprFS

rations HDFS > filesystem is read from disk to RAM and rest of the work is done with RAM. > So RAM should be big enough to fit the filesystem image. But HDFS has > configuration options like har files (Hadoop Archive) to defeat these > limitations. > > On Sun, Jun 5, 2016 at 11:14

Re: HDFS2 vs MaprFS

~25 PB of storage. At Facebook, HDFS has around >> 2600 nodes, 300 million files and blocks, addressing up to 60PB of storage. >> While these are very large systems and good enough for majority of Hadoop >> users, a few deployments that might want to grow even larger could find the >

HDFS2 vs MaprFS

Hi, I read some (old?) articles from Internet about Mapr-FS vs HDFS. https://www.mapr.com/products/m5-features/no-namenode-architecture It states that HDFS Federation has a) "Multiple Single Points of Failure", is it really true? Why MapR uses HDFS but not HDFS2 in its comparison as this would

Maximum number of files in Hadoop v2

Hi, I read some (old?) articles from Internet about hadoop: "Due to the DataNode-NameNode block report mechanism, we cannot exceed 100-200K blocks (or files) per node, thereby limiting our 10-node cluster to less than 2M files." Is this true in Hadoop v2.x? Regards

Start and Stop Namenode

Hi, I am new to 2.2.0, after running the following command to start the first namenode, I used jps to check the cluster: ./sbin/hadoop-daemon.sh --script hdfs start namenode starting namenode, logging to

Re: Start and Stop Namenode

NameNode regards On 28 Nov, 2013, at 1:25 pm, Harsh J ha...@cloudera.com wrote: Yes you should expect to see a NameNode separately available but apparently its dying out. Check the NN's log on that machine to see why. On Thu, Nov 28, 2013 at 8:37 AM, Ascot Moss ascot.m...@gmail.com wrote

Which version of Hadoop

Hi, I am new to Hadoop, from Hadoop download I can find 4 versions: 1.0.x / 1.1.x / 2.x.x / 0.23.x May I know which one is the latest stable version that provides Namenode high availability for production environment? regards