Re: Re: Is Hadoop basically not suitable for a photo archive?

Kai, this is great. It is well down the path to solving the small/object-as-file problem. Good show! *Daemeon C.M. ReiydelleSan Francisco 1.415.501.0198London 44 020 8144 9872* On Mon, Sep 4, 2017 at 8:56 PM, Zheng, Kai wrote: > A nice discussion about support of small

Re: DR for Data Lake

Determine what is meant by "disaster recovery". What are the scenarious, what data. Architect to the business need, not the buzz words *“Anyone who isn’t embarrassed by who they were last year probably isn’t learning enough.” - Alain de Botton* *Daemeon C.M. ReiydelleUSA (+1)

Re: reconfiguring storage

Another option is to stop the node's relevant Hadoop services (including e.g spark, impala, etc. if applicable), move the existing local storage, mount the desired file system, and move the data over. Then just restart hadoop. As long as this does not take too long, you don't have write

Re: java.io.IOException on Namenode logs

A possibility is that the node showing errors was not able to get tcp connection, or heavy network conjestion, or (possibly) heavy garbage collection tomeouts. Would suspect network ... There is no sin except stupidity - Oscar Wilde ... Daemeon (Dæmœn) Reiydelle USA 1.415.501.0198 On Jul 3, 2017

Re: Kafka or Flume

For fairly simple transformations, Flume is great, and works fine subscribing ​to some pretty ​ high volumes of messages from Kafka ​ (I think we hit 50M/second at one point)​ . If you need to do complex transformations, e.g. database lookups for the Kafka to Hadoop ETL, then you will start having

Re: Anatomy of read in hdfs

Readers ARE parallel processes, one per map task. There are defaults in map phase, about how many readers there are for the input file(s). Default is one mapper task block (or file, where any file is smaller than the hdfs block size). There is no java framework per se for splitting up an file

Re: need java + hive developer in bangalore

Hello, and thanks for reaching out to me. Let me know if the first section of the attached resume is helpful? CV attached. Currently working in San Francisco Daemeon Reiydelle *...* *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Sat, Feb 18, 2017 at 6:33

Re: Does the JobHistoryServer register itself with ZooKeeper?

JHS does NOT use ZK for any purpose. HA options of NM and RM do. That is the ONLY ZK use in Hadoop. *...* *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Wed, Nov 16, 2016 at 4:29 PM, Benson Qiu wrote: > Hi Ravi, > > Nothing gave

Re: Code does not enter mapper, reducer class

What do you mean by "does not enter ... class(es)"? Does the log show that the scheduler ever accepts the job (You may have to turn logging up)? Are "other" jobs that are submitted to the same class under your user scheduled & executed? Wonder about which scheduler? What is the definition for the

Re: Supervisely, RAID0 provides best io performance whereas no RAID the worst

Have you considered the probability (mean time to failure - not mean time TO failure) of a disk, then factor the probability is 12 times as likely with a raid 0? Then compare that the the time to replicate in degraded mode where you have such a large number of drives on each node? Secondly, there

Re: HDFS2 vs MaprFS

There are indeed many tuning points here. If the name nodes and journal nodes can be larger, perhaps even bonding multiple 10gbyte nics, one can easily scale. I did have one client where the file counts forced multiple clusters. But we were able to differentiate by airframe types ... eg fixed wing

Re: Hadoop reference architecture

"Best" practices are either so generic as to be mostly useless, or dependent on very specific business processes, SLA's, and OLA's relevant to different data models, etc. If you can identify specific business use cases? Even basic attributes change the model: Containers on CoreOS? White boxes?

Re: Hadoop on premise versus cloud

In addition to the data privacy concern described well below, there are a couple of other areas you might consider (you can also respond privately where I can be a bit more candid). . My experience with most banks (I work with most of the players in the EU and US) are such that (1) below drives

Re: Hadoop or RDBMS

Based on the brief description, which includes the relatively small number of records, type of queries I can imagine the end customer would make, my question would be how ad hoc are the queries vs. how well managed by traditional RDBMS schemas? Then I would be interested to understand the nature

Re: How will Hadoop handle it when a datanode server with total hardware failure?

With rep of 3 you would have to lose 3 entire nodes to lose data. The rep factor is 3 nodes, not 3 spindles.. The number of disks (sort of) determine how hdfs spreads io across the spindles for the single copy of the data (one of 3 nodes with copies) that the node owns. Note that things get

Re: Swap requirements

You will find much information if you Google configuring Linux page stealing. This is actually the core of the problem with swap (and throwing away pages of shared libraries). Or talk to your devops team about how to avoid page stealing in systems with large memory storage footprints ...

Re: t2.micro on AWS; Is it enough for setting up Hadoop cluster ?

Do a reverse lookup and use the name you find. There are a few areas of Hadoopo that require reverse name lookup, but in general just create relevant entries (shared across the cluster, e.g. via Ansible if more than just a few nodes) in /etc/hosts. Not hard. On Thu, Mar 5, 2015 at 6:35 PM,

Re: How to resolve--- Unauthorized request to start container. This token is expired.

Duh. try running ntpdate as root (sudo). On Fri, Feb 27, 2015 at 11:39 PM, daemeon reiydelle daeme...@gmail.com wrote: try ntpdate -b -p8 whichever server However, you flat should not be seeing 13 minutes. Something wrong. Suggest nptdate -d -b -p8 whichever server and look at the results

Re: How to resolve--- Unauthorized request to start container. This token is expired.

try ntpdate -b -p8 whichever server However, you flat should not be seeing 13 minutes. Something wrong. Suggest nptdate -d -b -p8 whichever server and look at the results. On Thu, Feb 26, 2015 at 10:54 AM, Jan van Bemmelen j...@tokyoeye.net wrote: Hi Tariq, So this is not really an Hadoop

Re: Intermittent BindException during long MR jobs

When the access fails, do you have a way to check that the utilization on the target node ... i.e. was the target node utilization at 100%? On Thu, Feb 26, 2015 at 10:30 PM, hadoop.supp...@visolve.com wrote: Hello Krishna, Exception seems to be IP specific. It might be occurred due to

Re: video stream as input to sequence files

. Presently, I convert the video files to individual framess, make a sequence file out of them and transfer the sequence file to HDFS. This flow is not optimized and I need to optimize it. On Thu, Feb 26, 2015 at 3:00 AM, daemeon reiydelle daeme...@gmail.com wrote: Can you explain your use case

Re: video stream as input to sequence files

Can you explain your use case? *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a

Re: Restarting a resource manager kills the other in HA

Only one rm will be active at a time. The other is in standby. When you started the new rm, the configuration files direct the new rm to come up and take over, the old primary will go to stand by (or should!). Working as designed except you will see slowdown in scheduling. I suspect what you want

Re: Hadoop and nslcd

look for busy network resulting in network timeouts to LDAP server (assuming LDAP itself is not overloaded). *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of

Re: Yarn AM is abending job when submitting a remote job to cluster

I would guess you do not have your ssl certs set up, client or server, based on the error. *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used

Re: hadoop cluster with non-uniform disk spec

What have you set dfs.datanode.fsdataset.volume.choosing.policy to (assuming you are on a current version of Hadoop)? Is the policy set to org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy? *...* *“Life should not be a journey to the grave with the

Re: Stopping ntpd signals SIGTERM, then causes namenode exit

Absolutely a critical error to lose the configured ntpd time source in Hadoop. The replication and many other services require absolutely millisecond time sync between the nodes. Interesting that your SRE design called for ntpd running on each node. Curious. What is the problem you are trying to

Re: Max Connect retries

Are your nodes actually stuck or are you in e.g. a reduce step that is reading so much data across the network that the node SEEMS unreachable? Since you mention gets stuck for a while at 25%, that suggests that eventually the node finishes up its work ... *...* *“Life should not be

Re: Localizer failed but it works after I restart yarn

My first concern is that temp is a virtual file system and cannot exceed real memory plus swap space. You may not see file system full errors flow up as the root cause of the stack trace, but you will see null pointers. So move your temp directory onto your file system (~/Documents/HDFSTMP or

Re: Re: Can not start HA namenode with security enabled

No Kerberos TGT was issued. This looks like an auth issue to Kerberos for e.g. user hadoop. Check your Kerb server. *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of

Re: Can I configure multiple M/Rs and normal processes to one workflow?

Null map step (at a guess?), 3 step reduce. No problem. Suspect 3 may be rather long running? *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used

Re: Can I configure multiple M/Rs and normal processes to one workflow?

. On 2015년 2월 5일 (목) at 오전 4:51 daemeon reiydelle daeme...@gmail.com wrote: Null map step (at a guess?), 3 step reduce. No problem. Suspect 3 may be rather long running? *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved

Re: Multiple separate Hadoop clusters on same physical machines

Fantastic! I was delighted to have recently worked for a large search engine company that has moved significant components of their hadoop to to docker containers on Ubunto, seeing amazing performance/density improvements. And yes, the build process is really picky. Thanks SO much! *...*

Re: Multiple separate Hadoop clusters on same physical machines

Make virtualization an option. Federation will NOT solve your problems. *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out,

Re: Hadoop Security Community

Add me to the list of interested parties. I am heavily involved with security and controls of Hadoop, Google MR, use of OSSEC on Hadoop clusters, etc. *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather

Re: After cluster rolling reboot, nodemanager could not authenticate to resource manager

Check your ip addresses and host names of the RM (could be an issue around which interface the nodes are now using?) *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud

Re: yarn cache settings

If you are running /var/lib/hadoop/tmp dir in the / file system, you may want to reconsider that. Disk IO will cause issues with the OS as it attempts to use it's file system. *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well

Re: Low-latency queries, HDFS exclusively or should I go, e.g.: MongoDB?

At the end of the day, the more data that is pulled from multiple physical nodes, the (relatively) slower your response time to respond to queries. Until you reach a point where that response time exceeds your business requirements, keep it simple. As volumes grow with distributed data sources to

Re: OpenJDK vs OracleJDK, which is preferred when running hadoop cluster?

Formally OpenJDK is not supported as of a recent posting, and I have had incompatibility issues myself. This is a work in progress so YMMV. *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in

Re: Re: How many blocks does one input split have?

There would be thousands of tasks, but not all fired off at the same time. The number of parallel tasks is configurable but typically 1 per data node core. *...* On Wed, Dec 17, 2014 at 6:31 PM, bit1...@163.com bit1...@163.com wrote: Thanks Mark and Dieter for the reply. Actually, I got

Re: What happens to data nodes when name node has failed for long time?

I found the terminology of primary and secondary to be a bit confusing in describing operation after a failure scenario. Perhaps it is helpful to think that the Hadoop instance is guided to select a node as primary for normal operation. If that node fails, then the backup becomes the new primary.

Re: Both hadoop fsck and dfsadmin can not detect missing replica in time?

Exactly HOW did you manually remove the block? sent from my mobile Daemeon C.M. Reiydelle USA 415.501.0198 London +44.0.20.8144.9872 On Nov 12, 2014 9:45 PM, sam liu samliuhad...@gmail.com wrote: Hi Experts, In my hdfs, there is a file named /tmp/test.txt belonging to 1 block with 2 replica.

Re: Datanode disk configuration

I would consider a jbod with 16-64mb stride. This would be a choice where one or more (e.g. MR) steps will be io bound. Otherwise one or more tasks will be hit with the low read/write times of having large amounts of data behind a single spindle On Nov 12, 2014 8:37 AM, Brian C. Huffman

Re: Datanode disk configuration

Yes. That is why you should consider striping across raid 0 (JBOD) *...“The race is not to the swift,nor the battle to the strong,but to those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Wed, Nov

Re: Sr.Technical Architect/Technical Architect/Sr. Hadoop /Big Data Developer for CA, GA, NJ, NY, AZ Locations_(FTE)Full Time Employment

no relo *...“The race is not to the swift,nor the battle to the strong,but to those who can see it coming and jump aside.” - Hunter ThompsonDaemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Wed, Nov 12, 2014 at 11:31 AM, Amendra Singh Gangwar

Re:

I observed what I thought was a similar problem but found that I was actually hitting the physical disks much harder/more efficiently ... I found that I had some spindles with pretty deep queue depths. When I sat back and actually measured, I found that my throughput had substantially increased,

Re: Hadoop Learning Environment

What you want as a sandbox depends on what you are trying to learn. If you are trying to learn to code in e.g PigLatin, Sqooz, or similar, all of the suggestions (perhaps excluding BigTop due to its setup complexities) are great. Laptop? perhaps but laptop's are really kind of infuriatingly slow

Re: Fwd: ETL using Hadoop

Hadoop is in effect a massively fast etl with high latency as the tradeoff. Other solutions allow different tradeoffs. And some of those occur in Map phase, some in a reduce phase (e.g. Stream or columnar stores). On Oct 7, 2014 11:32 PM, Dattatrya Moin dattatryam...@gmail.com wrote: Hi , We