RE: Windows and Linux hadoop cluster

Security: Windows doesn’t have a working OpenSSL implementation Malware, Virus and other typical windows based threads Disk level permission or encryption Performance: Different thread handling per OS Yarn implementation differs, can cause negative performance Windows CPU / Core scaling isn’t

RE: Windows and Linux hadoop cluster

Hi, That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1. https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html cheers, --alex -- b:

Re: JobHistory location/configuration

Robert, you should also take a look at: mapreduce.jobhistory.max-age-ms configures how long the files are held (default one week). And you’re right, just wait the configured time to see the logs. > On Jan 20, 2016, at 11:21 AM, Robert Schmidtke wrote: > > Nvm, it

Re: ANSWER PLEASE

LOL - thats gorgeous! Well spoken, Kai :) > On Oct 28, 2015, at 12:50 PM, Kai Voigt wrote: > > No, the correct answer is „Don’t cheat on a Cloudera exam“ :-) This has been > reported to certificat...@cloudera.com > > Looks like you won’t get

Re: Teragen-Terasort=10GB fails

FSError: java.io.IOException: No space left on device check the local disks for enough free tmp space in /hadoop/ best, -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com On 27 May 2015, at 6:29 am, Pratik Gadiya pratik_gad...@persistent.com wrote: Hi All

Re: Question about Block size configuration

If you set the BS lesser then 64MB, you’ll get into Namenode issues when a larger file will be read by a client. The client will ask for every block the NN - imagine what happen when you want to read a 1TB file. The optimal BS size is 128MB. You have to have in mind, that every block will be

Re: Access on HDFS

Per default only hdfs (superuser) has permissions to write into hdfs root (/). Just sudo into hdfs, create the root-folders, change the permissions to your user and start again. BR, -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com On May 8, 2015, at 6:26 PM, Pratik

Re: Connect c language with HDFS

Google: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/LibHdfs.html -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com On May 4, 2015, at 10:57 AM, unmesha sreeveni unmeshab...@gmail.com wrote: Hi Can we connect c with HDFS using cloudera

Re: Connect c language with HDFS

That depends on the installation source (rpm, tgz or parcels). Usually, when you use parcels, libhdfs.so* should be within /opt/cloudera/parcels/CDH/lib64/ (or similar). Or just use linux' locate (locate libhdfs.so*) to find the library. -- Alexander Alten-Lorenz m: wget.n...@gmail.com b

Re: YARN Exceptions

Please have a closer look at the quoted error - the user (User edhdtaesvc not found) doesnt exists in your hadoop installation, which let the MR job fail. BR, AL -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com On Apr 25, 2015, at 4:40 PM, Kumar Jayapal kjayapa

Re: ResourceLocalizationService: Localizer failed when running pi example

As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs). But without a complete stack trace, its a blind call. BR, AL -- mapredit.blogspot.com On Apr 18, 2015, at 6:24 PM, Fernando O. fot...@gmail.com wrote: Hey All,

Re: Data doesn't write in HDFS

Hi Have a closer look at: java.io.IOException: File /testing/syncfusion/C#/JMS_message.1427429746967.log.tmp could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. BR, AL On 27 Mar 2015, at

Re: Total memory available to NameNode

Hi Mich, the book Hadoop Operations may a good start: https://books.google.de/books?id=drbI_aro20oCpg=PA308lpg=PA308dq=hadoop+memory+namenodesource=blots=t_yltgk_i7sig=_6LXkcSjfuwwqfz_kDGDi9ytgqUhl=ensa=Xei=Nt8TVfn9AcjLPZyXgKACved=0CFYQ6AEwBg#v=onepageq=hadoop%20memory%20namenodef=false

Re: Total memory available to NameNode

is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Alexander Alten-Lorenz [mailto:wget.n...@gmail.com] Sent: 26 March 2015 10:30 To: user@hadoop.apache.org Subject: Re: Total memory available to NameNode Hi Mich

Re: Trusted-realm vs default-realm kerberos issue

Do you have mapping rules, which tells Hadoop that the trusted realm is allowed to login? http://mapredit.blogspot.de/2015/02/hadoop-and-trusted-mitv5-kerberos-with.html http://mapredit.blogspot.de/2015/02/hadoop-and-trusted-mitv5-kerberos-with.html BR, Alex On 24 Mar 2015, at 18:21,

Re: Ambari based uninstall

Steve, I wrote up that howto some month ago, worth to test? http://mapredit.blogspot.de/2014/06/remove-hdp-and-ambari-completely.html http://mapredit.blogspot.de/2014/06/remove-hdp-and-ambari-completely.html R, Alexander On 26 Feb 2015, at 16:21, Steve Edison sediso...@gmail.com wrote:

Re: Impala CREATE TABLE AS AVRO Requires Redundant Schema - Why?

Hi, Impala is a product of Cloudera. You might request help per: https://groups.google.com/a/cloudera.org/forum/#!forum/impala-user https://groups.google.com/a/cloudera.org/forum/#!forum/impala-user BR, Alex On 26 Feb 2015, at 17:15, Vitale, Tom thomas.vit...@credit-suisse.com wrote: I

Re: recombining split files after data is processed

initial processing? --- Regards, Jonathan Aquilina Founder Eagle Eye T On 2015-02-23 08:05, Alexander Alten-Lorenz wrote: Hi, You can use an single reducer (http://wiki.apache.org/hadoop/HowManyMapsAndReduces http://wiki.apache.org/hadoop/HowManyMapsAndReduces) for smaller datasets

Re: java.net.UnknownHostException on one node only

Important is, that the canonical name must be the FQDN of the server, see the example. 1.1.1.1 one.one.org one namenode 1.1.1.2 two.one.og two datanode If DNS is used the system’s hostname must match the FQDN in forward as well as reverse name resolution. BR, Alex On 22 Feb 2015, at

Re: Kerberos Security in Hadoop

I wrote a AD = MiTv5 trust tutorial a few days ago, just for actuality. http://mapredit.blogspot.de/2015/02/hadoop-and-trusted-mitv5-kerberos-with.html http://mapredit.blogspot.de/2015/02/hadoop-and-trusted-mitv5-kerberos-with.html BR - Alexander On 19 Feb 2015, at 01:49, Krish Donald

Re: Apache Sentry

Hi, Pros: Safe and easy to use for DWH based on Hadoop (Beeline, Metastore, HiveServer2, Impala, HDFS …) Cons: works not with HCatalog, Pig, old HiveServer1, /usr/bin/hive (u have to use beeline), HBase - Hive Handler doesn’t work either BR, Alex On 19 Feb 2015, at 02:53, Krish Donald

Re: Yarn AM is abending job when submitting a remote job to cluster

Hi, https://issues.apache.org/jira/browse/YARN-1116 https://issues.apache.org/jira/browse/YARN-1058 Looks like that the history server received a unclean shutdown or an previous job doesn’t finished, or wasn’t cleaned up after finishing the job (2015-02-15 07:51:07,241 INFO [main]

Re: Coding mappers and reducers in java

Hi, http://www.slideshare.net/AmazonWebServices/amazon-elastic-mapreduce-deep-dive-and-best-practices-bdt404-aws-reinvent-2013 http://www.slideshare.net/AmazonWebServices/amazon-elastic-mapreduce-deep-dive-and-best-practices-bdt404-aws-reinvent-2013 seems quite good. BR, AL On 18 Feb 2015,

Re: Copying many files to HDFS

Kevin, Slurper can help here: https://github.com/alexholmes/hdfs-file-slurper https://github.com/alexholmes/hdfs-file-slurper BR, Alexander On 13 Feb 2015, at 14:28, Kevin kevin.macksa...@gmail.com wrote: Hi, I am setting up a Hadoop cluster (CDH5.1.3) and I need to copy a thousand

Re: Failed to start datanode due to bind exception

/var/run/hdfs-sockets has to be the right permissions. Per default 755 hdfs:hdfs BR, Alexander On 10 Feb 2015, at 19:39, Rajesh Thallam rajesh.thal...@gmail.com wrote: There are no contents in the hdfs-sockets directory Apache Hadoop Base version if 2.5.0 (using CDH 5.3.0) On Tue, Feb

Re: commodity hardware

Typically that term means standard hardware which should be present per default in an enterprise without any extras like RAID, highSpeed NICs, dual power supply and so on. But that change more and more, since some new independent frameworks and tools enter the market, like Spark, Kafka, Storm

Re: Transferring security tokens to remote machines

Hi Robert, forgive me if I’m wrong, but so far as I understand Flink uses nearly the same model as HDFS (not at all). Means the master receives an action and distribute that to the workers (more or less ;)) HDFS as example uses not an push mechanism, the DN clients fetch the token from the NN

Re: Stopping ntpd signals SIGTERM, then causes namenode exit

I would spot on Jan 7 14:52:48 host1 ntpd[44765]: no servers reachable looks for me like an network / DNS issue. You can check per dmesg whats going on, too. BR - Alexander On 09 Feb 2015, at 17:57, daemeon reiydelle daeme...@gmail.com wrote: Absolutely a critical error to lose the

Re: tools.DistCp: Invalid arguments

/ConnectionRefused http://wiki.apache.org/hadoop/ConnectionRefused On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: hdfs://hadoop-coc-2:50070/ hdfs://hadoop-coc-2:50070/ No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com

Re: tools.DistCp: Invalid arguments

this command, I get the same error…. What path should I put here? $ hadoop distcp hdfs://hadoop-coc-1:50070/input1 hdfs://hadoop-coc-2:50070/input1 Thanks, On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: Have a closer look: hdfs://hadoop-coc-2:50070/ hdfs

Re: tools.DistCp: Invalid arguments

Have a closer look: hdfs://hadoop-coc-2:50070/ No Path is given. On 02 Feb 2015, at 20:52, xeonmailinglist xeonmailingl...@gmail.com wrote: Hi, I am trying to copy data using distcp but I get this error. Both hadoop runtime are working properly. Why is this happening?

Re: Multiple separate Hadoop clusters on same physical machines

http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/ http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/ Ambari based, but works quite well On 02 Feb 2015, at 10:33, Ashish Kumar9 ashis...@in.ibm.com wrote: Is there any good reference

Re: Multiple separate Hadoop clusters on same physical machines

I see no possibility how federation may help to have different Clusters on _same_ machines. On top, federation isn’t production ready, since the NN can have massively issues with GC on high loaded systems, which will be the case here. To have multiple, maybe single node, clusters the best way

Re: Start Hadoop, ERROR security.UserGroupInformation: PriviledgedActionException

Jobtracker isn’t running, the log said. Check if your script startup one. BR, AL On 16 Jan 2015, at 21:55, Ruhua Jiang ruhua.ji...@gmail.com wrote: 15/01/14 12:21:08 ERROR security.UserGroupInformation: PriviledgedActionException as:hpc-ruhua

Re: issue about run MR job use system user in CDH5

Please post vendor specific questions to the mailinglists of the vendor: https://groups.google.com/a/cloudera.org/forum/#!forum/cdh-user Look closer at: security.UserGroupInformation: PriviledgedActionException as:hbase (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException:

Re[2]: HBase metadata

http://docs.cascading.org/tutorials/lingual-hbase/ is the most common, I guess. Phoenix is quite new, and Kiji was to much overhead in my case. - Alex mapredit.blogspot.com -- Originalnachricht -- Von: Martin, Nick nimar...@pssd.com An: user@hadoop.apache.org user@hadoop.apache.org

Re: Conversion from MongoDB to hadoop

http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-hadoop/ may help. - Alexander -- Originalnachricht -- Von: Ranjini Rathinam ranjinibe...@gmail.com An: user@hadoop.apache.org Gesendet: 12.05.2014 07:48:42 Betreff: Conversion from MongoDB to hadoop Hi, How to convert

Re: running YARN in Production

Matt, Apache Yarn is quite stable, and works in production well - so far. Additional, yarn is backward compatible, so MRv1 jobs running too. For vendor specific questions contact your distribution vendor. - Alex Sent from my iPad On 24 Apr 2014, at 22:17, Matt K matvey1...@gmail.com

Re: State of Art in Hadoop Log aggregation

without having to login to 100 different machines and was wondering what is the state of are in this regard. Suggestions/Pointers are very welcome!! Sagar -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Sqoop and Hadoop

via sqoop? In other words, I am using hadoop and mapreduce systems itself alone, is it possible to import data from postgresql to that basic hadoop system via sqoop? -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Migration needed when updating within an Hadoop release

much data you've stored. Michael has written a excellent blog post about: http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/ Cheers, Alex -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Migration needed when updating within an Hadoop release

Excuse the typo, should be : Normally, when you change the major release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches. On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz wget.n

Re: Migration needed when updating within an Hadoop release

in this context? Have data model changes/ conversions/ migrations ever happened between minor releases? Yes, but not regularly. Please see my next answer. - Is there a similar wiki page describing HBase upgrade? http://hbase.apache.org/upgrading.html Cheers, Alex -- Alexander Alten-Lorenz http

Re:

Hi Matteo, Are you able to add more space to your test machines? Also, what says the pi example (hadoop jar hadoop-examples pi 10 10 ? - Alex On Jun 4, 2013, at 4:34 PM, Lanati, Matteo matteo.lan...@lrz.de wrote: Hi again, unfortunately my problem is not solved. I downloaded Hadoop v.

Re: Source code jar for hadoop 1.2.0

Benjamin -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Apache Flume Properties File

...@yahoo.com wrote: Hi, I just installed Apache Flume 1.3.1 and trying to run a small example to test. Can any one suggest me how can I do this? I am going through the documentation right now. Thanks, Raj -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop

Re: MS sql server hadoop connector

) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) at org.apache.sqoop.Sqoop.main(Sqoop.java:238) -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Contribute to Hadoop Community

Hi, http://wiki.apache.org/hadoop/HowToContribute I forget to post this before. Cheers, Alex On Feb 19, 2013, at 8:23 AM, Varsha Raveendran varsha.raveend...@gmail.com wrote: Thank you very much for a quick response. On Tue, Feb 19, 2013 at 12:12 PM, Alexander Alten-Lorenz wget.n

Re: Contribute to Hadoop Community

. I have looked through http://blog.cloudera.com/blog/2012/12/how-to-contribute-to-apache-hadoop-projects-in-24-minutes/ and a couple of other sites but still I am unclear on how to initiate the process. Please guide me. Thanks Regards, Varsha -- Alexander Alten-Lorenz http

Re: Need Information on Hadoop Cluster Set up

this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS End of Disclaimer INFOSYS*** -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: [OT] MapR m3

. More than 50 nodes not support. please give idea, we planing to move production cluster. -Dhanasekaran. -Dhanasekaran. Did I learn something today? If not, I wasted it. -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Support of RHEL version

://ccp.cloudera.com/display/CDH4DOC/Known+Issues+and+Work+Arounds+in+CDH4 echo never /sys/kernel/mm/redhat_transparent_hugepage/defrag -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Support of RHEL version

. If this is a hbase compaction issue it would be all pervading as long as it si RHEL62 and above! Am I reading it right? -Dheeren -Original Message- From: Alexander Alten-Lorenz [mailto:wget.n...@gmail.com] Sent: Thursday, January 24, 2013 9:41 AM To: cdh-u...@cloudera.org Subject

Re: On a lighter note

Actually Der Untergang ;) Alexander Alten-Lorenz http://mapredit.blogspot.com Twitter: @mapredit German Hadoop LinkedIn Group: http://goo.gl/N8pCF On Jan 18, 2013, at 23:18, Ted Dunning tdunn...@maprtech.com wrote: Well, I think the actual name was untergang. Same meaning. Sent from my

Re: JobCache directory cleanup

. -- Best Regards Ivan Tretyakov Deployment Engineer Grid Dynamics +7 812 640 38 76 Skype: ivan.tretyakov www.griddynamics.com itretya...@griddynamics.com -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Google+ Hadoop Community

/113953784510797557961 And I request rest of the guys to go there and provide your able guidance and valuable help to all the beginners, like you have always done. Regards, Mohammad Tariq -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Multiuser setup on Hive

user2 logs into hive and he is able to see and delete database data2 How do I prevent this? Regards, Austin -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF