RE: Windows and Linux hadoop cluster

Security: Windows doesn’t have a working OpenSSL implementation Malware, Virus and other typical windows based threads Disk level permission or encryption Performance: Different thread handling per OS Yarn implementation differs, can cause negative performance Windows CPU / Core scaling isn’t th

RE: Windows and Linux hadoop cluster

Hi, That should be possible, but will have performance impacts / additional configurations and potential misbehavior. But in general, it should work for Yarn, but not for MRv1. https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/SecureContainer.html cheers, --alex -- b: mapred

Re: JobHistory location/configuration

Robert, you should also take a look at: mapreduce.jobhistory.max-age-ms configures how long the files are held (default one week). And you’re right, just wait the configured time to see the logs. > On Jan 20, 2016, at 11:21 AM, Robert Schmidtke wrote: > > Nvm, it would seem mapreduce.jobhis

Re: ANSWER PLEASE

LOL - thats gorgeous! Well spoken, Kai :) > On Oct 28, 2015, at 12:50 PM, Kai Voigt wrote: > > No, the correct answer is „Don’t cheat on a Cloudera exam“ :-) This has been > reported to certificat...@cloudera.com > > Looks like you won’t get that certificat

Re: Teragen-Terasort>=10GB fails

FSError: java.io.IOException: No space left on device check the local disks for enough free tmp space in /hadoop/ best, -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com > On 27 May 2015, at 6:29 am, Pratik Gadiya > wrote: > > > Hi All, > &g

Re: Question about Block size configuration

If you set the BS lesser then 64MB, you’ll get into Namenode issues when a larger file will be read by a client. The client will ask for every block the NN - imagine what happen when you want to read a 1TB file. The optimal BS size is 128MB. You have to have in mind, that every block will be re

Re: Access on HDFS

Per default only hdfs (superuser) has permissions to write into hdfs root (/). Just sudo into hdfs, create the root-folders, change the permissions to your user and start again. BR, -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com > On May 8, 2015, at 6:26 PM, Pra

Re: Connect c language with HDFS

That depends on the installation source (rpm, tgz or parcels). Usually, when you use parcels, libhdfs.so* should be within /opt/cloudera/parcels/CDH/lib64/ (or similar). Or just use linux' "locate" (locate libhdfs.so*) to find the library. -- Alexander Alten-Lorenz m: wget.n

Re: Connect c language with HDFS

Google: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/LibHdfs.html -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com > On May 4, 2015, at 10:57 AM, unmesha sreeveni wrote: > > Hi > Can we connect c with HDFS using cloudera hadoop

Re: YARN Exceptions

Please have a closer look at the quoted error - the user (User edhdtaesvc not found) doesnt exists in your hadoop installation, which let the MR job fail. BR, AL -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com > On Apr 25, 2015, at 4:40 PM, Kumar Jayapal wr

Re: Large number of small files

-us/articles/200865073-Loading-My-CSV-Data-into-Impala-as-a-Parquet-Table> -- Alexander Alten-Lorenz m: wget.n...@gmail.com b: mapredit.blogspot.com > On Apr 24, 2015, at 11:10 AM, Marko Dinic wrote: > > Anand, > > Thank you for your answer, but wouldn't that mean th

Re: ResourceLocalizationService: Localizer failed when running pi example

As you said, that looks like a config issue. I would spot on the NM's local scratch dir (yarn.nodemanager.local-dirs). <> But without a complete stack trace, its a blind call. BR, AL -- mapredit.blogspot.com > On Apr 18, 2015, at 6:24 PM, Fernando O. wrote: > > Hey All, > It's me again

Re: Data doesn't write in HDFS

Hi Have a closer look at: java.io.IOException: File /testing/syncfusion/C#/JMS_message.1427429746967.log.tmp could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. BR, AL > On 27 Mar 2015, at 05:48

Re: Total memory available to NameNode

responsibility of the recipient to ensure that this email is virus free, > therefore neither Peridale Ltd, its subsidiaries nor their employees accept > any responsibility. > > From: Alexander Alten-Lorenz [mailto:wget.n...@gmail.com] > Sent: 26 March 2015 10:30 > To: user@had

Re: Total memory available to NameNode

Hi Mich, the book Hadoop Operations may a good start: https://books.google.de/books?id=drbI_aro20oC&pg=PA308&lpg=PA308&dq=hadoop+memory+namenode&source=bl&ots=t_yltgk_i7&sig=_6LXkcSjfuwwqfz_kDGDi9ytgqU&hl=en&sa=X&ei=Nt8TVfn9AcjLPZyXgKAC&ved=0CFYQ6AEwBg#v=onepage&q=hadoop%20memory%20namenode&f=fals

Re: Trusted-realm vs default-realm kerberos issue

Do you have mapping rules, which tells Hadoop that the trusted realm is allowed to login? http://mapredit.blogspot.de/2015/02/hadoop-and-trusted-mitv5-kerberos-with.html BR, Alex > On 24 Mar 2015, at 18:21, Mic

Re: Impala CREATE TABLE AS AVRO Requires "Redundant" Schema - Why?

Hi, Impala is a product of Cloudera. You might request help per: https://groups.google.com/a/cloudera.org/forum/#!forum/impala-user BR, Alex > On 26 Feb 2015, at 17:15, Vitale, Tom wrote: > > I used sqoop to import an MS

Re: Ambari based uninstall

Steve, I wrote up that howto some month ago, worth to test? http://mapredit.blogspot.de/2014/06/remove-hdp-and-ambari-completely.html R, Alexander > On 26 Feb 2015, at 16:21, Steve Edison wrote: > > Team, > > I am

Re: mssql bulk copy dat files

Not per default. But you can use sqoop to offload the DBs into something-delimited text. http://mapredit.blogspot.de/2011/10/sqoop-and-microsoft-sql-server.html http://www.microsoft.com/en-us/download/details.aspx?id=27584

Re: recombining split files after data is processed

after initial processing? > > > --- > Regards, > Jonathan Aquilina > Founder Eagle Eye T > On 2015-02-23 08:05, Alexander Alten-Lorenz wrote: > >> Hi, >> >> You can use an single reducer >> (http://wiki.apache.org/hadoop/HowManyMapsAndReduce

Re: recombining split files after data is processed

Hi, You can use an single reducer (http://wiki.apache.org/hadoop/HowManyMapsAndReduces ) for smaller datasets, or ‚getmerge‘: hadoop dfs -getmerge /hdfs/path local_file_name BR, Alex > On 23 Feb 2015, at 08:00, Jonathan Aquilina wrote:

Re: java.net.UnknownHostException on one node only

Important is, that the canonical name must be the FQDN of the server, see the example. 1.1.1.1 one.one.org one namenode 1.1.1.2 two.one.og two datanode If DNS is used the system’s hostname must match the FQDN in forward as well as reverse name resolution. BR, Alex > On 22 Feb 2015, at 21:51

Re: How to Tune Hadoop Cluster from Administrator prospective

Hi, http://mapredit.blogspot.de/2014/11/hadoop-server-performance-tuning.html It covers only the Linux part, not the hadoop part himself. For that a new article will follow in the next weeks. BR, Alexander > On 21

Re: Yarn AM is abending job when submitting a remote job to cluster

okens are now used in nonsecure setups, which > explains my earlier ssl question. > > Is the solution simply to delete those staging files from the cluster? > > - rd > > > Sent from my Verizon Wireless 4G LTE smartphone > > > ---- Original message &

Re: Yarn AM is abending job when submitting a remote job to cluster

Hi, https://issues.apache.org/jira/browse/YARN-1116 Looks like that the history server received a unclean shutdown or an previous job doesn’t finished, or wasn’t cleaned up after finishing the job (2015-02-15 07:51:07,241 INFO [main] org.apache

Re: Apache Sentry

Hi, Pros: Safe and easy to use for DWH based on Hadoop (Beeline, Metastore, HiveServer2, Impala, HDFS …) Cons: works not with HCatalog, Pig, old HiveServer1, /usr/bin/hive (u have to use beeline), HBase - Hive Handler doesn’t work either BR, Alex > On 19 Feb 2015, at 02:53, Krish Donald wrot

Re: Kerberos Security in Hadoop

I wrote a AD <=> MiTv5 trust tutorial a few days ago, just for actuality. http://mapredit.blogspot.de/2015/02/hadoop-and-trusted-mitv5-kerberos-with.html BR - Alexander > On 19 Feb 2015, at 01:49, Krish Donald wr

Re: Coding mappers and reducers in java

Hi, http://www.slideshare.net/AmazonWebServices/amazon-elastic-mapreduce-deep-dive-and-best-practices-bdt404-aws-reinvent-2013 seems quite good. BR, AL > On 18 Feb 201

Re: Copying many files to HDFS

Kevin, Slurper can help here: https://github.com/alexholmes/hdfs-file-slurper BR, Alexander > On 13 Feb 2015, at 14:28, Kevin wrote: > > Hi, > > I am setting up a Hadoop cluster (CDH5.1.3) and I need to copy a thousand or > so files into

Re: Transferring security tokens to remote machines

Hi Robert, forgive me if I’m wrong, but so far as I understand Flink uses nearly the same model as HDFS (not at all). Means the master receives an action and distribute that to the workers (more or less ;)) HDFS as example uses not an push mechanism, the DN clients fetch the token from the NN

Re: commodity hardware

Typically that term means standard hardware which should be present per default in an enterprise without any extras like RAID, highSpeed NICs, dual power supply and so on. But that change more and more, since some new independent frameworks and tools enter the market, like Spark, Kafka, Storm et

Re: Failed to start datanode due to bind exception

/var/run/hdfs-sockets has to be the right permissions. Per default 755 hdfs:hdfs BR, Alexander > On 10 Feb 2015, at 19:39, Rajesh Thallam wrote: > > There are no contents in the hdfs-sockets directory > Apache Hadoop Base version if 2.5.0 (using CDH 5.3.0) > > On Tue, Feb 10, 2015 at 10:24

Re: Stopping ntpd signals SIGTERM, then causes namenode exit

The sigterm should be noted in the NN logs, whats happen around 14:00? BR, Alexander > On 10 Feb 2015, at 02:39, David chen wrote: > > Thanks for your reply. > I ever indeed misunderstand how ntpd works, so deployed the shell script on > every node to synchronize time. The reason why the sc

Re: Stopping ntpd signals SIGTERM, then causes namenode exit

I would spot on Jan 7 14:52:48 host1 ntpd[44765]: no servers reachable looks for me like an network / DNS issue. You can check per dmesg whats going on, too. BR - Alexander > On 09 Feb 2015, at 17:57, daemeon reiydelle wrote: > > Absolutely a critical error to lose the configured ntpd time

Re: tools.DistCp: Invalid arguments

I launch this command, I get >> the same error…. What path should I put here? >> >> $ hadoop distcp >> hdfs://hadoop-coc-1:50070/input1 >> hdfs://hadoop-coc-2:50070/input1 >> >> Thanks, >> On 02-02-2015 19:59, Alexander Alten

Re: tools.DistCp: Invalid arguments

adoop/ConnectionRefused > <http://wiki.apache.org/hadoop/ConnectionRefused> > On 02-02-2015 19:59, Alexander Alten-Lorenz wrote: > > > >> Have a closer look: >> >>> hdfs://hadoop-coc-2:50070/ >> No Path is given. >> >> >>>

Re: tools.DistCp: Invalid arguments

Have a closer look: > hdfs://hadoop-coc-2:50070/ No Path is given. > On 02 Feb 2015, at 20:52, xeonmailinglist wrote: > > Hi, > > I am trying to copy data using distcp but I get this error. Both hadoop > runtime are working properly. Why is this happening? > > > vagrant@hadoop-coc-1:~/Pro

Re: Multiple separate Hadoop clusters on same physical machines

http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/ Ambari based, but works quite well > On 02 Feb 2015, at 10:33, Ashish Kumar9 wrote: > > Is there any good reference material availabl

Re: Multiple separate Hadoop clusters on same physical machines

I see no possibility how federation may help to have different Clusters on _same_ machines. On top, federation isn’t production ready, since the NN can have massively issues with GC on high loaded systems, which will be the case here. To have multiple, maybe single node, clusters the best way is

Re: Start Hadoop, ERROR security.UserGroupInformation: PriviledgedActionException

Jobtracker isn’t running, the log said. Check if your script startup one. BR, AL > On 16 Jan 2015, at 21:55, Ruhua Jiang wrote: > > 15/01/14 12:21:08 ERROR security.UserGroupInformation: > PriviledgedActionException as:hpc-ruhua > cause:org.apache.hadoop.ipc.RemoteException: > org.apache.h

Re: issue about run MR job use system user in CDH5

Please post vendor specific questions to the mailinglists of the vendor: https://groups.google.com/a/cloudera.org/forum/#!forum/cdh-user Look closer at: security.UserGroupInformation: PriviledgedActionException as:hbase (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Perm

Re[2]: HBase metadata

http://docs.cascading.org/tutorials/lingual-hbase/ is the most common, I guess. Phoenix is quite new, and Kiji was to much overhead in my case. - Alex mapredit.blogspot.com -- Originalnachricht -- Von: "Martin, Nick" An: "user@hadoop.apache.org" Gesendet: 08.07.2014 19:01:27 Betreff

Re: Conversion from MongoDB to hadoop

http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-hadoop/ may help. - Alexander -- Originalnachricht -- Von: "Ranjini Rathinam" An: user@hadoop.apache.org Gesendet: 12.05.2014 07:48:42 Betreff: Conversion from MongoDB to hadoop Hi, How to convert the Mapreduce from Mon

Re: running YARN in Production

Matt, Apache Yarn is quite stable, and works in production well - so far. Additional, yarn is backward compatible, so MRv1 jobs running too. For vendor specific questions contact your distribution vendor. - Alex Sent from my iPad > On 24 Apr 2014, at 22:17, Matt K wrote: > > We run a numb

Re: State of Art in Hadoop Log aggregation

ogs themselves in a centralized > way when there is an issue without having to login to 100 different machines > and was wondering what is the state of are in this regard. > > Suggestions/Pointers are very welcome!! > > Sagar -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Sqoop and Hadoop

ning on hadoop via sqoop? > > In other words, I am using hadoop and mapreduce systems itself alone, is it > possible to import data from postgresql to that basic hadoop system via sqoop? -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Migration needed when updating within an Hadoop release

or releases in this context? Have data model changes/ > conversions/ migrations ever happened between minor releases? Yes, but not regularly. Please see my next answer. > - Is there a similar wiki page describing HBase upgrade? http://hbase.apache.org/upgrading.html Cheers, Alex -- Al

Re: Migration needed when updating within an Hadoop release

Excuse the typo, should be : Normally, when you change the >major< release, you need to upgrade HDFS (http://hadoop.apache.org/docs/stable/hdfs_user_guide.html#Upgrade+and+Rollback). This will happen when you switch major branches. On Jun 14, 2013, at 12:10 PM, Alexander Alten-Lorenz

Re: Migration needed when updating within an Hadoop release

lanned. Depends how much data you've stored. Michael has written a excellent blog post about: http://www.michael-noll.com/blog/2011/08/23/performing-an-hdfs-upgrade-of-an-hadoop-cluster/ Cheers, Alex -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re:

Hi Matteo, Are you able to add more space to your test machines? Also, what says the pi example (hadoop jar hadoop-examples pi 10 10 ? - Alex On Jun 4, 2013, at 4:34 PM, "Lanati, Matteo" wrote: > Hi again, > > unfortunately my problem is not solved. > I downloaded Hadoop v. 1.1.2a and made a

Re: Source code jar for hadoop 1.2.0

s > > Benjamin -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Apache Flume Properties File

job and good luck to you. > > > On Fri, May 24, 2013 at 2:13 PM, Raj Hadoop wrote: > Hi, > > I just installed Apache Flume 1.3.1 and trying to run a small example to > test. Can any one suggest me how can I do this? I am going through the > documentation right now. > > Thanks, > Raj > > > -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: MS sql server hadoop connector

at org.apache.sqoop.Sqoop.run(Sqoop.java:145) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) > at org.apache.sqoop.Sqoop.main(Sqoop.java:238) > > > > -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: ISSUE :Hadoop with HANA using sqoop

Hey Samir, Since you've posted this already @CDH users, please go ahead there. Cheers Alex On Feb 21, 2013, at 7:49 AM, samir das mohapatra wrote: > Harsh, > I copied whole logs and past here, It looks like only it is showing > "Caused by: com.sap" , > And One thing i did not get is

Re: Contribute to Hadoop Community

Hi, http://wiki.apache.org/hadoop/HowToContribute I forget to post this before. Cheers, Alex On Feb 19, 2013, at 8:23 AM, Varsha Raveendran wrote: > Thank you very much for a quick response. > > > On Tue, Feb 19, 2013 at 12:12 PM, Alexander Alten-Lorenz > wrote: > H

Re: Contribute to Hadoop Community

community. I have looked > through > http://blog.cloudera.com/blog/2012/12/how-to-contribute-to-apache-hadoop-projects-in-24-minutes/ > and a couple of other sites but still I am unclear on how to initiate the > process. > > Please guide me. > > > Thanks

Re: Need Information on Hadoop Cluster Set up

Infosys reserves the > right to monitor and review the content of all messages sent to or from this > e-mail > address. Messages sent to or from this e-mail address may be stored on the > Infosys e-mail system. > ***INFOSYS End of Disclaimer INFOSYS*** -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: [OT] MapR m3

More than 50 nodes not support. > please give idea, we planing to move production cluster. > > -Dhanasekaran. > > -Dhanasekaran. > Did I learn something today? If not, I wasted it. -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Support of RHEL version

hbase compaction > issue it would be all pervading as long as it si RHEL62 and above! > Am I reading it right? > -Dheeren > > -Original Message----- > From: Alexander Alten-Lorenz [mailto:wget.n...@gmail.com] > Sent: Thursday, January 24, 2013 9:41 AM > To: cdh-u...@cloud

Re: Support of RHEL version

update > Workaround: Add the following command to /etc/rc.local to disable transparent > hugepage compaction: > ?<https://ccp.cloudera.com/display/CDH4DOC/Known+Issues+and+Work+Arounds+in+CDH4> > echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag > > -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: On a lighter note

Actually "Der Untergang" ;) Alexander Alten-Lorenz http://mapredit.blogspot.com Twitter: @mapredit German Hadoop LinkedIn Group: http://goo.gl/N8pCF On Jan 18, 2013, at 23:18, Ted Dunning wrote: > Well, I think the actual name was "untergang". Same meaning. > >

Re: JobCache directory cleanup

correct way to do it? > > Thanks in advance! > > P.S. We are using CDH 4.1.1. > > -- > Best Regards > Ivan Tretyakov > > Deployment Engineer > Grid Dynamics > +7 812 640 38 76 > Skype: ivan.tretyakov > www.griddynamics.com > itretya...@griddynamics.com -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Google+ Hadoop Community

t here : >> https://plus.google.com/communities/113953784510797557961 >> >> And I request rest of the guys to go there and provide your able guidance >> and valuable help to all the beginners, like you have always done. >> >> Regards, >>Mohammad Tariq >> >> -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: Multiuser setup on Hive

as created a database data2 > > Now user2 logs into hive and he is able to see and delete database data2 > > How do I prevent this? > > Regards, > Austin -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF