I did not use maven earlier for this, but if I try it with maven and the following pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi=" http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>de.test.malte</groupId> <artifactId>MapRedMaven</artifactId> <version>0.0.1-SNAPSHOT</version> <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.4.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.4.1</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-mapreduce-client-core</artifactId> <version>2.4.1</version> </dependency> </dependencies> </project> I get the following exception on both, Linux and FreeBSD: 14/08/28 14:45:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75) at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1255) at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1251) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) at org.apache.hadoop.mapreduce.Job.connect(Job.java:1250) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1279) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303) at NewMaxTemperature$NewMaxTemperatureMapper.main(NewMaxTemperature.java:39) By the way, I am now trying to run this <http://gerrymcnicol.com/index.php/2014/01/02/hadoop-and-cassandra-part-4-writing-your-first-mapreduce-job/> example on only one node. It runs fine on Linux without using maven but crashes on Linux and FreeBSD with maven with the above Exception and without maven on FreeBSD with the NoSuchElementException. Regards Malte 2014-08-27 16:44 GMT+02:00 Artem Ervits <are9...@nyp.org>: > Check your pom.xml, some artifacts changed from earlier releases. > > > Artem Ervits > Data Analyst > New York Presbyterian Hospital > > ----- Original Message ----- > From: Malte Otten [mailto:malte.maltesm...@gmail.com] > Sent: Monday, August 25, 2014 02:12 PM > To: user@hbase.apache.org <user@hbase.apache.org> > Subject: Re: NoSuchElementException while executing MapReduce-job > > Hi Ted, > > thanks for your quick answer. > I don't think I'm running in a secure setup, because I didn't do any of > the things specified in the reference guide chapter from your link and > as far as I can see, the secure setup is not the default. My > hbase-site.xml looks like the following: > > <configuration> > <property> > <name>hbase.cluster.distributed</name> > <value>true</value> > </property> > <property> > <name>hbase.master.wait.on.regionservers.mintostart</name> > <value>1</value> > </property> > <property> > <name>hbase.rootdir</name> > <value>hdfs://my.hbase.server.de:8020/hbase</value> > </property> > <property> > <name>hbase.zookeeper.property.dataDir</name> > <value>/my/path/hbase-data/zookeeper</value> > </property> > <property> > <name>hbase.zookeeper.quorum</name> > <value>my.hbase.server.de</value> > </property> > </configuration> > > Do you have any other suggestions? > > Regards > Malte > > > Am 25.08.2014 um 18:01 schrieb Ted Yu: > > Looks like you were running in a secure setup. > > > > If that's the case, you should read: > > http://hbase.apache.org/book.html#hbase.secure.configuration > > > > Cheers > > > > > > On Mon, Aug 25, 2014 at 5:48 AM, Malte Maltesmann < > > malte.maltesm...@gmail.com> wrote: > > > >> Hi all, > >> > >> when I'm trying to execute a MapReduce-Job over the HBase-Java-API I > get an > >> exception with the follwoing Stacktrace: > >> > >> java.util.NoSuchElementException > >> at java.util.StringTokenizer.nextToken(StringTokenizer.java:349) > >> at > >> > >> > org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:565) > >> at > >> > >> > org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:534) > >> at > >> > >> > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:276) > >> at > >> > >> > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:240) > >> at > >> > >> > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162) > >> at > >> > >> > org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58) > >> at > >> > >> > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265) > >> at > >> > >> > org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301) > >> at > >> > >> > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389) > >> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) > >> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at javax.security.auth.Subject.doAs(Subject.java:415) > >> at > >> > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556) > >> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282) > >> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303) > >> ... > >> > >> In the RawLocalFileSystem.java I can see, that its trying to get some > >> Permissions here, but the StringTokenizer gets an empty String, so the > >> first t.nextToken() fails with the Exception above: > >> > >> /// loads permissions, owner, and group from `ls -ld` > >> private void loadPermissionInfo() { > >> IOException e = null; > >> try { > >> String output = FileUtil.execCommand(new > File(getPath().toUri()), > >> Shell.getGetPermissionCommand()); > >> StringTokenizer t = > >> new StringTokenizer(output, Shell.TOKEN_SEPARATOR_REGEX); > >> //expected format > >> //-rw------- 1 username groupname ... > >> String permission = t.nextToken(); > >> if (permission.length() > FsPermission.MAX_PERMISSION_LENGTH) { > >> //files with ACLs might have a '+' > >> permission = permission.substring(0, > >> FsPermission.MAX_PERMISSION_LENGTH); > >> } > >> setPermission(FsPermission.valueOf(permission)); > >> t.nextToken(); > >> > >> String owner = t.nextToken(); > >> // If on windows domain, token format is DOMAIN\\user and we > want > >> to > >> // extract only the user name > >> if (Shell.WINDOWS) { > >> int i = owner.indexOf('\\'); > >> if (i != -1) > >> owner = owner.substring(i + 1); > >> } > >> setOwner(owner); > >> > >> setGroup(t.nextToken()); > >> } catch (Shell.ExitCodeException ioe) { > >> if (ioe.getExitCode() != 1) { > >> e = ioe; > >> } else { > >> setPermission(null); > >> setOwner(null); > >> setGroup(null); > >> } > >> } catch (IOException ioe) { > >> e = ioe; > >> } finally { > >> if (e != null) { > >> throw new RuntimeException("Error while running command to > get " > >> + > >> "file permissions : " + > >> > StringUtils.stringifyException(e)); > >> } > >> } > >> } > >> > >> So my question is, what could have caused this Problem. My setup is the > >> following: > >> > >> Two nodes running on FreeBSD 9.0 > >> Hadoop 2.4.1 > >> HBase 0.98.4 > >> > >> The shell-command "ls -ld" delivers the same output on FreeBSD as on any > >> other Linux-distribution, so this shouldn't be the problem. > >> Also I can use simple operations from the HBase-API like get or put > without > >> any problems. > >> > >> I also posted this on stackoverflow > >> < > >> > http://stackoverflow.com/questions/25364802/hbase-mapreduce-nosuchelementexception > >> a week ago, but didn't get any answers. > >> > >> Thanks in advance, > >> Malte > >> > > > ________________________________ > > This electronic message is intended to be for the use only of the named > recipient, and may contain information that is confidential or privileged. > If you are not the intended recipient, you are hereby notified that any > disclosure, copying, distribution or use of the contents of this message is > strictly prohibited. If you have received this message in error or are not > the named recipient, please notify us immediately by contacting the sender > at the electronic mail address noted above, and delete and destroy all > copies of this message. Thank you. > > This electronic message is intended to be for the use only of the named > recipient, and may contain information that is confidential or privileged. > If you are not the intended recipient, you are hereby notified that any > disclosure, copying, distribution or use of the contents of this message is > strictly prohibited. If you have received this message in error or are not > the named recipient, please notify us immediately by contacting the sender > at the electronic mail address noted above, and delete and destroy all > copies of this message. Thank you. >