Now I've also started YARN ( just for the sake of trying anything), the config for mapred-site.xml and yarn-site.xml are those on apache website. A *jps *command shows:
11257 NodeManager 11129 ResourceManager 11815 Jps 10620 NameNode 10966 SecondaryNameNode On Thu, Mar 5, 2015 at 10:48 AM, Azuryy Yu <azury...@gmail.com> wrote: > Can you share your core-site.xml here? > > > On Thu, Mar 5, 2015 at 4:32 PM, Alexandru Calin < > alexandrucali...@gmail.com> wrote: > >> No change at all, I've added them at the start and end of the CLASSPATH, >> either way it still writes the file on the local fs. I've also restarted >> hadoop. >> >> On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu <azury...@gmail.com> wrote: >> >>> Yes, you should do it:) >>> >>> On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin < >>> alexandrucali...@gmail.com> wrote: >>> >>>> Wow, you are so right! it's on the local filesystem! Do I have to >>>> manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable >>>> ? Like this: >>>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml >>>> ? >>>> >>>> On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu <azury...@gmail.com> wrote: >>>> >>>>> you need to include core-site.xml as well. and I think you can find >>>>> '/tmp/testfile.txt' on your local disk, instead of HDFS. >>>>> >>>>> if so, My guess is right. because you don't include core-site.xml, >>>>> then your Filesystem schema is file:// by default, not hdfs://. >>>>> >>>>> >>>>> >>>>> On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin < >>>>> alexandrucali...@gmail.com> wrote: >>>>> >>>>>> I am trying to run the basic libhdfs example, it compiles ok, and >>>>>> actually runs ok, and executes the whole program, but I cannot see the >>>>>> file >>>>>> on the HDFS. >>>>>> >>>>>> It is said here <http://hadoop.apache.org/docs/r1.2.1/libhdfs.html>, >>>>>> that you have to include *the right configuration directory >>>>>> containing hdfs-site.xml* >>>>>> >>>>>> My hdfs-site.xml: >>>>>> >>>>>> <configuration> >>>>>> <property> >>>>>> <name>dfs.replication</name> >>>>>> <value>1</value> >>>>>> </property> >>>>>> <property> >>>>>> <name>dfs.namenode.name.dir</name> >>>>>> <value>file:///usr/local/hadoop/hadoop_data/hdfs/namenode</value> >>>>>> </property> >>>>>> <property> >>>>>> <name>dfs.datanode.data.dir</name> >>>>>> <value>file:///usr/local/hadoop/hadoop_store/hdfs/datanode</value> >>>>>> </property></configuration> >>>>>> >>>>>> I generate my classpath with this: >>>>>> >>>>>> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/ >>>>>> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for >>>>>> subdir in "${subdirs[@]}"do >>>>>> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name >>>>>> *.jar) >>>>>> do >>>>>> export CLASSPATH=$CLASSPATH:$file >>>>>> donedone >>>>>> >>>>>> and I also add export >>>>>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my >>>>>> *hdfs-site.xml* reside. >>>>>> >>>>>> MY LD_LIBRARY_PATH = >>>>>> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server >>>>>> Code: >>>>>> >>>>>> #include "hdfs.h"#include <stdio.h>#include <string.h>#include >>>>>> <stdio.h>#include <stdlib.h> >>>>>> int main(int argc, char **argv) { >>>>>> >>>>>> hdfsFS fs = hdfsConnect("default", 0); >>>>>> const char* writePath = "/tmp/testfile.txt"; >>>>>> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, >>>>>> 0, 0, 0); >>>>>> if(!writeFile) { >>>>>> printf("Failed to open %s for writing!\n", writePath); >>>>>> exit(-1); >>>>>> } >>>>>> printf("\nfile opened\n"); >>>>>> char* buffer = "Hello, World!"; >>>>>> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, >>>>>> strlen(buffer)+1); >>>>>> printf("\nWrote %d bytes\n", (int)num_written_bytes); >>>>>> if (hdfsFlush(fs, writeFile)) { >>>>>> printf("Failed to 'flush' %s\n", writePath); >>>>>> exit(-1); >>>>>> } >>>>>> hdfsCloseFile(fs, writeFile); >>>>>> hdfsDisconnect(fs); >>>>>> return 0;} >>>>>> >>>>>> It compiles and runs without error, but I cannot see the file on HDFS. >>>>>> >>>>>> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit. >>>>>> >>>>>> Any ideas on this ? >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
