Re: File is not written on HDFS after running libhdfs C API

[Azuryy Yu]

you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.

if so,  My guess is right.  because you don't include core-site.xml, then
your Filesystem schema is file:// by default, not hdfs://.



On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin 
wrote:

> I am trying to run the basic libhdfs example, it compiles ok, and actually
> runs ok, and executes the whole program, but I cannot see the file on the
> HDFS.
>
> It is said  here ,
> that you have to include *the right configuration directory containing
> hdfs-site.xml*
>
> My hdfs-site.xml:
>
> 
> 
> dfs.replication
> 1
> 
> 
>   dfs.namenode.name.dir
>   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
> 
> 
>   dfs.datanode.data.dir
>   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
> 
>
> I generate my classpath with this:
>
> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for subdir in 
> "${subdirs[@]}"do
> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name *.jar)
> do
> export CLASSPATH=$CLASSPATH:$file
> donedone
>
> and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
> where my *hdfs-site.xml* reside.
>
> MY LD_LIBRARY_PATH =
> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
> Code:
>
> #include "hdfs.h"#include #include #include 
> #include 
> int main(int argc, char **argv) {
>
> hdfsFS fs = hdfsConnect("default", 0);
> const char* writePath = "/tmp/testfile.txt";
> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 
> 0);
> if(!writeFile) {
>   printf("Failed to open %s for writing!\n", writePath);
>   exit(-1);
> }
> printf("\nfile opened\n");
> char* buffer = "Hello, World!";
> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
> strlen(buffer)+1);
> printf("\nWrote %d bytes\n", (int)num_written_bytes);
> if (hdfsFlush(fs, writeFile)) {
>printf("Failed to 'flush' %s\n", writePath);
>   exit(-1);
> }
>hdfsCloseFile(fs, writeFile);
>hdfsDisconnect(fs);
>return 0;}
>
> It compiles and runs without error, but I cannot see the file on HDFS.
>
> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>
> Any ideas on this ?
>
>
>

Re: File is not written on HDFS after running libhdfs C API

[Alexandru Calin]

Wow, you are so right! it's on the local filesystem!  Do I have to manually
specify hdfs-site.xml and core-site.xml in the CLASSPATH variable ? Like
this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?

On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu  wrote:

> you need to include core-site.xml as well. and I think you can find
> '/tmp/testfile.txt' on your local disk, instead of HDFS.
>
> if so,  My guess is right.  because you don't include core-site.xml, then
> your Filesystem schema is file:// by default, not hdfs://.
>
>
>
> On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
> alexandrucali...@gmail.com> wrote:
>
>> I am trying to run the basic libhdfs example, it compiles ok, and
>> actually runs ok, and executes the whole program, but I cannot see the file
>> on the HDFS.
>>
>> It is said  here ,
>> that you have to include *the right configuration directory containing
>> hdfs-site.xml*
>>
>> My hdfs-site.xml:
>>
>> 
>> 
>> dfs.replication
>> 1
>> 
>> 
>>   dfs.namenode.name.dir
>>   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
>> 
>> 
>>   dfs.datanode.data.dir
>>   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
>> 
>>
>> I generate my classpath with this:
>>
>> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
>> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for subdir in 
>> "${subdirs[@]}"do
>> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name 
>> *.jar)
>> do
>> export CLASSPATH=$CLASSPATH:$file
>> donedone
>>
>> and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
>> where my *hdfs-site.xml* reside.
>>
>> MY LD_LIBRARY_PATH =
>> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
>> Code:
>>
>> #include "hdfs.h"#include #include #include 
>> #include 
>> int main(int argc, char **argv) {
>>
>> hdfsFS fs = hdfsConnect("default", 0);
>> const char* writePath = "/tmp/testfile.txt";
>> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 
>> 0);
>> if(!writeFile) {
>>   printf("Failed to open %s for writing!\n", writePath);
>>   exit(-1);
>> }
>> printf("\nfile opened\n");
>> char* buffer = "Hello, World!";
>> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
>> strlen(buffer)+1);
>> printf("\nWrote %d bytes\n", (int)num_written_bytes);
>> if (hdfsFlush(fs, writeFile)) {
>>printf("Failed to 'flush' %s\n", writePath);
>>   exit(-1);
>> }
>>hdfsCloseFile(fs, writeFile);
>>hdfsDisconnect(fs);
>>return 0;}
>>
>> It compiles and runs without error, but I cannot see the file on HDFS.
>>
>> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>>
>> Any ideas on this ?
>>
>>
>>
>

Re: File is not written on HDFS after running libhdfs C API

[Azuryy Yu]

Yes,  you should do it:)

On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin 
wrote:

> Wow, you are so right! it's on the local filesystem!  Do I have to
> manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
> ? Like this:
> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
> ?
>
> On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu  wrote:
>
>> you need to include core-site.xml as well. and I think you can find
>> '/tmp/testfile.txt' on your local disk, instead of HDFS.
>>
>> if so,  My guess is right.  because you don't include core-site.xml, then
>> your Filesystem schema is file:// by default, not hdfs://.
>>
>>
>>
>> On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
>> alexandrucali...@gmail.com> wrote:
>>
>>> I am trying to run the basic libhdfs example, it compiles ok, and
>>> actually runs ok, and executes the whole program, but I cannot see the file
>>> on the HDFS.
>>>
>>> It is said  here ,
>>> that you have to include *the right configuration directory containing
>>> hdfs-site.xml*
>>>
>>> My hdfs-site.xml:
>>>
>>> 
>>> 
>>> dfs.replication
>>> 1
>>> 
>>> 
>>>   dfs.namenode.name.dir
>>>   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
>>> 
>>> 
>>>   dfs.datanode.data.dir
>>>   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
>>> 
>>>
>>> I generate my classpath with this:
>>>
>>> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
>>> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for subdir 
>>> in "${subdirs[@]}"do
>>> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name 
>>> *.jar)
>>> do
>>> export CLASSPATH=$CLASSPATH:$file
>>> donedone
>>>
>>> and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
>>> where my *hdfs-site.xml* reside.
>>>
>>> MY LD_LIBRARY_PATH =
>>> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
>>> Code:
>>>
>>> #include "hdfs.h"#include #include #include 
>>> #include 
>>> int main(int argc, char **argv) {
>>>
>>> hdfsFS fs = hdfsConnect("default", 0);
>>> const char* writePath = "/tmp/testfile.txt";
>>> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 
>>> 0, 0);
>>> if(!writeFile) {
>>>   printf("Failed to open %s for writing!\n", writePath);
>>>   exit(-1);
>>> }
>>> printf("\nfile opened\n");
>>> char* buffer = "Hello, World!";
>>> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
>>> strlen(buffer)+1);
>>> printf("\nWrote %d bytes\n", (int)num_written_bytes);
>>> if (hdfsFlush(fs, writeFile)) {
>>>printf("Failed to 'flush' %s\n", writePath);
>>>   exit(-1);
>>> }
>>>hdfsCloseFile(fs, writeFile);
>>>hdfsDisconnect(fs);
>>>return 0;}
>>>
>>> It compiles and runs without error, but I cannot see the file on HDFS.
>>>
>>> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>>>
>>> Any ideas on this ?
>>>
>>>
>>>
>>
>

Re: File is not written on HDFS after running libhdfs C API

[Alexandru Calin]

No change at all, I've added them at the start and end of the CLASSPATH,
either way it still writes the file on the local fs. I've also restarted
hadoop.

On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu  wrote:

> Yes,  you should do it:)
>
> On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin <
> alexandrucali...@gmail.com> wrote:
>
>> Wow, you are so right! it's on the local filesystem!  Do I have to
>> manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
>> ? Like this:
>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
>> ?
>>
>> On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu  wrote:
>>
>>> you need to include core-site.xml as well. and I think you can find
>>> '/tmp/testfile.txt' on your local disk, instead of HDFS.
>>>
>>> if so,  My guess is right.  because you don't include core-site.xml,
>>> then your Filesystem schema is file:// by default, not hdfs://.
>>>
>>>
>>>
>>> On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
>>> alexandrucali...@gmail.com> wrote:
>>>
 I am trying to run the basic libhdfs example, it compiles ok, and
 actually runs ok, and executes the whole program, but I cannot see the file
 on the HDFS.

 It is said  here ,
 that you have to include *the right configuration directory containing
 hdfs-site.xml*

 My hdfs-site.xml:

 
 
 dfs.replication
 1
 
 
   dfs.namenode.name.dir
   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
 
 
   dfs.datanode.data.dir
   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
 

 I generate my classpath with this:

 #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
 declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for subdir 
 in "${subdirs[@]}"do
 for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name 
 *.jar)
 do
 export CLASSPATH=$CLASSPATH:$file
 donedone

 and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
 where my *hdfs-site.xml* reside.

 MY LD_LIBRARY_PATH =
 /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
 Code:

 #include "hdfs.h"#include #include #include 
 #include 
 int main(int argc, char **argv) {

 hdfsFS fs = hdfsConnect("default", 0);
 const char* writePath = "/tmp/testfile.txt";
 hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 
 0, 0);
 if(!writeFile) {
   printf("Failed to open %s for writing!\n", writePath);
   exit(-1);
 }
 printf("\nfile opened\n");
 char* buffer = "Hello, World!";
 tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
 strlen(buffer)+1);
 printf("\nWrote %d bytes\n", (int)num_written_bytes);
 if (hdfsFlush(fs, writeFile)) {
printf("Failed to 'flush' %s\n", writePath);
   exit(-1);
 }
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}

 It compiles and runs without error, but I cannot see the file on HDFS.

 I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.

 Any ideas on this ?



>>>
>>
>

Re: File is not written on HDFS after running libhdfs C API

[Azuryy Yu]

Can you share your core-site.xml here?


On Thu, Mar 5, 2015 at 4:32 PM, Alexandru Calin 
wrote:

> No change at all, I've added them at the start and end of the CLASSPATH,
> either way it still writes the file on the local fs. I've also restarted
> hadoop.
>
> On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu  wrote:
>
>> Yes,  you should do it:)
>>
>> On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin <
>> alexandrucali...@gmail.com> wrote:
>>
>>> Wow, you are so right! it's on the local filesystem!  Do I have to
>>> manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
>>> ? Like this:
>>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
>>> ?
>>>
>>> On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu  wrote:
>>>
 you need to include core-site.xml as well. and I think you can find
 '/tmp/testfile.txt' on your local disk, instead of HDFS.

 if so,  My guess is right.  because you don't include core-site.xml,
 then your Filesystem schema is file:// by default, not hdfs://.



 On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
 alexandrucali...@gmail.com> wrote:

> I am trying to run the basic libhdfs example, it compiles ok, and
> actually runs ok, and executes the whole program, but I cannot see the 
> file
> on the HDFS.
>
> It is said  here ,
> that you have to include *the right configuration directory
> containing hdfs-site.xml*
>
> My hdfs-site.xml:
>
> 
> 
> dfs.replication
> 1
> 
> 
>   dfs.namenode.name.dir
>   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
> 
> 
>   dfs.datanode.data.dir
>   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
> 
>
> I generate my classpath with this:
>
> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for subdir 
> in "${subdirs[@]}"do
> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name 
> *.jar)
> do
> export CLASSPATH=$CLASSPATH:$file
> donedone
>
> and I also add export
> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
> *hdfs-site.xml* reside.
>
> MY LD_LIBRARY_PATH =
> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
> Code:
>
> #include "hdfs.h"#include #include #include 
> #include 
> int main(int argc, char **argv) {
>
> hdfsFS fs = hdfsConnect("default", 0);
> const char* writePath = "/tmp/testfile.txt";
> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 
> 0, 0);
> if(!writeFile) {
>   printf("Failed to open %s for writing!\n", writePath);
>   exit(-1);
> }
> printf("\nfile opened\n");
> char* buffer = "Hello, World!";
> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
> strlen(buffer)+1);
> printf("\nWrote %d bytes\n", (int)num_written_bytes);
> if (hdfsFlush(fs, writeFile)) {
>printf("Failed to 'flush' %s\n", writePath);
>   exit(-1);
> }
>hdfsCloseFile(fs, writeFile);
>hdfsDisconnect(fs);
>return 0;}
>
> It compiles and runs without error, but I cannot see the file on HDFS.
>
> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>
> Any ideas on this ?
>
>
>

>>>
>>
>

Re: File is not written on HDFS after running libhdfs C API

[Alexandru Calin]

This is how core-site.xml looks:



fs.defaultFS
hdfs://localhost:9000



On Thu, Mar 5, 2015 at 10:32 AM, Alexandru Calin  wrote:

> No change at all, I've added them at the start and end of the CLASSPATH,
> either way it still writes the file on the local fs. I've also restarted
> hadoop.
>
> On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu  wrote:
>
>> Yes,  you should do it:)
>>
>> On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin <
>> alexandrucali...@gmail.com> wrote:
>>
>>> Wow, you are so right! it's on the local filesystem!  Do I have to
>>> manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
>>> ? Like this:
>>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
>>> ?
>>>
>>> On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu  wrote:
>>>
 you need to include core-site.xml as well. and I think you can find
 '/tmp/testfile.txt' on your local disk, instead of HDFS.

 if so,  My guess is right.  because you don't include core-site.xml,
 then your Filesystem schema is file:// by default, not hdfs://.



 On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
 alexandrucali...@gmail.com> wrote:

> I am trying to run the basic libhdfs example, it compiles ok, and
> actually runs ok, and executes the whole program, but I cannot see the 
> file
> on the HDFS.
>
> It is said  here ,
> that you have to include *the right configuration directory
> containing hdfs-site.xml*
>
> My hdfs-site.xml:
>
> 
> 
> dfs.replication
> 1
> 
> 
>   dfs.namenode.name.dir
>   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
> 
> 
>   dfs.datanode.data.dir
>   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
> 
>
> I generate my classpath with this:
>
> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for subdir 
> in "${subdirs[@]}"do
> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name 
> *.jar)
> do
> export CLASSPATH=$CLASSPATH:$file
> donedone
>
> and I also add export
> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
> *hdfs-site.xml* reside.
>
> MY LD_LIBRARY_PATH =
> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
> Code:
>
> #include "hdfs.h"#include #include #include 
> #include 
> int main(int argc, char **argv) {
>
> hdfsFS fs = hdfsConnect("default", 0);
> const char* writePath = "/tmp/testfile.txt";
> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 
> 0, 0);
> if(!writeFile) {
>   printf("Failed to open %s for writing!\n", writePath);
>   exit(-1);
> }
> printf("\nfile opened\n");
> char* buffer = "Hello, World!";
> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
> strlen(buffer)+1);
> printf("\nWrote %d bytes\n", (int)num_written_bytes);
> if (hdfsFlush(fs, writeFile)) {
>printf("Failed to 'flush' %s\n", writePath);
>   exit(-1);
> }
>hdfsCloseFile(fs, writeFile);
>hdfsDisconnect(fs);
>return 0;}
>
> It compiles and runs without error, but I cannot see the file on HDFS.
>
> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>
> Any ideas on this ?
>
>
>

>>>
>>
>

Re: File is not written on HDFS after running libhdfs C API

[Alexandru Calin]

Now I've also started YARN ( just for the sake of trying anything), the
config for mapred-site.xml and yarn-site.xml are those on apache
website. A *jps
*command shows:

11257 NodeManager
11129 ResourceManager
11815 Jps
10620 NameNode
10966 SecondaryNameNode

On Thu, Mar 5, 2015 at 10:48 AM, Azuryy Yu  wrote:

> Can you share your core-site.xml here?
>
>
> On Thu, Mar 5, 2015 at 4:32 PM, Alexandru Calin <
> alexandrucali...@gmail.com> wrote:
>
>> No change at all, I've added them at the start and end of the CLASSPATH,
>> either way it still writes the file on the local fs. I've also restarted
>> hadoop.
>>
>> On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu  wrote:
>>
>>> Yes,  you should do it:)
>>>
>>> On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin <
>>> alexandrucali...@gmail.com> wrote:
>>>
 Wow, you are so right! it's on the local filesystem!  Do I have to
 manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
 ? Like this:
 CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
 ?

 On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu  wrote:

> you need to include core-site.xml as well. and I think you can find
> '/tmp/testfile.txt' on your local disk, instead of HDFS.
>
> if so,  My guess is right.  because you don't include core-site.xml,
> then your Filesystem schema is file:// by default, not hdfs://.
>
>
>
> On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
> alexandrucali...@gmail.com> wrote:
>
>> I am trying to run the basic libhdfs example, it compiles ok, and
>> actually runs ok, and executes the whole program, but I cannot see the 
>> file
>> on the HDFS.
>>
>> It is said  here ,
>> that you have to include *the right configuration directory
>> containing hdfs-site.xml*
>>
>> My hdfs-site.xml:
>>
>> 
>> 
>> dfs.replication
>> 1
>> 
>> 
>>   dfs.namenode.name.dir
>>   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
>> 
>> 
>>   dfs.datanode.data.dir
>>   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
>> 
>>
>> I generate my classpath with this:
>>
>> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
>> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for 
>> subdir in "${subdirs[@]}"do
>> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name 
>> *.jar)
>> do
>> export CLASSPATH=$CLASSPATH:$file
>> donedone
>>
>> and I also add export
>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
>> *hdfs-site.xml* reside.
>>
>> MY LD_LIBRARY_PATH =
>> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
>> Code:
>>
>> #include "hdfs.h"#include #include #include 
>> #include 
>> int main(int argc, char **argv) {
>>
>> hdfsFS fs = hdfsConnect("default", 0);
>> const char* writePath = "/tmp/testfile.txt";
>> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 
>> 0, 0, 0);
>> if(!writeFile) {
>>   printf("Failed to open %s for writing!\n", writePath);
>>   exit(-1);
>> }
>> printf("\nfile opened\n");
>> char* buffer = "Hello, World!";
>> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
>> strlen(buffer)+1);
>> printf("\nWrote %d bytes\n", (int)num_written_bytes);
>> if (hdfsFlush(fs, writeFile)) {
>>printf("Failed to 'flush' %s\n", writePath);
>>   exit(-1);
>> }
>>hdfsCloseFile(fs, writeFile);
>>hdfsDisconnect(fs);
>>return 0;}
>>
>> It compiles and runs without error, but I cannot see the file on HDFS.
>>
>> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>>
>> Any ideas on this ?
>>
>>
>>
>

>>>
>>
>

Re: File is not written on HDFS after running libhdfs C API

[Azuryy Yu]

you can try:

for file in `hadoop classpath | tr ':' ' ' | sort | uniq`   ;do
  export CLASSPATH=$CLASSPATH:$file
done


On Thu, Mar 5, 2015 at 4:48 PM, Alexandru Calin 
wrote:

> This is how core-site.xml looks:
>
> 
> 
> fs.defaultFS
> hdfs://localhost:9000
> 
> 
>
> On Thu, Mar 5, 2015 at 10:32 AM, Alexandru Calin <
> alexandrucali...@gmail.com> wrote:
>
>> No change at all, I've added them at the start and end of the CLASSPATH,
>> either way it still writes the file on the local fs. I've also restarted
>> hadoop.
>>
>> On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu  wrote:
>>
>>> Yes,  you should do it:)
>>>
>>> On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin <
>>> alexandrucali...@gmail.com> wrote:
>>>
 Wow, you are so right! it's on the local filesystem!  Do I have to
 manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
 ? Like this:
 CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
 ?

 On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu  wrote:

> you need to include core-site.xml as well. and I think you can find
> '/tmp/testfile.txt' on your local disk, instead of HDFS.
>
> if so,  My guess is right.  because you don't include core-site.xml,
> then your Filesystem schema is file:// by default, not hdfs://.
>
>
>
> On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
> alexandrucali...@gmail.com> wrote:
>
>> I am trying to run the basic libhdfs example, it compiles ok, and
>> actually runs ok, and executes the whole program, but I cannot see the 
>> file
>> on the HDFS.
>>
>> It is said  here ,
>> that you have to include *the right configuration directory
>> containing hdfs-site.xml*
>>
>> My hdfs-site.xml:
>>
>> 
>> 
>> dfs.replication
>> 1
>> 
>> 
>>   dfs.namenode.name.dir
>>   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
>> 
>> 
>>   dfs.datanode.data.dir
>>   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
>> 
>>
>> I generate my classpath with this:
>>
>> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
>> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for 
>> subdir in "${subdirs[@]}"do
>> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name 
>> *.jar)
>> do
>> export CLASSPATH=$CLASSPATH:$file
>> donedone
>>
>> and I also add export
>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
>> *hdfs-site.xml* reside.
>>
>> MY LD_LIBRARY_PATH =
>> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
>> Code:
>>
>> #include "hdfs.h"#include #include #include 
>> #include 
>> int main(int argc, char **argv) {
>>
>> hdfsFS fs = hdfsConnect("default", 0);
>> const char* writePath = "/tmp/testfile.txt";
>> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 
>> 0, 0, 0);
>> if(!writeFile) {
>>   printf("Failed to open %s for writing!\n", writePath);
>>   exit(-1);
>> }
>> printf("\nfile opened\n");
>> char* buffer = "Hello, World!";
>> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
>> strlen(buffer)+1);
>> printf("\nWrote %d bytes\n", (int)num_written_bytes);
>> if (hdfsFlush(fs, writeFile)) {
>>printf("Failed to 'flush' %s\n", writePath);
>>   exit(-1);
>> }
>>hdfsCloseFile(fs, writeFile);
>>hdfsDisconnect(fs);
>>return 0;}
>>
>> It compiles and runs without error, but I cannot see the file on HDFS.
>>
>> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>>
>> Any ideas on this ?
>>
>>
>>
>

>>>
>>
>

Re: File is not written on HDFS after running libhdfs C API

[Azuryy Yu]

You don't need to start Yarn if you only want to write HDFS using C API.
and you also don't need to restart HDFS.



On Thu, Mar 5, 2015 at 4:58 PM, Alexandru Calin 
wrote:

> Now I've also started YARN ( just for the sake of trying anything), the
> config for mapred-site.xml and yarn-site.xml are those on apache website. A 
> *jps
> *command shows:
>
> 11257 NodeManager
> 11129 ResourceManager
> 11815 Jps
> 10620 NameNode
> 10966 SecondaryNameNode
>
> On Thu, Mar 5, 2015 at 10:48 AM, Azuryy Yu  wrote:
>
>> Can you share your core-site.xml here?
>>
>>
>> On Thu, Mar 5, 2015 at 4:32 PM, Alexandru Calin <
>> alexandrucali...@gmail.com> wrote:
>>
>>> No change at all, I've added them at the start and end of the CLASSPATH,
>>> either way it still writes the file on the local fs. I've also restarted
>>> hadoop.
>>>
>>> On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu  wrote:
>>>
 Yes,  you should do it:)

 On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin <
 alexandrucali...@gmail.com> wrote:

> Wow, you are so right! it's on the local filesystem!  Do I have to
> manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
> ? Like this:
> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
> ?
>
> On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu  wrote:
>
>> you need to include core-site.xml as well. and I think you can find
>> '/tmp/testfile.txt' on your local disk, instead of HDFS.
>>
>> if so,  My guess is right.  because you don't include core-site.xml,
>> then your Filesystem schema is file:// by default, not hdfs://.
>>
>>
>>
>> On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin <
>> alexandrucali...@gmail.com> wrote:
>>
>>> I am trying to run the basic libhdfs example, it compiles ok, and
>>> actually runs ok, and executes the whole program, but I cannot see the 
>>> file
>>> on the HDFS.
>>>
>>> It is said  here ,
>>> that you have to include *the right configuration directory
>>> containing hdfs-site.xml*
>>>
>>> My hdfs-site.xml:
>>>
>>> 
>>> 
>>> dfs.replication
>>> 1
>>> 
>>> 
>>>   dfs.namenode.name.dir
>>>   file:///usr/local/hadoop/hadoop_data/hdfs/namenode
>>> 
>>> 
>>>   dfs.datanode.data.dir
>>>   file:///usr/local/hadoop/hadoop_store/hdfs/datanode
>>> 
>>>
>>> I generate my classpath with this:
>>>
>>> #!/bin/bashexport CLASSPATH=/usr/local/hadoop/
>>> declare -a subdirs=("hdfs" "tools" "common" "yarn" "mapreduce")for 
>>> subdir in "${subdirs[@]}"do
>>> for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name 
>>> *.jar)
>>> do
>>> export CLASSPATH=$CLASSPATH:$file
>>> donedone
>>>
>>> and I also add export
>>> CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
>>> *hdfs-site.xml* reside.
>>>
>>> MY LD_LIBRARY_PATH =
>>> /usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
>>> Code:
>>>
>>> #include "hdfs.h"#include #include #include 
>>> #include 
>>> int main(int argc, char **argv) {
>>>
>>> hdfsFS fs = hdfsConnect("default", 0);
>>> const char* writePath = "/tmp/testfile.txt";
>>> hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 
>>> 0, 0, 0);
>>> if(!writeFile) {
>>>   printf("Failed to open %s for writing!\n", writePath);
>>>   exit(-1);
>>> }
>>> printf("\nfile opened\n");
>>> char* buffer = "Hello, World!";
>>> tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, 
>>> strlen(buffer)+1);
>>> printf("\nWrote %d bytes\n", (int)num_written_bytes);
>>> if (hdfsFlush(fs, writeFile)) {
>>>printf("Failed to 'flush' %s\n", writePath);
>>>   exit(-1);
>>> }
>>>hdfsCloseFile(fs, writeFile);
>>>hdfsDisconnect(fs);
>>>return 0;}
>>>
>>> It compiles and runs without error, but I cannot see the file on
>>> HDFS.
>>>
>>> I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
>>>
>>> Any ideas on this ?
>>>
>>>
>>>
>>
>

>>>
>>
>

Re: File is not written on HDFS after running libhdfs C API

[Alexandru Calin]

After putting the CLASSPATH initialization in .bashrc it creates the file,
but it has 0 size and I also get this warnning:

file opened

Wrote 14 bytes
15/03/05 14:00:55 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/tmp/testfile.txt could only be replicated to 0 nodes instead of
minReplication (=1).  There are 0 datanode(s) running and no node(s) are
excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
FSDataOutputStream#close error:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/tmp/testfile.txt could only be replicated to 0 nodes instead of
minReplication (=1).  There are 0 datanode(s) running and no node(s) are
excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nat

Re: File is not written on HDFS after running libhdfs C API

[Alexandru Calin]

DataNode was not starting due to this error : java.io.IOException:
Incompatible clusterIDs in /usr/local/hadoop/hadoop_store/hdfs/datanode:
namenode clusterID = CID-b788c93b-a1d7-4351-bd91-28fdd134e9ba; datanode
clusterID = CID-862f3fad-175e-442d-a06b-d65ac57d64b2

I can't image how this happened, anyway .. I issued this command :
*bin/hdfs namenode -format -clusterId
CID-862f3fad-175e-442d-a06b-d65ac57d64b2*

And that got it started, the file is written correctly.

Thank you very much


On Thu, Mar 5, 2015 at 2:03 PM, Alexandru Calin 
wrote:

> After putting the CLASSPATH initialization in .bashrc it creates the file,
> but it has 0 size and I also get this warnning:
>
> file opened
>
> Wrote 14 bytes
> 15/03/05 14:00:55 WARN hdfs.DFSClient: DataStreamer Exception
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /tmp/testfile.txt could only be replicated to 0 nodes instead of
> minReplication (=1).  There are 0 datanode(s) running and no node(s) are
> excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1468)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy9.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy10.addBlock(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
> FSDataOutputStream#close error:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /tmp/testfile.txt could only be replicated to 0 nodes instead of
> minReplication (=1).  There are 0 datanode(s) running and no node(s) are
> excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)