Re: File is not written on HDFS after running libhdfs C API
Can you share your core-site.xml here?
On Thu, Mar 5, 2015 at 4:32 PM, Alexandru Calin alexandrucali...@gmail.com
wrote:
No change at all, I've added them at the start and end of the CLASSPATH,
either way it still writes the file on the local fs. I've also restarted
hadoop.
On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu azury...@gmail.com wrote:
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
? Like this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml,
then your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
I am trying to run the basic libhdfs example, it compiles ok, and
actually runs ok, and executes the whole program, but I cannot see the
file
on the HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory
containing hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for subdir
in ${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name
*.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
*hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0,
0, 0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
you can try:
for file in `hadoop classpath | tr ':' ' ' | sort | uniq` ;do
export CLASSPATH=$CLASSPATH:$file
done
On Thu, Mar 5, 2015 at 4:48 PM, Alexandru Calin alexandrucali...@gmail.com
wrote:
This is how core-site.xml looks:
configuration
property
namefs.defaultFS/name
valuehdfs://localhost:9000/value
/property
/configuration
On Thu, Mar 5, 2015 at 10:32 AM, Alexandru Calin
alexandrucali...@gmail.com wrote:
No change at all, I've added them at the start and end of the CLASSPATH,
either way it still writes the file on the local fs. I've also restarted
hadoop.
On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu azury...@gmail.com wrote:
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
? Like this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml,
then your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
I am trying to run the basic libhdfs example, it compiles ok, and
actually runs ok, and executes the whole program, but I cannot see the
file
on the HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory
containing hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for
subdir in ${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name
*.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
*hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT,
0, 0, 0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
This is how core-site.xml looks:
configuration
property
namefs.defaultFS/name
valuehdfs://localhost:9000/value
/property
/configuration
On Thu, Mar 5, 2015 at 10:32 AM, Alexandru Calin alexandrucali...@gmail.com
wrote:
No change at all, I've added them at the start and end of the CLASSPATH,
either way it still writes the file on the local fs. I've also restarted
hadoop.
On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu azury...@gmail.com wrote:
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
? Like this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml,
then your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
I am trying to run the basic libhdfs example, it compiles ok, and
actually runs ok, and executes the whole program, but I cannot see the
file
on the HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory
containing hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for subdir
in ${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name
*.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
*hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0,
0, 0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
Now I've also started YARN ( just for the sake of trying anything), the
config for mapred-site.xml and yarn-site.xml are those on apache
website. A *jps
*command shows:
11257 NodeManager
11129 ResourceManager
11815 Jps
10620 NameNode
10966 SecondaryNameNode
On Thu, Mar 5, 2015 at 10:48 AM, Azuryy Yu azury...@gmail.com wrote:
Can you share your core-site.xml here?
On Thu, Mar 5, 2015 at 4:32 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
No change at all, I've added them at the start and end of the CLASSPATH,
either way it still writes the file on the local fs. I've also restarted
hadoop.
On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu azury...@gmail.com wrote:
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
? Like this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml,
then your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
I am trying to run the basic libhdfs example, it compiles ok, and
actually runs ok, and executes the whole program, but I cannot see the
file
on the HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory
containing hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for
subdir in ${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name
*.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
*hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT,
0, 0, 0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
Wow, you are so right! it's on the local filesystem! Do I have to manually
specify hdfs-site.xml and core-site.xml in the CLASSPATH variable ? Like
this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml, then
your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
I am trying to run the basic libhdfs example, it compiles ok, and
actually runs ok, and executes the whole program, but I cannot see the file
on the HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory containing
hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for subdir in
${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name
*.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
where my *hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0,
0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
No change at all, I've added them at the start and end of the CLASSPATH,
either way it still writes the file on the local fs. I've also restarted
hadoop.
On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu azury...@gmail.com wrote:
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
? Like this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml,
then your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
I am trying to run the basic libhdfs example, it compiles ok, and
actually runs ok, and executes the whole program, but I cannot see the file
on the HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory containing
hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for subdir
in ${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name
*.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
where my *hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0,
0, 0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin alexandrucali...@gmail.com
wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
? Like this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml, then
your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
I am trying to run the basic libhdfs example, it compiles ok, and
actually runs ok, and executes the whole program, but I cannot see the file
on the HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory containing
hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for subdir
in ${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name
*.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
where my *hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0,
0, 0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
You don't need to start Yarn if you only want to write HDFS using C API.
and you also don't need to restart HDFS.
On Thu, Mar 5, 2015 at 4:58 PM, Alexandru Calin alexandrucali...@gmail.com
wrote:
Now I've also started YARN ( just for the sake of trying anything), the
config for mapred-site.xml and yarn-site.xml are those on apache website. A
*jps
*command shows:
11257 NodeManager
11129 ResourceManager
11815 Jps
10620 NameNode
10966 SecondaryNameNode
On Thu, Mar 5, 2015 at 10:48 AM, Azuryy Yu azury...@gmail.com wrote:
Can you share your core-site.xml here?
On Thu, Mar 5, 2015 at 4:32 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
No change at all, I've added them at the start and end of the CLASSPATH,
either way it still writes the file on the local fs. I've also restarted
hadoop.
On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu azury...@gmail.com wrote:
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs-site.xml and core-site.xml in the CLASSPATH variable
? Like this:
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml,
then your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
I am trying to run the basic libhdfs example, it compiles ok, and
actually runs ok, and executes the whole program, but I cannot see the
file
on the HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory
containing hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for
subdir in ${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name
*.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export
CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop , where my
*hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT,
0, 0, 0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on
HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml, then
your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM, Alexandru Calin alexandrucali...@gmail.com
wrote:
I am trying to run the basic libhdfs example, it compiles ok, and actually
runs ok, and executes the whole program, but I cannot see the file on the
HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html,
that you have to include *the right configuration directory containing
hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for subdir in
${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name *.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
where my *hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0,
0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
Re: File is not written on HDFS after running libhdfs C API
DataNode was not starting due to this error : java.io.IOException: Incompatible clusterIDs in /usr/local/hadoop/hadoop_store/hdfs/datanode: namenode clusterID = CID-b788c93b-a1d7-4351-bd91-28fdd134e9ba; datanode clusterID = CID-862f3fad-175e-442d-a06b-d65ac57d64b2 I can't image how this happened, anyway .. I issued this command : *bin/hdfs namenode -format -clusterId CID-862f3fad-175e-442d-a06b-d65ac57d64b2* And that got it started, the file is written correctly. Thank you very much On Thu, Mar 5, 2015 at 2:03 PM, Alexandru Calin alexandrucali...@gmail.com wrote: After putting the CLASSPATH initialization in .bashrc it creates the file, but it has 0 size and I also get this warnning: file opened Wrote 14 bytes 15/03/05 14:00:55 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/testfile.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) FSDataOutputStream#close error: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/testfile.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at
Re: File is not written on HDFS after running libhdfs C API
After putting the CLASSPATH initialization in .bashrc it creates the file, but it has 0 size and I also get this warnning: file opened Wrote 14 bytes 15/03/05 14:00:55 WARN hdfs.DFSClient: DataStreamer Exception org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/testfile.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy10.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1532) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1349) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588) FSDataOutputStream#close error: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/testfile.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1549) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3200) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:641) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy9.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399) at
File is not written on HDFS after running libhdfs C API
I am trying to run the basic libhdfs example, it compiles ok, and actually
runs ok, and executes the whole program, but I cannot see the file on the
HDFS.
It is said here http://hadoop.apache.org/docs/r1.2.1/libhdfs.html, that
you have to include *the right configuration directory containing
hdfs-site.xml*
My hdfs-site.xml:
configuration
property
namedfs.replication/name
value1/value
/property
property
namedfs.namenode.name.dir/name
valuefile:///usr/local/hadoop/hadoop_data/hdfs/namenode/value
/property
property
namedfs.datanode.data.dir/name
valuefile:///usr/local/hadoop/hadoop_store/hdfs/datanode/value
/property/configuration
I generate my classpath with this:
#!/bin/bashexport CLASSPATH=/usr/local/hadoop/
declare -a subdirs=(hdfs tools common yarn mapreduce)for
subdir in ${subdirs[@]}do
for file in $(find /usr/local/hadoop/share/hadoop/$subdir -name *.jar)
do
export CLASSPATH=$CLASSPATH:$file
donedone
and I also add export CLASSPATH=$CLASSPATH:/usr/local/hadoop/etc/hadoop ,
where my *hdfs-site.xml* reside.
MY LD_LIBRARY_PATH =
/usr/local/hadoop/lib/native:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server
Code:
#include hdfs.h#include stdio.h#include string.h#include
stdio.h#include stdlib.h
int main(int argc, char **argv) {
hdfsFS fs = hdfsConnect(default, 0);
const char* writePath = /tmp/testfile.txt;
hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 0);
if(!writeFile) {
printf(Failed to open %s for writing!\n, writePath);
exit(-1);
}
printf(\nfile opened\n);
char* buffer = Hello, World!;
tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer,
strlen(buffer)+1);
printf(\nWrote %d bytes\n, (int)num_written_bytes);
if (hdfsFlush(fs, writeFile)) {
printf(Failed to 'flush' %s\n, writePath);
exit(-1);
}
hdfsCloseFile(fs, writeFile);
hdfsDisconnect(fs);
return 0;}
It compiles and runs without error, but I cannot see the file on HDFS.
I have Hadoop 2.6.0 on Ubuntu 14.04 64bit.
Any ideas on this ?
