Hello
Is this possible to open already existing sequence file and append to it?
I could not find any pointers and tutorials anywhere.
Can someone help me out here?
with thanks and regards
Bala
Hello
In one of my use case, i am running a hadoop job using the following command
java -cp /etc/hadoop/conf myjob.class
This command gave some error that
cannot initialize cluster. please check the configuration for
mapreduce.framework.name and the correspond server address
i understand that
. I thought it is
straightforward and many people would have attempted it and hence getting
help in the form of documentation and blog would not be problem. I spent
two days in this but still could not find a way to do this.
'
regards
rab
On Fri, Jan 16, 2015 at 10:15 AM, rab ra rab...@gmail.com
Hello,
I have a servlet program deployed in jetty server listening to the port
8080. As soon as a request arrives from a client, it parses the request,
and instantiate MR program that is to be launched in Hadoop cluster. Here,
I cannot launch the hadoop job using hadoop command as 'hadoop jar jar
Hello,
I have a simple java program that sets up a MR job. I could successfully
execute this in Hadoop infrastructure (hadoop 2x) using 'hadoop jar
myjar'. But I want to achieve the same thing using java command as below.
java className
1. How can I pass hadoop configuration to this className?
Hello,
Is there a way to append to a sequence file? I need to open a new seq file,
write something to it, close it and save. Later, I want to open again, add
some more information to that seq file and close it. Is it possible? I am
using Hadoop 2x.
Same question to MapFile too?
regards
rab
Hello
In my use case, I need to build a single big sequence file. The key value
pairs are generated by map processes and a single reducer is used to
generate the sequence file. My value is a floatwritable (a list of float
values). I use hadoop streaming 2.4. i have a mapper that prints key
Hello
I m having an issue in running one simple map reduce job.
The portion of the code is below. It gives a warning that Hadoop command
line parsing was not peformed.
This occurs despite the class implements Tool interface. Any clue?
public static void main(String[] args) throws Exception {
);
.
}
I am able to work without any errors. Please make sure that you are doing
the same code above.
On Mon, Sep 1, 2014 at 4:18 PM, rab ra rab...@gmail.com wrote:
Hello
I m having an issue in running one simple map reduce job.
The portion of the code
{
StringTokenizer st = new StringTokenizer(val.toString()); while (st.
hasMoreTokens()){ txt.set(key.fileName + st.nextToken()); context.write(
txt, count); } } }
Cheers,
Felix
On Aug 20, 2014, at 8:19 AM, rab ra rab...@gmail.com wrote:
Thanks for the response.
Yes, I know
),
getDefaultReplication(f),
getDefaultBlockSize(f));
}
On Wed, Aug 27, 2014 at 2:12 PM, rab ra rab...@gmail.com wrote:
hello
Here is d code snippet, I use to append
def outFile = ${outputFile}.txt
Path pt = new Path(${hdfsName}/${dir}/${outFile
...@pivotal.io wrote:
would you please past the code in the loop?
On Sat, Aug 23, 2014 at 2:47 PM, rab ra rab...@gmail.com wrote:
Hi
By default, it is true in hadoop 2.4.1. Nevertheless, I have set it to
true explicitly in hdfs-site.xml. Still, I am not able to achieve append.
Regards
StringTokenizer(val.toString()); while (st.
hasMoreTokens()){ txt.set(key.fileName + st.nextToken()); context.write(
txt, count); } } }
Cheers,
Felix
On Aug 20, 2014, at 8:19 AM, rab ra rab...@gmail.com wrote:
Thanks for the response.
Yes, I know wholeFileInputFormat. But i am not sure filename
Hello,
I need few clarifications for the following questions related to
sequenceFiles
1. I have a bunch of sequence file. Each file has 8 keys and corresponding
values. The values are float array bytes, and key is a name which is a
string. Now, storing these smaller files and processing is not
://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
On Sat, Aug 23, 2014 at 1:41 AM, rab ra rab...@gmail.com wrote:
Hello,
I am currently using Hadoop 2.4.1.I am running a MR job using hadoop
streaming utility.
The executable needs to write large amount of information
Hi,
1. Typically,we used to copy the slaves file all the participating nodes
though I do not have concrete theory to back up this. Atleast, this is what
I was doing in hadoop 1.2 and I am doing the same in hadoop 2x
2. I think, you should investigate the yarn GUI and see how many maps it
has
Hello,
I am currently using Hadoop 2.4.1.I am running a MR job using hadoop
streaming utility.
The executable needs to write large amount of information in a file.
However, this write is not done in single attempt. The file needs to be
appended with streams of information generated.
In the
, Aug 21, 2014 at 9:07 PM, Felix Chern idry...@gmail.com wrote:
If I were you, I’ll first generate a file with those file name:
hadoop fs -ls term_file
Then run the normal map reduce job
Felix
On Aug 21, 2014, at 1:38 AM, rab ra rab...@gmail.com wrote:
Thanks for the link
is not possible today. You may
want to write a per-task file and use that entire directory as your output.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Wed, Jul 9, 2014 at 10:42 PM, rab ra rab...@gmail.com wrote:
hello
I have one use-case that spans multiple map tasks in hadoop
://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java
Regards,
Shahab
On Wed, Aug 20, 2014 at 1:46 AM, rab ra rab...@gmail.com wrote:
Hello,
I have a use case wherein i need to process huge set of files stored in
HDFS. Those files are non-splittable
Hello,
I have a use case wherein i need to process huge set of files stored in
HDFS. Those files are non-splittable and they need to be processed as a
whole. Here, I have the following question for which I need answers to
proceed further in this.
1. I wish to schedule the map process in task
Hello,
I am trying to successfully configure hadoop 2.4.0 to run more than one map
task in a node. I have done this in hadoop 1x and I found it was
straightforward. But in Hadoop 2x, with yarn coming in, I found bit
difficult to follow the documentation. Can someone give me the link or
share some
Hello
I have one use-case that spans multiple map tasks in hadoop environment. I
use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
into a file stored in hdfs. This file is shared across all the map tasks.
Though, they all computes thier output but some of them are missing
hello
I have one use-case that spans multiple map tasks in hadoop environment. I
use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
into a file stored in hdfs. This file is shared across all the map tasks.
Though, they all computes thier output but some of them are
Hi
Can anyone please answer my query?
-Rab
-- Forwarded message --
From: rab ra rab...@gmail.com
Date: 24 Jan 2014 10:55
Subject: HDFS data transfer is faster than SCP based transfer?
To: user@hadoop.apache.org
Hello
I have a use case that requires transfer of input files from
Hello
I have a use case that requires transfer of input files from remote storage
using SCP protocol (using jSCH jar). To optimize this use case, I have
pre-loaded all my input files into HDFS and modified my use case so that it
copies required files from HDFS. So, when tasktrackers works, it
Hello,
I am facing a problem in using Hadoop RPC encryption while transfer feature
in hadoop 2.2.0. I have 3 node cluster
Service running in node 1 (master)
Resource manager
Namenode
DataNode
SecondaryNamenode
Service running in slaves ( node 2 3)
NodeManager
I am trying to make data
Unless you FileSystem's mkdir() method , i m not sure you create a folder
in hdfs
On 14 Nov 2013 11:58, unmesha sreeveni unmeshab...@gmail.com wrote:
I am trying to create a file with in in folder. but when i tried to run
this in cluster i noticed that this in folder is not within hdfs.
why
instead of the DefaultTaskController.
On Mon, Nov 11, 2013 at 10:07 PM, rab ra rab...@gmail.com wrote:
-- Forwarded message --
From: rab ra rab...@gmail.com
Date: 11 Nov 2013 20:11
Subject: Hadoop on multiple user mode
To: user@hadoop.apache.org user@hadoop.apache.org
Hi
I would like to use LinuxTaskController with hadoop 1.2.1. Accordingly i
changed the configuration. When i started my service all but tasktracker
are up. When i see the tasktracker log it says LinuxTaskController class
not found. Pls note I did not build the taskcontroler execurable.
Can
Hello
I want to configure hadoop so that it is started as user admin and more
than one user can launch job. I notice that while i submit job as a guest
user, the map process is executed as admin user. I print user home in my
main code as well as inside map process. Is there a way span map process
://wiki.apache.org/hadoop/FAQ#Does_Hadoop_require_SSH.3F
regarding your SSH confusion.
On Mon, Nov 11, 2013 at 10:21 AM, rab ra rab...@gmail.com wrote:
Hello
I have a question. To transfer the files to datanodes what protocol
hadoop
uses? SSH or http or https
--
Harsh J
--
*Thanks Regards
-- Forwarded message --
From: rab ra rab...@gmail.com
Date: 11 Nov 2013 20:11
Subject: Hadoop on multiple user mode
To: user@hadoop.apache.org user@hadoop.apache.org
Hello
I want to configure hadoop so that it is started as user admin and more
than one user can launch job. I
Hello
I have a question. To transfer the files to datanodes what protocol hadoop
uses? SSH or http or https
Hello,
I have a use case scenario wherein I need to schedule map process to
particular node. Ideally, I want to send the map processes to the node of
my interest. Is it possible ? If not, Is there any workaround? Please share
some pointers to appropriate literature in this regard
Hello
In one of my use case, I am sending map processes to large number of hadoop
nodes. Assuming that the nodes are obtained from public cloud. I would like
to ensure that the security of the nodes are not compromised. For
this,planning to implement voting mechanism wherein multiple copies, lets
-- Forwarded message --
From: rab ra rab...@gmail.com
Date: 28 Aug 2013 13:26
Subject: Issue with fs.delete
To: us...@hadoop.apache.org us...@hadoop.apache.org
Hello,
I am having a trouble in deleting a file from hdfs. I am using hadoop 1.2.1
stable release. I use the following
://host:port/path/to/file and not hdfs:/path/to/file.
On Wed, Aug 28, 2013 at 1:44 PM, rab ra rab...@gmail.com wrote:
-- Forwarded message --
From: rab ra rab...@gmail.com
Date: 28 Aug 2013 13:26
Subject: Issue with fs.delete
To: us...@hadoop.apache.org us
Hello
Any hint on how to pass parameters to mappers in 1.2.1 hadoop release?
Dear Yong,
Thanks for your elaborate answer. Your answer really make sense and I am
ending something close to it expect shared storage.
In my usecase, I am not allowed to use any shared storage system. The
reason being that the slave nodes may not be safe for hosting sensible
data. (Because,
Thanks for the reply.
I am basically exploring possible ways to work with hadoop framework for
one of my use case. I have my limitations in using hdfs but agree with the
fact that using map reduce in conjunction with hdfs makes sense.
I successfully tested wholeFileInputFormat by some googling.
Hello,
Here is the new bie question of the day.
For one of my use cases, I want to use hadoop map reduce without HDFS.
Here, I will have a text file containing a list of file names to process.
Assume that I have 10 lines (10 files to process) in the input text file
and I wish to generate 10 map
-- Forwarded message --
From: rab ra rab...@gmail.com
Date: 22 Aug 2013 15:14
Subject: Create a file in local file system in map method
To: us...@hadoop.apache.org us...@hadoop.apache.org
Hi
i am not able to create a file in my local file system from my map method.
Is there a way
Hello,
Here is the new bie question of the day.
For one of my use cases, I want to use hadoop map reduce without HDFS.
Here, I will have a text file containing a list of file names to process.
Assume that I have 10 lines (10 files to process) in the input text file
and I wish to generate 10 map
44 matches
Mail list logo
