On 16/11/11 14:52, stephen mulcahy wrote:
So, digging further - hadoop seems to want to create a file
//jobToken
for each job I submit.
I assume this file is related to the new security stuff. Can I disable
this activity until I require the security functionality or can I get
this file create
On Thu, Nov 17, 2011 at 12:59 PM, ke yuan wrote:
> yes ,you're right,but
> 1)waste of disk space ,this is not right,this will not waster the disk
> space of datanode,if you don't believe ,you can see the code
Agree that this is wrong, there should be zero wastage. You only store
what you have, no
yes ,you're right,but
1)waste of disk space ,this is not right,this will not waster the disk
space of datanode,if you don't believe ,you can see the code
2) difficulty to balance HDFS,this may be true
3) low Map stage data locality; why?
2011/11/17 He Chen
> Hi Jay Vyas
>
> Ke yuan's method may
Hi guys !
Q> I see that createCache() method of JobInProgress is involved in
assignment of input splits across nodes in Hadoop.
Which classes are involved in assignment of input splits of jobs to nodes ?
I am interested in modifying this assignment policy. How can i do it ?
Q> How can i access
Hi Jay Vyas
Ke yuan's method may decrease the number of mapper because in default
the number of mapper for a job = the number of blocks in this job's input
file.
Make sure you only change the block size for your specific job's input
file. Not Hadoop cluster's configuration.
If you change the bl
just the blocksize 128M or 256M,it may reduce the number of mappers per job
2011/11/17 Jay Vyas
> Hi guys : In a shared cluster environment, whats the best way to reduce the
> number of mappers per job ? Should you do it with inputSplits ? Or simply
> toggle the values in the JobConf (i.e. inc
After a month's hiatus for Hadoop World, we're back! The December Hadoop
meetup will be held Wednesday, December 14, from 6pm to 8pm. This meetup
will be hosted by Splunk at their office on Brannan St.
As usual, we will use the discussion-based "unconference" format. At the
beginning of the meetup
The HBase-Writer team is happy to announce that HBase-Writer 0.90.3 is
available for download:
http://code.google.com/p/hbase-writer/downloads/list
HBase-Writer 0.90.3 is a maintenance release that fixes library
compatibility since older
versions of Heritrix and HBase. More details may be fou
Hi guys : In a shared cluster environment, whats the best way to reduce the
number of mappers per job ? Should you do it with inputSplits ? Or simply
toggle the values in the JobConf (i.e. increase the number of bytes in an
input split) ?
--
Jay Vyas
MMSB/UCHC
Hi all,
I'm wondering if there is a way to get output messages that are printed
from the main class of a Hadoop job.
Usually "2>&1>> out.log" would wok, but in this case it only saves the
output messages printed in the main class before starting the job.
What I want is the output messages th
We will be adding more memory into our master node in the near future.
We generally don't mind if our map/reduce jobs are unable to run for a
short period but we are more concerned about the impact this may have on
our HBase cluster. Will HBase continue to work will hadoops name-node
and/or HMa
On 16/11/11 15:13, Harsh J wrote:
Hello Stephen,
This is surely a bug. Could you file a new JIRA for this?
But I feel its pretty strange that the native libs do not come packed
inside their dedicated native folder and are instead mixed around with
.jars under lib/ itself. Right now these files
Hello Stephen,
This is surely a bug. Could you file a new JIRA for this?
But I feel its pretty strange that the native libs do not come packed
inside their dedicated native folder and are instead mixed around with
.jars under lib/ itself. Right now these files appear to be duplicated
as well, and
Solved it. IIUC that is because, by default, the conf/ subdirectory is
not part of classpath in 0.23. You need to specify it using the
--config switch:
$ hdfs --config ~/hadoop/conf/ namenode
whereas before you'd have typed
$ hadoop namenode
On Wed, Nov 16, 2011 at 1:33 PM, Petru Dimulescu
wr
So, digging further - hadoop seems to want to create a file
//jobToken
for each job I submit.
I assume this file is related to the new security stuff. Can I disable
this activity until I require the security functionality or can I get
this file created somewhere else?
Or should the permiss
On 16/11/11 14:07, stephen mulcahy wrote:
On 14/11/11 20:46, Raj V wrote:
Hi Stephen
THis is probably happening during jobtracker start. Can you provide
any relevant logs from the task tracker log fiile?
You are correct, there is even a helpful message
2011-11-16 15:05:58,076 WARN org.apache
On 14/11/11 20:46, Raj V wrote:
Hi Stephen
THis is probably happening during jobtracker start. Can you provide any
relevant logs from the task tracker log fiile?
You are correct, there is even a helpful message
2011-11-16 15:05:58,076 WARN org.apache.hadoop.mapred.JobTracker:
Incorrect perm
Hello,
I had a simple one-node configuration for 0.20, base of two files,
located in conf/:
core-site.xml :
fs.default.name
hdfs://192.168.1.107:8999
hdfs-site.xml :
dfs.data.dir
/stuff/hadoop/data
dfs.n
Oh dear, I feel such a fool. However, in the spirit of knowledge-sharing I
thought I’d pass back my results (I hate it
when I find a thread where somebody has exactly the same problem I’m having and
they then just close it by saying
they’ve fixed it, without saying *how*).
It seems that my probl
I see my query is echoed in
https://issues.apache.org/jira/browse/HADOOP-6453?focusedCommentId=13147877&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13147877
-stephen
--
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park,
Hi,
I'm testing out native lib support on our test amd64 test cluster
running 0.20.205 running the following
./bin/hadoop jar hadoop-test-0.20.205.0.jar testsequencefile -seed 0
-count 1000 -compressType RECORD xxx -codec
org.apache.hadoop.io.compress.GzipCodec -check 2
it fails with
WARN
21 matches
Mail list logo