I want to use map-reduce to sample data by some conditions. If I find
enough data, I want to output it to reducer and stop all
mappers(including running and not started). Is there any method to do
this?
-
To unsubscribe, e-mail:
I want to setup a yarn cluster. It's not allowed to visit from
public network so I changed many configuration files to let all
service bind to the internal network address.
All thing is fine exception the nodemanager which bind
0.0.0.0:13562. I searched the default-mapred-default.xml and
it says can't find tools at
Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/Classes/classes.jar
My JAVA_HOME is /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/home.
I can find tools.jar in $JAVA_HOME/lib directory.
Why it try to find tools.jar in
hi all,
I know it's not a problem related to hadoop but administrator can
not find any clues.
I have a machine with 24 core and 64GB memory with ubuntu 12.04
LTS. we use virtual box to create 4 virtual machine. Each vm has 10GB
memory and 6 core.
I have setup a small hadoop 1.2.1
I want to output different key ranges to different directory.
As of old api, there is a MultipleTextOutputFormat. I just need
rewrite generateFileNameForKeyValue.
But I can't find it in new api.
There is a MultipleOutputs. But it's not that good because it need
predefine keys by
2g
On Thu, Apr 3, 2014 at 1:30 PM, Stanley Shi s...@gopivotal.com wrote:
This doesn't seem like related with the data size.
How much memory do you use for the reducer?
Regards,
*Stanley Shi,*
On Thu, Apr 3, 2014 at 8:04 AM, Li Li fancye...@gmail.com wrote:
I have a map reduce
*mapred.child.java.opts=-Xmx2g*
On Thu, Apr 3, 2014 at 5:10 PM, Li Li fancye...@gmail.com wrote:
2g
On Thu, Apr 3, 2014 at 1:30 PM, Stanley Shi s...@gopivotal.com wrote:
This doesn't seem like related with the data size.
How much memory do you use for the reducer?
Regards,
*Stanley
.
On Thu, Apr 3, 2014 at 5:10 PM, Li Li fancye...@gmail.com wrote:
*mapred.child.java.opts=-Xmx2g*
On Thu, Apr 3, 2014 at 5:10 PM, Li Li fancye...@gmail.com wrote:
2g
On Thu, Apr 3, 2014 at 1:30 PM, Stanley Shi s...@gopivotal.com wrote:
This doesn't seem like related with the data size
the reducer hangs and the log is:
2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_00_0 copy failed:
attempt_201402271518_0260_m_01_1 from localhost.localdomain
2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
of localhost may not
right.
Make sure localhost.localdomain can be resolved.
On Wed, Apr 2, 2014 at 2:15 PM, Li Li fancye...@gmail.com wrote:
the reducer hangs and the log is:
2014-04-02 14:03:57,235 WARN org.apache.hadoop.mapred.ReduceTask:
attempt_201402271518_0260_r_00_0 copy failed
I have a map reduce program that do some matrix operations. in the
reducer, it will average many large matrix(each matrix takes up
400+MB(said by Map output bytes). so if there 50 matrix to a reducer,
then the total memory usage is 20GB. so the reduce task got exception:
FATAL
I have a program that do some map-reduce job and then read the result
of the job.
I learned that hdfs is not strong consistent. when it's safe to read the result?
as long as output/_SUCCESS exist?
, Dieter
2014-03-28 9:33 GMT+01:00 Li Li fancye...@gmail.com:
I have a program that do some map-reduce job and then read the result
of the job.
I learned that hdfs is not strong consistent. when it's safe to read the
result?
as long as output/_SUCCESS exist?
for io.compression.codecs config parameter ?
Thanks
On Tue, Feb 11, 2014 at 10:11 PM, Li Li fancye...@gmail.com wrote:
I am runing example of wordcout but encount an exception:
I googled and know lzo compression's license is incompatible with apache's
so it's not built in.
the question is I am using
thanks. it's correct now.
On Thu, Feb 13, 2014 at 9:37 AM, Ted Yu yuzhih...@gmail.com wrote:
Please remove LzoCodec from config.
Cheers
On Feb 12, 2014, at 5:12 PM, Li Li fancye...@gmail.com wrote:
property
nameio.compression.codecs/name
I am runing example of wordcout but encount an exception:
I googled and know lzo compression's license is incompatible with apache's
so it's not built in.
the question is I am using default configuration of hadoop 1.2.1, why
it need lzo?
anothe question is, what's Cleaning up the staging area
hi all,
I am trying to set up a hadoop cluster. But when I use start-all.sh to
start it. it throws exception:
2012-11-17 10:40:21,662 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:work cause:org.apache.hadoop.hdfs.server.namenode.SafeModeException:
Dear developer,
I am just making research using zookeeper as a load
balancer.Recently,I plan to test the max load it can handle.But I have
some confuse about which I must consult to you .
Now I can handle about 300 clients with one server,when I set the
session time out is 3.
In
18 matches
Mail list logo
