please also set correct JAVA_HOME in hadoop-env.sh.
On Wed, Mar 25, 2015 at 1:53 PM, Anand Murali anand_vi...@yahoo.com wrote:
Dear All:
Request help/advise as I am unable to start Hadoop. Performed follow steps
in Ubuntu 14.10
1. ssh localhost
2. Did following exports in user defined
please take a look at LongWritable.java, you need to implement a similiar
class.
On Wed, Mar 18, 2015 at 3:43 PM, Vandana kumari kvandana1...@gmail.com
wrote:
Hello all
I want to pass an array of float type to mapper class. How to do that. Any
help?
--
Thanks and regards
Vandana kumari
Did you start JHS(job history server)?
On Wed, Mar 18, 2015 at 1:51 AM, Ravi Prakash ravi...@ymail.com wrote:
Perhaps yarn.resourcemanager.max-completed-applications ?
On Tuesday, March 17, 2015 10:02 AM, hitarth trivedi
t.hita...@gmail.com wrote:
Hi,
When I submit a job to yarn
Hi,
please :
run cp -a /opt/snappy/lib/libsnappy.* /opt/hadoop/hadoophome/lib/native
on each datanode. also you need to install Snappy on each datanode firstly.
On Sat, Mar 7, 2015 at 6:57 PM, donhoff_h 165612...@qq.com wrote:
Hi, experts.
I meet the following problem when configuring
tried JAVA_LIBRARY_PATH and LD_LIBRARY_PATH.
But they didn't work.
-- Original --
*From: * Azuryy Yu;azury...@gmail.com;
*Send time:* Monday, Mar 16, 2015 2:43 PM
*To:* user@hadoop.apache.orguser@hadoop.apache.org;
*Subject: * Re: Snappy Configuration
Hi,
Can you set only one reduce task? why did you want set up two reudce tasks
and only one work?
On Mon, Mar 16, 2015 at 9:04 AM, Drake민영근 drake@nexr.com wrote:
Hi,
If you write custom partitioner, just call them to confrim the key match
with which partition.
You can get the number
, 2015 at 10:22 AM, Azuryy Yu azury...@gmail.com wrote:
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs-site.xml and core-site.xml
restarted
hadoop.
On Thu, Mar 5, 2015 at 10:22 AM, Azuryy Yu azury...@gmail.com wrote:
Yes, you should do it:)
On Thu, Mar 5, 2015 at 4:17 PM, Alexandru Calin
alexandrucali...@gmail.com wrote:
Wow, you are so right! it's on the local filesystem! Do I have to
manually specify hdfs
/local/hadoop/etc/hadoop/core-site.xml
?
On Thu, Mar 5, 2015 at 10:04 AM, Azuryy Yu azury...@gmail.com wrote:
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core
-site.xml and yarn-site.xml are those on apache website. A
*jps
*command shows:
11257 NodeManager
11129 ResourceManager
11815 Jps
10620 NameNode
10966 SecondaryNameNode
On Thu, Mar 5, 2015 at 10:48 AM, Azuryy Yu azury...@gmail.com wrote:
Can you share your core-site.xml here?
On Thu, Mar 5
you need to include core-site.xml as well. and I think you can find
'/tmp/testfile.txt' on your local disk, instead of HDFS.
if so, My guess is right. because you don't include core-site.xml, then
your Filesystem schema is file:// by default, not hdfs://.
On Thu, Mar 5, 2015 at 3:52 PM,
Hi VK,
I have a similar requirement. we need a real time data analysis platform.
Actually, you don't pay more attention on the Spark or Apache Drill,
because data for LOAP cubes was calculated before cube build.
you just consider two questions:
1) how to calculate the data for cube quickly?
For HDFS federation, data share all datanodes, but namespace is separate,
so did you write some data on hadoop-coc-2 namespace? you don't need to
login hadoop-coc-2 then write data, just config a new client, which connect
to hadoop-coc-2 for write.
On Tue, Mar 3, 2015 at 6:20 PM,
That's good to know,
On Tue, Mar 3, 2015 at 8:12 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp
wrote:
Hi all,
One year after the previous post, we collected and analyzed
JIRA tickets again to investigate the activities of Apache Hadoop
community in 2014.
Hi,
http://svn.apache.org/viewcvs.cgi/hadoop/common/trunk/
I cannot open this URL. does that anybody can access it?
another, I cannot svn up the new release in branch-2. It always stay in
Aug 2014.
Oh, thanks.
On Fri, Feb 6, 2015 at 4:58 PM, Andre Kelpe ake...@concurrentinc.com
wrote:
Hadoop has moved to git: https://wiki.apache.org/hadoop/GitAndHadoop
-- André
On Fri, Feb 6, 2015 at 9:13 AM, Azuryy Yu azury...@gmail.com wrote:
Hi,
http://svn.apache.org/viewcvs.cgi/hadoop/common
please refer to :
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
On Tue, Feb 3, 2015 at 5:38 PM, Mr.J tjst...@kgrid.co.kr wrote:
my cluster A, and cluster B. To upgrade to version 2.6
In what order should I upgrade?
Journalnode 1 Journalnode
Can you build Hadoop on Linux? or can you check out the whole code from
branch-2 again, then build it?
On Thu, Feb 5, 2015 at 10:21 AM, Venkat Ramakrishnan
venkat.archit...@gmail.com wrote:
I need to compile Hadoop, and I am not able to
figure out why this error is occuring.
Can someone
please look at:
http://mail-archives.apache.org/mod_mbox/tajo-user/201502.mbox/browser
On Tue, Jan 27, 2015 at 5:13 PM, Daniel Haviv danielru...@gmail.com wrote:
Can you elaborate on why you prefer Tajo?
Daniel
On 27 בינו׳ 2015, at 10:35, Azuryy Yu azury...@gmail.com wrote:
You almost
Hi,
You cannot use new File(...) as parameter, which should be new
Path(/outputmp)
On Fri, Feb 6, 2015 at 3:51 AM, Ravi Prakash ravi...@ymail.com wrote:
Hi Xeon!
Can you try using the FileContext or FileSystem API?
HTH
Ravi
On Thursday, February 5, 2015 8:13 AM, xeonmailinglist
Are you sure you can 'cat' the lastest batch of the data on HDFS?
for Flume, the data is available only after file rolled, because Flume only
call FileSystem.close() during file rolling.
On Mon, Jan 26, 2015 at 8:17 PM, Uthayan Suthakar
uthayan.sutha...@gmail.com wrote:
I have a Flume which
You almost list all open sourced MPP real time SQL-ON-Hadoop.
I prefer Tajo, which was relased by 0.9.0 recently, and still working in
progress for 1.0
On Mon, Jan 26, 2015 at 10:19 PM, Samuel Marks samuelma...@gmail.com
wrote:
Since Hadoop https://hive.apache.org came out, there have been
Hi Frank,
can you file an issue to add this configuration to the hdfs-default.xml?
On Mon, Jan 26, 2015 at 5:39 PM, Frank Lanitz frank.lan...@sql-ag.de
wrote:
Hi,
Am 23.01.2015 um 19:23 schrieb Chris Nauroth:
The time period for determining if a datanode is dead is calculated as a
Hi,
I think the best way is deploy HDFS federation with Hadoop 2.x.
On Mon, Jan 26, 2015 at 5:18 PM, Harun Reşit Zafer
harun.za...@tubitak.gov.tr wrote:
Hi everyone,
We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
pig, hive etc.) on 7 physical servers. We want to
please check your network issue. generally this was casued by unstable
network device.
On Wed, Nov 5, 2014 at 5:55 PM, Hayden Marchant hayd...@amobee.com wrote:
I have a MapReduce job running on Hadoop 2.0.0, and on some 'heavy' jobs,
I am seeing the following errors in the reducer.
please check your mapred-site.xml is available under conf.
On Tue, Oct 21, 2014 at 2:47 PM, ch huang justlo...@gmail.com wrote:
hi,maillist:
my cluster move from one IDC to another IDC ,when all done ,i run
job ,and find the job run on local box not on cluster ,why? it is normal on
yes, you can.
stop the cluster,
change your hdfs-site.xml on your datanode. (dfs.datanode.dir) to the large
volume.
copy two small data volumes to the large volumes, which was configured on
above.
start cluster.
then you are done.
On Tue, Oct 21, 2014 at 9:57 PM, Brian C. Huffman
after upgrade, there are current an previous subdirs in the namenode
metadata directory. so during rollback, just delete current directory and
rename previous to current again.
On Tue, Oct 21, 2014 at 9:27 AM, Manoj Samel manojsamelt...@gmail.com
wrote:
Is the pre-upgrade metadata also kept
Did you specified how many map tasks?
On Fri, Oct 17, 2014 at 4:58 PM, ch huang justlo...@gmail.com wrote:
hi,maillist:
i now use distcp to migrate data from CDH4.4 to CDH5.1 , i
find when copy small file,it very good, but when transfer big data ,it very
slow ,any good method
just stop your cluster, then start your HDFS with '-rollback'. but it's
only if you don't finalize HDFS upgrade using command line.
On Fri, Oct 17, 2014 at 8:15 AM, Manoj Samel manojsamelt...@gmail.com
wrote:
Hadoop 2.4.0 mentions that FSImage is stored using protobuf. So upgrade
from 2.3.0 to
Hadoop streaming is the best option for you. It doesn't has high I/O
overhead if you don't add a high I/O in your c++ code.
hadoop streaming use buidin MapReduce, it just redirect input/out stream
for your c++ application.
On Tue, Oct 14, 2014 at 10:33 PM, Y. Z. zhaoyansw...@gmail.com wrote:
yes. it always supports hadoop pipe in v2.
On Wed, Oct 15, 2014 at 11:33 AM, Y Z zhaoyansw...@gmail.com wrote:
Thanks, Azuryy!
I found some examples about Pipes. Is Hadoop Pipes still support in Hadoop
2.2?
Sincerely,
Yongan
On 10/14/2014 11:20 PM, Azuryy Yu wrote:
Hadoop streaming
Hi Moin,
Yes, you can replace your ETL using hadoop. but it would be a big change.
such as data collection, pre-process, ETL tasks rewrite etc.
I don't think there is .NET API in Hadoop.
On Wed, Oct 8, 2014 at 2:31 PM, Dattatrya Moin dattatryam...@gmail.com
wrote:
Hi ,
We have our own ETL
Hi Brian,
Did you try set dfs.datanode.fsdataset.volume.choosing.policy to
org.apache.hadoop.hdfs.server.datanode.fsdataset.AvailableSpaceVolumeChoosingPolicy
then there are some other related options with this policy. you can google
it.
On Wed, Oct 8, 2014 at 4:44 PM, Aitor Cedres
yes, all blockpool directory names are all same is expected if you don't
configure HDFS federation.
On Wed, Oct 8, 2014 at 7:39 PM, Brian C. Huffman
bhuff...@etinternational.com wrote:
Hmmm.. It seems that there's only one block pool per disk. So that
won't help me. :-(
Also, I see the
I think Brian gave the answer.
On Tue, Oct 7, 2014 at 9:13 PM, Brian C. Huffman
bhuff...@etinternational.com wrote:
What about setting the dfs.datanode.fsdataset.volume.choosing.policy to
org.apache.hadoop.hdfs.server. datanode.fsdataset.
AvailableSpaceVolumeChoosingPolicy?
Would that
To make sure your dfs.namenode.name.dir is by default.
then, how did you find /user exists? hdfs dfs -ls ? or you checked
dfs.datanode.data.dir?
if the latter, then don't worry.
On Wed, Oct 8, 2014 at 11:56 AM, Tianyin Xu t...@cs.ucsd.edu wrote:
Hi,
I wanna run some experiments on Hadoop
Hi Alex,
you can try Apache Flume.
On Wed, May 7, 2014 at 10:48 AM, Alex Lee eliy...@hotmail.com wrote:
Sensors' may send tcpip data to server. Each sensor may send tcpip data
like a stream to the server, the quatity of the sensors and the data rate
of the data is high.
Firstly, how the
resolved, I've filed a JIRA for this.
https://issues.apache.org/jira/browse/YARN-1998
On Tue, Apr 29, 2014 at 6:46 PM, Azuryy Yu azury...@gmail.com wrote:
Hi,
how to change the time zone of startTime and finishTime on the yarn web ui?
I cannot find the code, I just found render() returns
Hi,
what's mean of HBASE_REGIONSERVER_MLOCK? I cannot find the doucment for it.
there is only Uncomment and adjust to keep all the Region Server pages
mapped to be memory resident in the hbase-env.sh, can you explain in
detail?
Thanks for any inputs.
On Thu, Apr 24, 2014 at 5:46 PM, Ted Yu yuzhih...@gmail.com wrote:
Please take a look at https://issues.apache.org/jira/browse/HBASE-6567
Cheers
On Apr 24, 2014, at 3:33 AM, Azuryy Yu azury...@gmail.com wrote:
Hi,
what's mean of HBASE_REGIONSERVER_MLOCK? I cannot find the doucment
There is because your HDFS has no space left. please check your datanodes
are all started. also please check dfs.datanode.du.reserved in
hdfs-site.xml to make sure you don't reserve large capacity.
On Fri, Apr 18, 2014 at 7:42 AM, Shengjun Xin s...@gopivotal.com wrote:
Did you start datanode
Hadoop 2.4.0 doesn't has the known issue now. I think it's a stable release
even if it's not in the stable download list. the only one issue I met is
that you should upgrade Hive to Hive-0.12.0 after upgrade to 2.4.0 for the
API compatible.
On Fri, Apr 18, 2014 at 1:07 AM, MrAsanjar .
Do you want add -Xmx4g to your MR tasks? if so, just add it as
mapred.child.java.opts in the mapred-site.xml
On Fri, Apr 18, 2014 at 9:35 AM, Andy Srine andy.sr...@gmail.com wrote:
Quick question. How would I pass the following JVM option to the Hadoop
command line?
-Xmx4G
hadoop jar jar
Please use: hadoop jar myjob.jar myjob.MyJob input output
On Tue, Apr 15, 2014 at 3:06 PM, laozh...@sina.cn laozh...@sina.cn wrote:
Hello EveryOne:
I am new to hadoop,and i am reading Hadoop in action.
When i tried to run a demo from this book,I got a problem and could not
find answer
Please use: hadoop jar myjob.jar myjob.MyJob input output
On Tue, Apr 15, 2014 at 3:06 PM, laozh...@sina.cn laozh...@sina.cn wrote:
Hello EveryOne:
I am new to hadoop,and i am reading Hadoop in action.
When i tried to run a demo from this book,I got a problem and could not
find answer
. all mapper ends with 143.
I've no idea what it means
2014-04-03 8:45 GMT+08:00 Azuryy Yu azury...@gmail.com:
Hi,
Does it normal for each container end with TERMINATED(143) ?
The whole MR job is successful, but all containers in the map phase end
with 143.
There are no any useful logs
Thanks Ravi, I am using Graphviz as Jeff said. it's enough to me.
On Thu, Apr 3, 2014 at 5:12 AM, Ravi Prakash ravi...@ymail.com wrote:
Hi Azuryy!
You have to use dot to convert it to png
On Tuesday, April 1, 2014 6:38 PM, Azuryy Yu azury...@gmail.com wrote:
Hi,
I compiled Yarn
Hi Krishna,
Don't worrry about that, there is no ssh 0.0.0.0 during start NM, it's a
local service.
On Fri, Apr 4, 2014 at 9:12 AM, Shengjun Xin s...@gopivotal.com wrote:
you can use yarn-daemon.sh to start nodemanager without ssh
On Thu, Apr 3, 2014 at 10:36 PM, Krishna Kishore Bonagiri
Hi,
Does it normal for each container end with TERMINATED(143) ?
The whole MR job is successful, but all containers in the map phase end
with 143.
There are no any useful logs in the NM, AM, Container logs.
Another minor question:
There are only WARN logs in the stderr:
log4j:WARN No appenders
There is no special patch format, just using svn diff or git diff to
generate the patch, which depeds on your code repository.
On Tue, Apr 1, 2014 at 5:14 PM, Avinash Kujur avin...@gmail.com wrote:
hello everyone,
Is there any kind of specific format of the patch in which we need to
write
There is no special patch format, just using svn diff or git diff to
generate the patch, which depeds on your code repository.
On Tue, Apr 1, 2014 at 5:14 PM, Avinash Kujur avin...@gmail.com wrote:
hello everyone,
Is there any kind of specific format of the patch in which we need to
write
Hi,
I compiled Yarn event model using maven, but how to open .gv file to view
it?
Thanks.
Did you configure mapreduce.jobhistory.address and
mapreduce.jobhistory.webapp.address in the mapred-site.xml?
On Mon, Mar 31, 2014 at 3:10 PM, Rahul Singh smart.rahul.i...@gmail.comwrote:
Hi,
I have configured hadoop 2.3 single node setup and ran a sample job
using hadoop command,
it was defined at hadoop-config.sh
On Fri, Mar 28, 2014 at 1:19 PM, divye sheth divs.sh...@gmail.com wrote:
Which version of hadoop are u using? AFAIK the hadoop mapred home is the
directory where hadoop is installed or in other words untarred.
Thanks
Divye Sheth
On Mar 28, 2014 10:43
Hi Avin,
You should be added as an sub-project's contributor, then you can be an
assignee. so you can find how to be an contributor on the Wiki.
On Fri, Mar 28, 2014 at 6:50 PM, Avinash Kujur avin...@gmail.com wrote:
hi,
how can i be assignee fro a particular issue?
i can't see any option
start-dfs..
*Command for namenode format:*
hdfs namenode -format
Thanks Regards
Brahma Reddy Battula
--
*From:* Azuryy Yu [azury...@gmail.com]
*Sent:* Tuesday, March 25, 2014 2:39 PM
*To:* user@hadoop.apache.org
*Subject:* Re: namenode could
Hi,
which version HDFS you used?
On Wed, Mar 26, 2014 at 3:17 PM, Bharath Kumar bharath...@gmail.com wrote:
Hi All,
I am a novice hadoop user . I tried removing a node from my cluster of 2
nodes by adding the ip in excludes file and running dfsadmin -refreshNodes
command . But
Are you sure /home/hadoop/mydata/hdfs/namenode exists and has a right
permission?
On Tue, Mar 25, 2014 at 4:51 PM, haihong lu ung3...@gmail.com wrote:
Dear all:
I had a problem that the namenode could not start when i ran start-dfs.sh
. Show the message as below:
FATAL
Hi,
I do think Hadoop-0.2x is not similar to 2.0.
Hadoop version is composed of major-version.minor-version.released
such as: Hadoop-2.0.5, major version is 2, 0 is minor versioin. 5 the the
fifth release on 2.0
currently, Hadoop-1.2 and Hadoop-2.2 are all stable, but there are big
difference
Hi,
you'd bettern check out from svn, then build with maven: mvn
eclipse:eclipse, which will generate eclipse projects to you.
On Mon, Mar 24, 2014 at 3:24 PM, Avinash Kujur avin...@gmail.com wrote:
hi,
i downloaded the hadoop source code from github. after importing those
files in eclipse
Can you give the URL where you looked at?
On Mon, Mar 24, 2014 at 4:49 PM, 张建伟 connerlyzh...@gmail.com wrote:
hi,
In the apache site,I find a document said :
A client request to create a file does not reach the NameNode
immediately. In fact, initially the HDFS client caches the file data
the HDFS client caches the file data into a temporary
local file. Application writes are transparently redirected to this
temporary local file.
2014-03-24 17:01 GMT+08:00 Azuryy Yu azury...@gmail.com:
Can you give the URL where you looked at?
On Mon, Mar 24, 2014 at 4:49 PM, 张建伟
This is a known issue, but not a bug.
one safe way is remove files older than sevral hours under these dirs.
On Tue, Mar 25, 2014 at 8:57 AM, ch huang justlo...@gmail.com wrote:
hi,maillist:
i find many dirs in
It'll be supported in 2.4.
please look at here:
https://issues.apache.org/jira/browse/HDFS-5138
On Fri, Mar 21, 2014 at 3:46 PM, Meng QingPing mqingp...@gmail.com wrote:
Hi,
Hadoop dfs upgrade fail when HA enabled. Can Hadoop add feature to
upgrade dfs based on HA configure automatically
property
namehadoop.tmp.dir/name
value/home/hadoop/project/hadoop-data/value
/property
On Tue, Mar 18, 2014 at 2:06 PM, Azuryy Yu azury...@gmail.com wrote:
I don't think this is the case, because there is;
property
namehadoop.tmp.dir/name
value/home/hadoop/project
I don't think this is the case, because there is;
property
namehadoop.tmp.dir/name
value/home/hadoop/project/hadoop-data/value
/property
On Tue, Mar 18, 2014 at 1:55 PM, Stanley Shi s...@gopivotal.com wrote:
one possible reason is that you didn't set the namenode working directory,
should take while
doing this? Data loss is a NO NO for me.
Thanks
Divye Sheth
On Wed, Mar 5, 2014 at 1:28 PM, Azuryy Yu azury...@gmail.com wrote:
Hi,
That probably break something if you apply the patch from 2.x to 0.20.x,
but it depends on.
AFAIK, Balancer had a major refactor in HDFSv2
precautions that I should take?
Thanks Azuryy.
On Wed, Mar 5, 2014 at 2:19 PM, Azuryy Yu azury...@gmail.com wrote:
you can write a simple tool to move blocks peer to peer. I had such tool
before, but I cannot find it now.
background: our cluster is not balanced, load balancer is very slow, so i
Hi,
1) Is it possible to do an in-place migration, while keeping all
data in HDFS safely?
yes. stop the HDFS firstly, then run start-dfs.sh -upgrade
2) If it is yes, is there any doc/guidance to do this?
you just want a HDFS upgrade, so I don't think there are some useful doc.
3)
Hi,
That probably break something if you apply the patch from 2.x to 0.20.x,
but it depends on.
AFAIK, Balancer had a major refactor in HDFSv2, so you'd better fix it by
yourself based on HDFS-1804.
On Wed, Mar 5, 2014 at 3:47 PM, divye sheth divs.sh...@gmail.com wrote:
Thanks Harsh. The
Have you installed protobuf on your computer?
https://code.google.com/p/protobuf/downloads/list
On Tue, Mar 4, 2014 at 3:08 PM, nagarjuna kanamarlapudi
nagarjuna.kanamarlap...@gmail.com wrote:
Hi Ted,
I didn't do that earlier.
Now , I did it
mvn:eclipse eclipse
and tried importing
Generally, this is caused by insufficient space.
please check the total capacity of your cluster and used, remaining ratio,
and check dfs.datanode.du.reserved in the hdfs-site.xml
if this value is larger than your remained capacity, then you got this
Exception.
On Tue, Feb 25, 2014 at 10:35
Hi,
you cannot add jar like this way.
please look at DistributeCache in the Hadoop Java Doc.
please call DistributeCache.addArchive() in your main Class before submit
the MR job.
On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta gau...@datatorrent.comwrote:
Jonathan,
You have to make sure
.
On Fri, Jan 3, 2014 at 1:59 PM, Harsh J ha...@cloudera.com wrote:
Hi again,
I did not mention the TaskTracker log, but the actual Task JVM's
syslog/stderr/stdout. Do those log *any* problems at all?
On Fri, Jan 3, 2014 at 11:16 AM, Azuryy Yu azury...@gmail.com wrote:
Hi Harsh,
Thanks
actuall these are all logs in the stderr, and stdout is empty
On Fri, Jan 3, 2014 at 4:12 PM, Azuryy Yu azury...@gmail.com wrote:
Hi Harsh,
There is only three warnnings in stderr
*stderr logs*
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.mapred.Child
Hi,
Our prod cluster met some issues recently,
All map tasks finished successfully, but reduce task hanged.
but It's not happened on all TaskTrackers, only sometimes. we used
mapred-1.0.4
There is 0.0% reduce copy forever until kill task manually.
reduce logs on the TaskTracker:
Add addtional:
Our MR version is 1.2.1, not 1.0.4
There is no useful information in the JT log.
On Fri, Jan 3, 2014 at 12:20 PM, Azuryy Yu azury...@gmail.com wrote:
Hi,
Our prod cluster met some issues recently,
All map tasks finished successfully, but reduce task hanged.
but It's
TaskTrackers
in trying to obtain the data?
On Fri, Jan 3, 2014 at 9:54 AM, Azuryy Yu azury...@gmail.com wrote:
Add addtional:
Our MR version is 1.2.1, not 1.0.4
There is no useful information in the JT log.
On Fri, Jan 3, 2014 at 12:20 PM, Azuryy Yu azury...@gmail.com wrote
In detail:
'and these people's job never hanged...'
these people's map and reduce tasks never hanged.
On Fri, Jan 3, 2014 at 1:46 PM, Azuryy Yu azury...@gmail.com wrote:
Hi Harsh,
Thanks.
There is no any error logs for attempt_201312201200_34795_r_00_0 in
the tasktracker log. only
Hi,
you can use org.apache.hadoop.streaming.StreamInputFormat using map reduce
to convert XML to text.
such as your xml like this:
xml
namelll/name
/xml
you need to specify stream.recordreader.begin and stream.recordreader.end
in the Configuration:
Configuration conf = new Configuration();
You can run a mapreduce firstly, Join these data sets into one data set.
then analyze the joined dataset.
On Mon, Dec 30, 2013 at 3:58 PM, Fengyun RAO raofeng...@gmail.com wrote:
Hi,
HDFS splits files into blocks, and mapreduce runs a map task for each
block. However, Fields could be
Generally, MRv2 indicates Yarn. you can try:
yarn application
then there are full help lists.
On Tue, Dec 31, 2013 at 12:32 PM, 小网客 smallnetvisi...@foxmail.com wrote:
ui or hadoop job command like：hadoop job -list
--
-
Hi Kandoi,
It depends on:
how many cores on each VNode
how complicated of your analysis application
But I don't think it's normal spent 3hr to process 30GB data even on your
*not good* hareware.
On Tue, Dec 17, 2013 at 6:39 PM, Kandoi, Nikhil nikhil.kan...@emc.comwrote:
Hello everyone,
Hi Tao,
No, you need to set mapred.local.dir in the mapred-site.xml with comma
separated list of path to spread I/O .
On Mon, Dec 16, 2013 at 5:11 PM, Tao Xiao xiaotao.cs@gmail.com wrote:
Thanks.
In order to spread I/O among multiple disks, should I assign a
comma-separated list of
Hi Jiayu,
For the Sequence file as an input, CompressCodec class was serialized in
the file header, then Sequence Filereader will know the compression algo.
thanks.
On Mon, Dec 16, 2013 at 8:28 AM, Jiayu Ji jiayu...@gmail.com wrote:
Thanks Tao. I know I can tell it is a lzo file based on the
Jeff,
DFSClient don't use copied Configuration from RM.
did you add hostname or IP addr in the conf/slaves? if hostname, Can you
check /etc/hosts? does there have confilicts? and y
On Mon, Dec 16, 2013 at 5:01 AM, Jeff Stuckman stuck...@umd.edu wrote:
Thanks for the response. I have the
Why enable test flag under non-test environment?
On Mon, Dec 16, 2013 at 11:10 AM, Xiaobin She xiaobin...@gmail.com wrote:
To Vinod,
thank you for your reply, I found that I got this error message because I
did not add the following property in the hdfs-site.xml file
28 property
29
Can you check the datanode log to find what error happended?
On Sat, Dec 14, 2013 at 9:27 AM, Karim Awara karim.aw...@kaust.edu.sawrote:
Hi,
I was setting up hadoop 2.2 following
thesehttp://raseshmori.wordpress.com/2012/09/23/install-hadoop-2-0-1-yarn-nextgen/instructions
on a single
I cannot open you shared URL.
but if you write or create file by user A under a HDFS path , which is not
created by user A, then it's permission failed.
one solution is 'hdfs fs -chown -R A:A your_path' , another is 'hdfs fs
-chmod -R 777 your_path'
On Tue, Dec 3, 2013 at 5:09 PM, unmesha
Hi Jay,
Can you point out a detail about 1.x that job tracker having an
ever-expanding hash map. I want fix it.
On Tue, Dec 3, 2013 at 3:08 AM, Jay Vyas jayunit...@gmail.com wrote:
version is rewally important here..
- If 1.x, then Where (NN , JT , TT ?)
- if 2.x, then where? (AM, NM, ...
Hi Sandy,
I think there are some permission limited on your shared URL. maybe only
Cloudera employees allowed.
On Tue, Dec 3, 2013 at 4:26 PM, Sandy Ryza sandy.r...@cloudera.com wrote:
Hi Geelong,
Check out Todd Lipcon's presentation on tuning MapReduce performance:
, Dec 3, 2013 at 5:46 PM, Azuryy Yu azury...@gmail.com wrote:
Hi Sandy,
I think there are some permission limited on your shared URL. maybe only
Cloudera employees allowed.
On Tue, Dec 3, 2013 at 4:26 PM, Sandy Ryza sandy.r...@cloudera.comwrote:
Hi Geelong,
Check out Todd Lipcon's
One suggestion is change your hive partition, add a hive partition every
five minutes, and your HDFS file also roller every five minutes.
On Wed, Dec 4, 2013 at 11:56 AM, ch huang justlo...@gmail.com wrote:
hi,mailist:
when HDFS 's file is in appending ,no other reader can get
Ok, so you go for yourself.
On Wed, Dec 4, 2013 at 12:15 PM, ch huang justlo...@gmail.com wrote:
it seems not a good suggestion,get lot of partition dir and data file will
be a big compact to NN
On Wed, Dec 4, 2013 at 12:08 PM, Azuryy Yu azury...@gmail.com wrote:
One suggestion is change
This is still because your fence method configuraed improperly.
plseae paste your fence configuration. and double check you can ssh on
active NN to standby NN without password.
On Tue, Dec 3, 2013 at 10:23 AM, YouPeng Yang yypvsxf19870...@gmail.comwrote:
Hi
Another auto-failover testing
Your disk is full from the log.
On 2013-11-28 5:27 AM, Siddharth Tiwari siddharth.tiw...@live.com wrote:
Hi Team
I am getting following strange error, can you point me to the possible
reason.
I have set heap size to 4GB but still getting it. please help
*syslog logs*
2013-11-27
Siddharth,
please check 'mapred.local.dir', but I would like advice you check GC logs
and OS logs. pay more attention on OS logs. I suspect you start too many
threads concurrently, then consumed all OS avaliable resources.
On Thu, Nov 28, 2013 at 9:08 AM, Vinayakumar B
yes. you need to increase it, a simple way is put it in your /etc/profile
On Thu, Nov 28, 2013 at 9:59 AM, Siddharth Tiwari siddharth.tiw...@live.com
wrote:
Hi Vinay and Azuryy
Thanks for your responses.
I get these error when I just run a teragen.
Also, do you suggest me to increase
I don't think this is a normal way, and It's not suggested. we can deploy
cluster cross IDC, cross different network, but don't cross OS.
at least currently.
On Tue, Nov 26, 2013 at 6:56 AM, Andrew Machtolff amachto...@askcts.comwrote:
Yes, I set one up as a test. I had a windows cluster of
1 - 100 of 246 matches
Mail list logo
