Move blocks between Nodes

will be appreciated. -- Bing Jiang

Re: Hadoop Terasort Benchmark Failure - Need Inputs

hi, Ashish I have ever seen a similar issue, and reported the issue https://issues.apache.org/jira/browse/MAPREDUCE-5782 I have some workaround from that jira. -Bing 2014-11-30 4:07 GMT+08:00 Ashish Kumar9 : > Hi , > > I am facing issue when i run teragen / terasort benchmark . Can someone >

Re: User Logs & Task StateMachine logs

2014-12-01 1:25 GMT+08:00 Srinivas Chamarthi : > Hi, > > I have built hadoop from trunk and executing a MR program. But for some > reason, I am not seeing any user logs on my windows box. can someone tell > me what should I do to enable logging of the jobs ? > > I am actually working on tweaking t

Re: Hadoop / HBase hotspotting / overloading specific nodes

> I have tried running hadoop balancer during the job run and this helped > but only really succeeded in prolonging the eventual job failure. > > How can I get Hadoop / HBase to distribute the data to HDFS more evenly > when it is favoring the nodes that the regions are on? > > Am I missing something here? > > Thanks for any help. > -- Bing Jiang

Re: How can I get the number files on HDFS have been visited and who have visited them?

Could you turn on hdfs audit log? It tracks the access to hdfs system. 2014-09-29 10:22 GMT+08:00 Yu Yang : > Hey Guys, > > I want to count the number files on HDFS have been visited and who have > visited them. > Is there an API which I can use to do this ? > > thank you

Re: Hadoop shuffling traffic

ted ? Does it > generate any syslog in the logs . > -> How to find the total amount of shuffling traffic? > > > > Thanks & Regards, > > Abdul Navaz > Research Assistant > University of Houston Main Campus, Houston TX > Ph: 281-685-0388 > >

Re: Tez and MapReduce

By the way, mapreduce.framework.name can be set yarn or yarn-tez. It will make differences. 2014-09-02 8:24 GMT+08:00 jay vyas : > Yes as an example of running a mapreduce job followed by a tez you can see > our last post on this > https://blogs.apache.org/bigtop/entry/testing_apache_tez_with_ap

Re: should i need tune mapred.child.java.opts option if i use yarn and MRV2?

I remember there is a maximum container memory config, It is ResourceManage-side config. 2014-05-23 13:58 GMT+08:00 ch huang : > hi,mailist: > i want to know if this option still cause limitation in YARN? > -- Bing Jiang weibo: http://weibo.com/jiangbingl

Re: Hadoop2 LZO issue with pig

putFormat.java:88)2592 > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initNextRecordReader(PigRecordReader.java:256) > > Have copied the native lib folder accordingly, but facing issue. Is there > any changes to do in hadoop conf or lib folder? >

Re: What makes a map to fail?

s failure. >> >> On Sun, Jan 5, 2014 at 11:03 PM, Saeed Adel Mehraban >> wrote: >> > Hi all, >> > My task jobs are failing due to many failed maps. I want to know what >> makes >> > a map to fail? Is it something like exceptions or what? >> >

Re: Haoop: Container is running beyond physical memory limits

s/userlogs/application_1388722003823_0001/container_1388722003823_0001_01_62/stdout > 2>/home/data/hadoop/logs/userlogs/application_1388722003823_0001/container_1388722003823_0001_01_62/stderr > > I tried some configurations, but still failed, who can do me a favor, thxs! >

Re: issue about running hive MR job in hadoop

tainerExecutor.startLocalizer(DefaultContainerExecutor.java:103) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:861) > .Failing this attempt.. Failing the application. > -- Bing Jiang Tel：(86)134-2619-1361 weibo: http://weibo.com/jiangbinglover BLOG: www.binospac

Re: Is there any design document for YARN

socc 2013 has published a paper about Yarn. On Nov 26, 2013 1:34 PM, "Jeff Zhang" wrote: > > Hi , > > I am reading the yarn code, so wondering whether there's any design > document for the yarn. I found the blog post on hortonworks is very useful. > But more details document would be helpful. Tha

Re: best solution for data ingestion

ari* >>> Have a refreshing day !!! >>> *"Every duty is holy, and devotion to duty is the highest form of >>> worship of God.” * >>> *"Maybe other people will try to limit me but I don't limit myself"* >>> >> >> > -- Bing Jiang Tel：(86)134-2619-1361 weibo: http://weibo.com/jiangbinglover BLOG: www.binospace.com BLOG: http://blog.sina.com.cn/jiangbinglover Focus on distributed computing, HDFS/HBase

Re: trace class calls in hadoop hdfs

gt; distribution is prohibited. Please consider the environment before printing > this email. -- Bing Jiang Tel：(86)134-2619-1361 weibo: http://weibo.com/jiangbinglover BLOG: www.binospace.com BLOG: http://blog.sina.com.cn/jiangbinglover Focus on distributed computing, HDFS/HBase

Re: produce a large sequencefile (1TB)

can randomly access each row in the sequence file. In > practice, this is simply a MapFile. > > Any idea how to resolve this dilemma is greatly appreciated. > > Jerry > > > > On Mon, Aug 19, 2013 at 8:14 PM, Bing Jiang wrote: > >> hi,Jerry. >> I think you a

Re: produce a large sequencefile (1TB)

hi,Jerry. I think you are worrying about the volumn of mapreduce local file, but would you give us more details about your apps. On Aug 20, 2013 6:09 AM, "Jerry Lam" wrote: > Hi Hadoop users and developers, > > I have a use case that I need produce a large sequence file of 1 TB in > size when e

Re: Removing Dead Nodes from Cluster

> > > Thanks, > Manickam P > -- Bing Jiang Tel：(86)134-2619-1361 weibo: http://weibo.com/jiangbinglover BLOG: www.binospace.com BLOG: http://blog.sina.com.cn/jiangbinglover Focus on distributed computing, HDFS/HBase

Too Many CLOSE_WAIT make performance down

not close fd which has been opened, and I restart RS gracefully, the problem has been tackled. Ok, My question is : Can someone tell me in which conditions do RS will ignore the file handler? Any ideas will be nice. Thanks! -- Bing Jiang Tel：(86)134-2619-1361 weibo: http://weibo.c

Re: two name node both in standby state?why?

e$2.callBlockingMethod(NamenodeProtocolProtos.java:8762) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002) > at org.apache.hadoop.ipc.Server$Handler$1

【data migrate from hdfs0.20.* to hdfs-2.0.5(HA)】

. org.apache.hadoop.ipc.RemoteException: Server IPC version 8 cannot > communicate with client version 4 > Could anyone give me an efficient way to make it? Thanks. -- Bing Jiang weibo: http://weibo.com/jiangbinglover BLOG: www.binospace.com BLOG: http://blog.sina.com.cn/jiangbinglover Focus on distributed computing

Re: about format name node in HA envionment

format one namenode，and launch another namenode using： bin/hdfs namenode -bootstrapStandby 2013/7/26 ch huang > hi ,all: > i use Quorum-based Storage for name node HA,my question is i need format > both 2 name node ? or just format one name node ? > -- Bing Jiang Tel：(86)1

Re: Datanode support different Namespace

don't use the same clusterID. all data nodes and >> namenodes should use the same clusterID. >> >> >> On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang wrote: >> >>> Hi, all >>> >>> We try to use hadoop-2.0.5-alpha, using two namespaces, one is f

Re: 【dfs.namenode.shared.edits.dir can support different NameServices】

; qjournal://n1.com:8485;n2.com:8485/cluster1 > > > > dfs.namenode.shared.edits.dir > qjournal://n1.com:8485;n2.com:8485/cluster2 > > > > On Thu, Jul 4, 2013 at 2:46 PM, Bing Jiang wrote: > >> hi, Chris. >> I have traced the source code,

Datanode support different Namespace

(BPServiceActor.java:225) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664) at java.lang.Thread.run(Thread.java:662) It is proved that one datanode has been required to attached to only one namespace? Any views about it will be thankful. Regards~ -- Bing Jiang Tel：(86)134-2619-1361 weibo: http:/

Re: 【dfs.namenode.shared.edits.dir can support different NameServices】

ject/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java#L147 > > Hope this helps, > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > > On Wed, Jul 3, 2013 at 6:21 AM, Bing Jiang wrote: > >> hi,all. >> Usin

【dfs.namenode.shared.edits.dir can support different NameServices】

~ -- Bing Jiang Tel：(86)134-2619-1361 weibo: http://weibo.com/jiangbinglover BLOG: http://blog.sina.com.cn/jiangbinglover National Research Center for Intelligent Computing Systems Institute of Computing technology Graduate University of Chinese Academy of Science

Re: Hadoop web interface

; Bharati > > Fortigate Filtered > > -- Bing Jiang Tel：(86)134-2619-1361 weibo: http://weibo.com/jiangbinglover BLOG: http://blog.sina.com.cn/jiangbinglover National Research Center for Intelligent Computing Systems Institute of Computing technology Graduate University of Chinese Academy of Science