Re: error in hadoop hdfs while building the code.

[Avinash Kujur]

+ import org.apache.hadoop.ipc.RefreshCallQueueProtocol;
+ import org.apache.hadoop.ipc.protocolPB.RefreshCallQueueProtocolPB;
+ import
org.apache.hadoop.ipc.protocolPB.RefreshCallQueueProtocolClientSideTranslatorPB;

+ private static RefreshCallQueueProtocol
  +createNNProxyWithRefreshCallQueueProtocol(InetSocketAddress address,
  Configuration conf, UserGroupInformation ugi) throws IOException {
  + RefreshCallQueueProtocolPB proxy = (RefreshCallQueueProtocolPB)
createNameNodeProxy(address, conf, ugi,
RefreshCallQueueProtocolPB.class, 0);
+ return new RefreshCallQueueProtocolClientSideTranslatorPB(proxy);
  }


All the lines with + signs are showing error with message
"RefreshCallQueueProtocol can nto be resolved" and the type of problem is
"java problem".


regards,
Avinash


On Tue, Mar 11, 2014 at 11:09 PM, unmesha sreeveni wrote:

> I think it is Hadoop problem not java
> https://issues.apache.org/jira/browse/HADOOP-5396
>
>
> On Wed, Mar 12, 2014 at 11:37 AM, Avinash Kujur  wrote:
>
>> hi,
>>  i am getting error like "RefreshCallQueueProtocol can not be resolved".
>> it is a java problem.
>>
>> help me out.
>>
>> Regards,
>> Avinash
>>
>
>
>
> --
> *Thanks & Regards*
>
> Unmesha Sreeveni U.B
> Junior Developer
>
> http://www.unmeshasreeveni.blogspot.in/
>
>
>

Re: GC overhead limit exceeded

[divye sheth]

Hi Haihong,

Please check out the link below, I believe it should solve your problem.

http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

Thanks
Divye Sheth


On Wed, Mar 12, 2014 at 11:33 AM, haihong lu  wrote:

> Thanks, even if i had added this parameter, but had no effect.
>
>
> On Tue, Mar 11, 2014 at 12:11 PM, unmesha sreeveni 
> wrote:
>
>> Try to increase the memory for datanode and see.This need to restart
>> hadoop
>> export HADOOP_DATANODE_OPTS="-Xmx10g"
>> This will set the heap to 10gb
>> You can also add this in start of hadoop-env.sh file
>>
>>
>> On Tue, Mar 11, 2014 at 9:02 AM, haihong lu  wrote:
>>
>>> i have tried both of the methods you side, but the problem still exists.
>>> Thanks all the same. by the way, my hadoop version is 2.2.0, so the
>>> parameter  "mapreduce.map.memory.mb =3072" added to mapred-site.xml
>>> maybe has no effect. I have looked for this parameter in the document of
>>> hadoop, but did not found it.
>>>
>>>
>>> On Fri, Mar 7, 2014 at 4:57 PM, shashwat shriparv <
>>> dwivedishash...@gmail.com> wrote:
>>>
 Check this out


 http://ask.gopivotal.com/hc/en-us/articles/201850408-Namenode-fails-with-java-lang-OutOfMemoryError-GC-overhead-limit-exceeded



 * Warm Regards_**∞_*
 * Shashwat Shriparv*
  [image: 
 http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9][image:
 https://twitter.com/shriparv] [image:
 https://www.facebook.com/shriparv] 
 [image:
 http://google.com/+ShashwatShriparv][image:
 http://www.youtube.com/user/sShriparv/videos][image:
 http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]



 On Fri, Mar 7, 2014 at 12:04 PM, haihong lu  wrote:

> Hi:
>
>  i have a problem when run Hibench with hadoop-2.2.0, the wrong
> message list as below
>
> 14/03/07 13:54:53 INFO mapreduce.Job:  map 19% reduce 0%
>  14/03/07 13:54:54 INFO mapreduce.Job:  map 21% reduce 0%
> 14/03/07 14:00:26 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_20_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:00:27 INFO mapreduce.Job:  map 20% reduce 0%
> 14/03/07 14:00:40 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_08_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:00:41 INFO mapreduce.Job:  map 19% reduce 0%
> 14/03/07 14:00:59 INFO mapreduce.Job:  map 20% reduce 0%
> 14/03/07 14:00:59 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_15_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:01:00 INFO mapreduce.Job:  map 19% reduce 0%
> 14/03/07 14:01:03 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_23_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:01:11 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_26_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:01:35 INFO mapreduce.Job:  map 20% reduce 0%
> 14/03/07 14:01:35 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_19_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:01:36 INFO mapreduce.Job:  map 19% reduce 0%
> 14/03/07 14:01:43 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_07_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:02:00 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_00_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:02:01 INFO mapreduce.Job:  map 18% reduce 0%
> 14/03/07 14:02:23 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_21_0, Status : FAILED
> Error: Java heap space
> 14/03/07 14:02:24 INFO mapreduce.Job:  map 17% reduce 0%
> 14/03/07 14:02:31 INFO mapreduce.Job:  map 18% reduce 0%
> 14/03/07 14:02:33 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_29_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:02:34 INFO mapreduce.Job:  map 17% reduce 0%
> 14/03/07 14:02:38 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_10_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:02:41 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_18_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:02:43 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_14_0, Status : FAILED
> Error: GC overhead limit exceeded
> 14/03/07 14:02:47 INFO mapreduce.Job: Task Id :
> attempt_1394160253524_0010_m_28_0, Status : FAILED
> Error: Java heap space
> 14/03/

Re: error in hadoop hdfs while building the code.

[unmesha sreeveni]

I think it is Hadoop problem not java
https://issues.apache.org/jira/browse/HADOOP-5396


On Wed, Mar 12, 2014 at 11:37 AM, Avinash Kujur  wrote:

> hi,
>  i am getting error like "RefreshCallQueueProtocol can not be resolved".
> it is a java problem.
>
> help me out.
>
> Regards,
> Avinash
>



-- 
*Thanks & Regards*

Unmesha Sreeveni U.B
Junior Developer

http://www.unmeshasreeveni.blogspot.in/

error in hadoop hdfs while building the code.

[Avinash Kujur]

hi,
 i am getting error like "RefreshCallQueueProtocol can not be resolved".
it is a java problem.

help me out.

Regards,
Avinash

Re: GC overhead limit exceeded

[haihong lu]

Thanks, even if i had added this parameter, but had no effect.


On Tue, Mar 11, 2014 at 12:11 PM, unmesha sreeveni wrote:

> Try to increase the memory for datanode and see.This need to restart hadoop
> export HADOOP_DATANODE_OPTS="-Xmx10g"
> This will set the heap to 10gb
> You can also add this in start of hadoop-env.sh file
>
>
> On Tue, Mar 11, 2014 at 9:02 AM, haihong lu  wrote:
>
>> i have tried both of the methods you side, but the problem still exists.
>> Thanks all the same. by the way, my hadoop version is 2.2.0, so the
>> parameter  "mapreduce.map.memory.mb =3072" added to mapred-site.xml
>> maybe has no effect. I have looked for this parameter in the document of
>> hadoop, but did not found it.
>>
>>
>> On Fri, Mar 7, 2014 at 4:57 PM, shashwat shriparv <
>> dwivedishash...@gmail.com> wrote:
>>
>>> Check this out
>>>
>>>
>>> http://ask.gopivotal.com/hc/en-us/articles/201850408-Namenode-fails-with-java-lang-OutOfMemoryError-GC-overhead-limit-exceeded
>>>
>>>
>>>
>>> * Warm Regards_**∞_*
>>> * Shashwat Shriparv*
>>>  [image: 
>>> http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9][image:
>>> https://twitter.com/shriparv] [image:
>>> https://www.facebook.com/shriparv] 
>>> [image:
>>> http://google.com/+ShashwatShriparv][image:
>>> http://www.youtube.com/user/sShriparv/videos][image:
>>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]
>>>
>>>
>>>
>>> On Fri, Mar 7, 2014 at 12:04 PM, haihong lu  wrote:
>>>
 Hi:

  i have a problem when run Hibench with hadoop-2.2.0, the wrong
 message list as below

 14/03/07 13:54:53 INFO mapreduce.Job:  map 19% reduce 0%
  14/03/07 13:54:54 INFO mapreduce.Job:  map 21% reduce 0%
 14/03/07 14:00:26 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_20_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:00:27 INFO mapreduce.Job:  map 20% reduce 0%
 14/03/07 14:00:40 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_08_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:00:41 INFO mapreduce.Job:  map 19% reduce 0%
 14/03/07 14:00:59 INFO mapreduce.Job:  map 20% reduce 0%
 14/03/07 14:00:59 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_15_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:01:00 INFO mapreduce.Job:  map 19% reduce 0%
 14/03/07 14:01:03 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_23_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:01:11 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_26_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:01:35 INFO mapreduce.Job:  map 20% reduce 0%
 14/03/07 14:01:35 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_19_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:01:36 INFO mapreduce.Job:  map 19% reduce 0%
 14/03/07 14:01:43 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_07_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:02:00 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_00_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:02:01 INFO mapreduce.Job:  map 18% reduce 0%
 14/03/07 14:02:23 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_21_0, Status : FAILED
 Error: Java heap space
 14/03/07 14:02:24 INFO mapreduce.Job:  map 17% reduce 0%
 14/03/07 14:02:31 INFO mapreduce.Job:  map 18% reduce 0%
 14/03/07 14:02:33 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_29_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:02:34 INFO mapreduce.Job:  map 17% reduce 0%
 14/03/07 14:02:38 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_10_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:02:41 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_18_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:02:43 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_14_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:02:47 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_28_0, Status : FAILED
 Error: Java heap space
 14/03/07 14:02:50 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_02_0, Status : FAILED
 Error: GC overhead limit exceeded
 14/03/07 14:02:51 INFO mapreduce.Job:  map 16% reduce 0%
 14/03/07 14:02:51 INFO mapreduce.Job: Task Id :
 attempt_1394160253524_0010_m_05_0, Status : FAILED
 Error: GC overhead limit exceeded
 14

Size of data directory same on all nodes in cluster

[Vimal Jain]

Hi,
I have setup 2 node Hbase cluster on top of 2 node HDFS cluster.
When i perform "du -sh" command on data directory ( where hadoop stores
data ) on both machines , its shows the same size.
As per my understanding , of entire data half of the data is stored in one
machine and other half on other machine.
Please help.

-- 
Thanks and Regards,
Vimal Jain

Re: In hadoop All racks belongs to same subnet !

[Stanley Shi]

There's no limitation on same-subnet.

Regards,
*Stanley Shi,*



On Wed, Mar 12, 2014 at 1:31 PM, navaz  wrote:

> Hi all
>
> Question regarding hadoop architecture . Generally in hadoop cluster nodes
> are placed in racks and all the nodes connected to top of the rack switch .
> And all these top of the rack switch connected by another switch or router
> ?
> Is all nodes in the cluster belong to same subnet ? Or  hdfs write & read
> has to pass through internet ?
>
> Regards
> Navaz
>
>
> --
> *Abdul Navaz*
> *Masters in Network Communications*
> *University of Houston*
> *Houston, TX - 77204-4020*
> *Ph - 281-685-0388 <281-685-0388>*
> *fabdulna...@uh.edu* 
>
>
>

In hadoop All racks belongs to same subnet !

[navaz]

Hi all

Question regarding hadoop architecture . Generally in hadoop cluster nodes
are placed in racks and all the nodes connected to top of the rack switch .
And all these top of the rack switch connected by another switch or router
?
Is all nodes in the cluster belong to same subnet ? Or  hdfs write & read
has to pass through internet ?

Regards
Navaz


-- 
*Abdul Navaz*
*Masters in Network Communications*
*University of Houston*
*Houston, TX - 77204-4020*
*Ph - 281-685-0388*
*fabdulna...@uh.edu* 

Re: Using resource manager as standalone component from hadoop yarn

[Jeff Zhang]

You can start the resource manager without starting any node manager.  And
the source code of resource manager and node manager are in different sub
pom project.


On Wed, Mar 12, 2014 at 7:06 AM, anu238 .  wrote:

> Hi All,
> I am sorry for the blast, I wanted to use only the resource manager from
> the hadoop yarn code base for building my platform for my use case, I was
> wondering if anyone here have used only the resource manager from hadoop
> yarn code base and how did they separated the resource manager component
> from the other code.
>
> Thanks & Regards,
> Anurag
>

Using resource manager as standalone component from hadoop yarn

[anu238 .]

Hi All,
I am sorry for the blast, I wanted to use only the resource manager from
the hadoop yarn code base for building my platform for my use case, I was
wondering if anyone here have used only the resource manager from hadoop
yarn code base and how did they separated the resource manager component
from the other code.

Thanks & Regards,
Anurag

Re: subsrcibe

[Alpha Bagus Sunggono]

You have already subscribed to this list.


2014-03-12 2:22 GMT+07:00 anu238 . :

> i want to subscribe to this list.
> Thanks
> A
>



-- 
Alpha Bagus Sunggono, CBSP
(Certified Brownies Solution Provider)

Re: Solving "heap size error"

[Mahmood Naderan]

As I posted earlier, here is the result of a successful test

5.4GB XML file (which is larger than enwiki-latest-pages-articles10.xml) with 
4GB of RAM and -Xmx128m tooks 5 minutes to complete.

I didn't find a larger wikipedia XML file. Need to test 10GB, 20GB and 30GB 
files


 
Regards,
Mahmood



On Tuesday, March 11, 2014 11:41 PM, Andrew Musselman 
 wrote:
 
Can you please try running this on a smaller file first, per Suneel's
comment a while back:

"Please first try running this on a smaller dataset like
'enwiki-latest-pages-articles10.xml' as opposed to running on the entire
english wikipedia."



On Tue, Mar 11, 2014 at 12:56 PM, Mahmood Naderan wrote:

> Hi,
> Recently I have faced a heap size error when I run
>
>   $MAHOUT_HOME/bin/mahout wikipediaXMLSplitter -d
>
 $MAHOUT_HOME/examples/temp/enwiki-latest-pages-articles.xml -o
> wikipedia/chunks -c 64
>
> Here is the specs
> 1- XML file size = 44GB
> 2- System memory = 54GB (on virtualbox)
> 3- Heap size = 51GB (-Xmx51000m)
>
> At the time of failure, I see that 571 chunks are created (hadoop dfs -ls)
> so 36GB of the original file has been processed. Now here are my questions
>
> 1- Is there any way to resume the process? As stated before, 571 chunks
> have been created. So by resuming, it can create the rest of the chunks
> (572~).
>
> 2- Is it possible to parallelize the process? Assume, 100GB of heap is
> required to process the XML file and my system cannot
 afford that. Then we
> can create 20 threads each requires 5GB of heap. Next by feeding the first
> 10 threads we can use the available 50GB of heap and after completion, we
> can feed the next set of threads.
>
>
> Regards,
> Mahmood

Solving "heap size error"

[Mahmood Naderan]

Hi,
Recently I have faced a heap size error when I run 

  $MAHOUT_HOME/bin/mahout wikipediaXMLSplitter -d 
$MAHOUT_HOME/examples/temp/enwiki-latest-pages-articles.xml -o 
wikipedia/chunks -c 64

Here is the specs
1- XML file size = 44GB
2- System memory = 54GB (on virtualbox)
3- Heap size = 51GB (-Xmx51000m)

At the time of failure, I see that 571 chunks are created (hadoop dfs -ls) so 
36GB of the original file has been processed. Now here are my questions

1- Is there any way to resume the process? As stated before, 571 chunks have 
been created. So by resuming, it can create the rest of the chunks (572~).

2- Is it possible to parallelize the process? Assume, 100GB of heap is required 
to process the XML file and my system cannot afford that. Then we can create 20 
threads each requires 5GB of heap. Next by feeding the first 10 threads we can 
use the available 50GB of heap and after completion, we can feed the next set 
of threads.

 
Regards,
Mahmood

Re: problem with HDFS caching in Hadoop 2.3

[Colin McCabe]

On Fri, Mar 7, 2014 at 7:37 AM, hwpstorage  wrote:
> Hello,
>
> It looks like the HDFS caching does not work well.
> The cached log file is around 200MB. The hadoop cluster has 3 nodes, each
> has 4GB memory.
>
> -bash-4.1$ hdfs cacheadmin -addPool wptest1
> Successfully added cache pool wptest1.
>
> -bash-4.1$ /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listPools
> Found 1 result.
> NAME OWNER  GROUP  MODELIMIT  MAXTTL
> wptest1  hdfs   hdfs   rwxr-xr-x   unlimited   never
>
> -bash-4.1$ hdfs cacheadmin -addDirective -path hadoop003.log -pool wptest1
> Added cache directive 1
>
> -bash-4.1$  time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real0m2.796s
> user0m4.263s
> sys 0m0.203s
>
> -bash-4.1$  time /hadoop/hadoop-2.3.0/bin/hadoop fs -tail hadoop003.log
> real0m3.050s
> user0m4.176s
> sys 0m0.192s
>
> It is weird that the cache status shows 0 byte cached:-bash-4.1$
> /hadoop/hadoop-2.3.0/bin/hdfs cacheadmin -listDirectives -stats -path
> hadoop003.log -pool wptest1
> Found 1 entry
> ID POOL  REPL EXPIRY  PATH   BYTES_NEEDED
> BYTES_CACHED  FILES_NEEDED  FILES_CACHED
>   1 wptest1  1 never   /user/hdfs/hadoop003.log  209715206
> 0 1 0

If you take a look at this output, you can see that nothing is actually cached.

One way to figure out why this is is to look at the logs of the
NameNode and DataNode.  Some of the relevant logs are at DEBUG or
TRACE level, so you may need to turn up the logs.  The
CacheReplicationMonitor and FsDatasetCache classes are good places to
start.

Also be sure to check that you have set dfs.datanode.max.locked.memory.

As Andrew commented, "hadoop tail" is not a good command to use for
measuring performance, since you have a few seconds of Java startup
time, followed by any HDFS setup time, followed by reading a single
kilobyte of data.  If you want to use the shell, the simplest thing to
do is to use cat and read a large file, so that those startup costs
don't dominate the measurement.

best,
Colin


>
> -bash-4.1$ file /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0
> /hadoop/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0: ELF 64-bit LSB shared
> object, x86-64, version 1 (SYSV), dynamically linked, not stripped
>
> I also tried the word count example with the same file. The execution time
> is always 40 seconds. (The map/reduce job without cache is 42 seconds)
> Is there anything wrong?
> Thanks a lot

subsrcibe

[anu238 .]

i want to subscribe to this list.
Thanks
A

task failure by machine

[Anfernee Xu]

Hi,

I'm running Yarn 2.2.0 release, I'm wondering how can I learn the task
failure rate by machine?
In older release, there's a web page can tell me that information, so I can
exclude the machine with higher failure rate from the cluster, is there any
equivalent feature in 2.2.0 release?

Thanks

-- 
--Anfernee

Re: Hadoop2.x reading data

[Harsh J]

This is a Pig problem, not a "Hadoop 2.x" one - can you please ask it
at u...@pig.apache.org? You may have to subscribe to it first.

On Tue, Mar 11, 2014 at 1:03 PM, Viswanathan J
 wrote:
> Hi,
>
> I'm currently trying to use Hadoop-2.x and noticed when I try to load the
> file in pig and it shows as follows while reading but file has multiple
> records and also weird thing is if I dump the variable its shows the pig
> tuples,
>
> Successfully read 0 records from: "/tmp/sample.txt"
>
> Any reason?
>
> --
> Regards,
> Viswa.J



-- 
Harsh J

Re: regarding hadoop source code

[Jian He]

Those are just test failures, I suggest you skip the tests as you earlier
did and do mvn clean install.


On Tue, Mar 11, 2014 at 5:10 AM, Avinash Kujur  wrote:

> after executing this command :
> mvn clean install
>
> i am getting this error.
>
> Failed tests:
>   TestMetricsSystemImpl.testMultiThreadedPublish:232 expected:<0> but
> was:<5>
>   TestNetUtils.testNormalizeHostName:617 null
>
> TestFsShellReturnCode.testGetWithInvalidSourcePathShouldNotDisplayNullInConsole:307
> Return code should be 1 expected:<1> but was:<-1>
>   TestFsShellReturnCode.testInvalidDefaultFS:379 Return code should be 0
>   TestActiveStandbyElector.testWithoutZKServer:719 Did not throw zookeeper
> connection loss exceptions!
>
> Tests in error:
>   TestZKFailoverController.testGracefulFailover:444->Object.wait:-2 »
> test time...
>   TestConfigurationDeprecation.testConcurrentDeprecateAndManipulate:400 »
> test ...
>
> Tests run: 2312, Failures: 5, Errors: 2, Skipped: 103
>
>
> [INFO]
> 
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Hadoop Main  SUCCESS [
> 9.745 s]
> [INFO] Apache Hadoop Project POM . SUCCESS [
> 2.895 s]
> [INFO] Apache Hadoop Annotations . SUCCESS [
> 13.955 s]
> [INFO] Apache Hadoop Project Dist POM  SUCCESS [
> 3.720 s]
> [INFO] Apache Hadoop Assemblies .. SUCCESS [
> 1.121 s]
> [INFO] Apache Hadoop Maven Plugins ... SUCCESS [
> 14.362 s]
> [INFO] Apache Hadoop MiniKDC . SUCCESS [01:05
> min]
> [INFO] Apache Hadoop Auth  SUCCESS [02:31
> min]
> [INFO] Apache Hadoop Auth Examples ... SUCCESS [
> 3.487 s]
> [INFO] Apache Hadoop Common .. FAILURE [27:23
> min]
> [INFO] Apache Hadoop NFS . SKIPPED
> [INFO] Apache Hadoop Common Project .. SKIPPED
> [INFO] Apache Hadoop HDFS  SKIPPED
> [INFO] Apache Hadoop HttpFS .. SKIPPED
>
> [INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
> [INFO] Apache Hadoop HDFS-NFS  SKIPPED
> [INFO] Apache Hadoop HDFS Project  SKIPPED
> [INFO] hadoop-yarn ... SKIPPED
> [INFO] hadoop-yarn-api ... SKIPPED
> [INFO] hadoop-yarn-common  SKIPPED
> [INFO] hadoop-yarn-server  SKIPPED
> [INFO] hadoop-yarn-server-common . SKIPPED
> [INFO] hadoop-yarn-server-nodemanager  SKIPPED
> [INFO] hadoop-yarn-server-web-proxy .. SKIPPED
> [INFO] hadoop-yarn-server-applicationhistoryservice .. SKIPPED
> [INFO] hadoop-yarn-server-resourcemanager  SKIPPED
> [INFO] hadoop-yarn-server-tests .. SKIPPED
> [INFO] hadoop-yarn-client  SKIPPED
> [INFO] hadoop-yarn-applications .. SKIPPED
> [INFO] hadoop-yarn-applications-distributedshell . SKIPPED
> [INFO] hadoop-yarn-applications-unmanaged-am-launcher  SKIPPED
> [INFO] hadoop-yarn-site .. SKIPPED
> [INFO] hadoop-yarn-project ... SKIPPED
> [INFO] hadoop-mapreduce-client ... SKIPPED
> [INFO] hadoop-mapreduce-client-core .. SKIPPED
> [INFO] hadoop-mapreduce-client-common  SKIPPED
> [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED
> [INFO] hadoop-mapreduce-client-app ... SKIPPED
> [INFO] hadoop-mapreduce-client-hs  SKIPPED
> [INFO] hadoop-mapreduce-client-jobclient . SKIPPED
> [INFO] hadoop-mapreduce-client-hs-plugins  SKIPPED
> [INFO] Apache Hadoop MapReduce Examples .. SKIPPED
> [INFO] hadoop-mapreduce .. SKIPPED
> [INFO] Apache Hadoop MapReduce Streaming . SKIPPED
> [INFO] Apache Hadoop Distributed Copy  SKIPPED
> [INFO] Apache Hadoop Archives  SKIPPED
> [INFO] Apache Hadoop Rumen ... SKIPPED
> [INFO] Apache Hadoop Gridmix . SKIPPED
> [INFO] Apache Hadoop Data Join ... SKIPPED
> [INFO] Apache Hadoop Extras .. SKIPPED
> [INFO] Apache Hadoop Pipes ... SKIPPED
> [INFO] Apache Hadoop OpenStack support ... SKIPPED
> [INFO] Apache Hadoop Client .. SKIPPED
> [INFO] Apache Hadoop Mini-Cluster  SKIPPED
> [INFO] Apache Hadoop Scheduler Load Simulator

Hadoop2.x reading data

[Viswanathan J]

Hi,

I'm currently trying to use Hadoop-2.x and noticed when I try to load the
file in pig and it shows as follows while reading but file has multiple
records and also weird thing is if I dump the variable its shows the pig
tuples,

Successfully read 0 records from: "/tmp/sample.txt"

Any reason?

-- 
Regards,
Viswa.J

Re: regarding hadoop source code

[Avinash Kujur]

after executing this command :
mvn clean install

i am getting this error.

Failed tests:
  TestMetricsSystemImpl.testMultiThreadedPublish:232 expected:<0> but
was:<5>
  TestNetUtils.testNormalizeHostName:617 null

TestFsShellReturnCode.testGetWithInvalidSourcePathShouldNotDisplayNullInConsole:307
Return code should be 1 expected:<1> but was:<-1>
  TestFsShellReturnCode.testInvalidDefaultFS:379 Return code should be 0
  TestActiveStandbyElector.testWithoutZKServer:719 Did not throw zookeeper
connection loss exceptions!

Tests in error:
  TestZKFailoverController.testGracefulFailover:444->Object.wait:-2 »  test
time...
  TestConfigurationDeprecation.testConcurrentDeprecateAndManipulate:400 »
test ...

Tests run: 2312, Failures: 5, Errors: 2, Skipped: 103

[INFO]

[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main  SUCCESS [  9.745
s]
[INFO] Apache Hadoop Project POM . SUCCESS [  2.895
s]
[INFO] Apache Hadoop Annotations . SUCCESS [ 13.955
s]
[INFO] Apache Hadoop Project Dist POM  SUCCESS [  3.720
s]
[INFO] Apache Hadoop Assemblies .. SUCCESS [  1.121
s]
[INFO] Apache Hadoop Maven Plugins ... SUCCESS [ 14.362
s]
[INFO] Apache Hadoop MiniKDC . SUCCESS [01:05
min]
[INFO] Apache Hadoop Auth  SUCCESS [02:31
min]
[INFO] Apache Hadoop Auth Examples ... SUCCESS [  3.487
s]
[INFO] Apache Hadoop Common .. FAILURE [27:23
min]
[INFO] Apache Hadoop NFS . SKIPPED
[INFO] Apache Hadoop Common Project .. SKIPPED
[INFO] Apache Hadoop HDFS  SKIPPED
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS-NFS  SKIPPED
[INFO] Apache Hadoop HDFS Project  SKIPPED
[INFO] hadoop-yarn ... SKIPPED
[INFO] hadoop-yarn-api ... SKIPPED
[INFO] hadoop-yarn-common  SKIPPED
[INFO] hadoop-yarn-server  SKIPPED
[INFO] hadoop-yarn-server-common . SKIPPED
[INFO] hadoop-yarn-server-nodemanager  SKIPPED
[INFO] hadoop-yarn-server-web-proxy .. SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice .. SKIPPED
[INFO] hadoop-yarn-server-resourcemanager  SKIPPED
[INFO] hadoop-yarn-server-tests .. SKIPPED
[INFO] hadoop-yarn-client  SKIPPED
[INFO] hadoop-yarn-applications .. SKIPPED
[INFO] hadoop-yarn-applications-distributedshell . SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher  SKIPPED
[INFO] hadoop-yarn-site .. SKIPPED
[INFO] hadoop-yarn-project ... SKIPPED
[INFO] hadoop-mapreduce-client ... SKIPPED
[INFO] hadoop-mapreduce-client-core .. SKIPPED
[INFO] hadoop-mapreduce-client-common  SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ... SKIPPED
[INFO] hadoop-mapreduce-client-app ... SKIPPED
[INFO] hadoop-mapreduce-client-hs  SKIPPED
[INFO] hadoop-mapreduce-client-jobclient . SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins  SKIPPED
[INFO] Apache Hadoop MapReduce Examples .. SKIPPED
[INFO] hadoop-mapreduce .. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming . SKIPPED
[INFO] Apache Hadoop Distributed Copy  SKIPPED
[INFO] Apache Hadoop Archives  SKIPPED
[INFO] Apache Hadoop Rumen ... SKIPPED
[INFO] Apache Hadoop Gridmix . SKIPPED
[INFO] Apache Hadoop Data Join ... SKIPPED
[INFO] Apache Hadoop Extras .. SKIPPED
[INFO] Apache Hadoop Pipes ... SKIPPED
[INFO] Apache Hadoop OpenStack support ... SKIPPED
[INFO] Apache Hadoop Client .. SKIPPED
[INFO] Apache Hadoop Mini-Cluster  SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator  SKIPPED
[INFO] Apache Hadoop Tools Dist .. SKIPPED
[INFO] Apache Hadoop Tools ... SKIPPED
[INFO] Apache Hadoop Distribution  SKIPPED
[INFO]

[INFO] BUILD FAILURE
[INFO]
-

Get execution time hadoop pipes C++

[Andrea Barbato]

I need to make a performance comparison between a sequential code and a
hadoop code written in C++ with the pipes. What is the easiest solution?

Re: Hadoop 2.2.0 not showing progress

[Silvina Caíno Lores]

The job was actually running and it finished successfully. I was able to
monitor it by using wget and the resource manager web  interface
(http://host:port/ws/v1/cluster/apps) and the yarn application command that
Xuan suggested.

Thank you all for your help.


On 11 March 2014 06:11, sudhakara st  wrote:

> RM not able schdule your jobs it waiting indefinitely to schedule jobs,
> may due to  not able communicate with NM's or not able create  AM or
> sufiicient resoucrce are not avaialbel in containers etc. Check your
> configuration for Java heap, yarn.app.mapreduce.am.resource.mb, 
> mapreduce.reduce.memory.mb,
> mapreduce.map.memory.mb, Check RM web URL for available resources and
> scheduler information and also check RM,NM logs
>
>
> On Mon, Mar 10, 2014 at 9:54 PM, Xuan Gong  wrote:
>
>> Hey, Silvina:
>>
>>You may find more information about this application from RM web UI or
>> Yarn Command Line (Type yarn application -help to find out commands).
>>
>> Thanks
>>
>> Xuan Gong
>>
>>
>> On Mon, Mar 10, 2014 at 4:16 AM, Silvina Caíno Lores <
>> silvi.ca...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I've been noticing lately that sometimes my Hadoop jobs do not report
>>> progress in the terminal. They seem like they are stuck at the "Running
>>> job: job_ " message, however YarnChilds are running and properly
>>> executing.
>>>
>>> I know that my job didn't fail but it's very inconvenient not being able
>>> to monitor how is it going, specially to estimate completion time. This
>>> doesn't happen all the time, any clues on what is going on?
>>>
>>> Thanks in advance,
>>> Silvina
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
>
> --
>
> Regards,
> ...sudhakara
>
>