Re: New to the list
Have you read http://wiki.apache.org/hadoop/HowToContribute ? There're several components in hadoop: common, hdfs, mapreduce, yarn ... Which one are you interested in ? Cheers On Wed, Dec 3, 2014 at 7:19 AM, Abhishek Singh <23singhabhis...@gmail.com> wrote: > Hello members, > > I am new to Hadoop. Please suggest some projects which I can carry on my > own as part of POC learning. I tried Horton sandbox but I wish run on my > system's setup(Not any VM). Also, I'm aware of ecosystem and basics. > Any sort of help would be appreciated. > > Please excuse me if this isn't the right place to ask. > > > Thanks in Advance, > > Abhishek Singh > > On Wed, Dec 3, 2014 at 11:52 AM, adarsh deshratnam < > adarsh.deshrat...@gmail.com> wrote: > >> Hi Eric, >> Welcome to the hadoop user list . Its an active list. >> >> >> Thanks, >> Adarsh D >> >> On Wed, Dec 3, 2014 at 11:46 AM, Erich Kolb wrote: >> >>> Hello Everyone! >>> >>> >>> >>> Is this list actually active or am I not actually subscribed? I signed >>> up earlier today and haven’t received anything. >>> >>> >>> >>> - Erich >>> >> >> >
Re: New to the list
Hello members, I am new to Hadoop. Please suggest some projects which I can carry on my own as part of POC learning. I tried Horton sandbox but I wish run on my system's setup(Not any VM). Also, I'm aware of ecosystem and basics. Any sort of help would be appreciated. Please excuse me if this isn't the right place to ask. Thanks in Advance, Abhishek Singh On Wed, Dec 3, 2014 at 11:52 AM, adarsh deshratnam < adarsh.deshrat...@gmail.com> wrote: > Hi Eric, > Welcome to the hadoop user list . Its an active list. > > > Thanks, > Adarsh D > > On Wed, Dec 3, 2014 at 11:46 AM, Erich Kolb wrote: > >> Hello Everyone! >> >> >> >> Is this list actually active or am I not actually subscribed? I signed >> up earlier today and haven’t received anything. >> >> >> >> - Erich >> > >
Re: unsubscribe me
use email: user-unsubscr...@hadoop.apache.org to unsubscribe. Thanks, Adarsh d On Wed, Dec 3, 2014 at 4:43 PM, chandu banavaram wrote: > please unsubscribe me >
Re: unsubscribe me
Email to user-unsubscr...@hadoop.apache.org to unsubscribe. Regards, Ramkumar Bashyam On Wed, Dec 3, 2014 at 4:43 PM, chandu banavaram wrote: > please unsubscribe me >
unsubscribe me
please unsubscribe me
Re: Question about the QJM HA namenode
hadoop-2.3.0-cdh5.1.0
hi, i move QJM from the l-hbase1.dba.dev.cn0 to another machine, and the down
time reduced to
5 mins, and the log on the l-hbase2.dba.dev.cn0 like below:
{log}
2014-12-03 15:55:51,306 INFO
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer: Loaded 197 edits
starting from txid 6599
2014-12-03 15:55:51,306 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager: Marking all
datandoes as stale
2014-12-03 15:55:51,307 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Reprocessing replication
and invalidation queues
2014-12-03 15:55:51,307 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing replication
queues
2014-12-03 15:55:51,307 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Will take over writing
edit logs at txnid 6797
2014-12-03 15:55:51,313 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog:
Starting log segment at 6797
2014-12-03 15:55:51,373 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog:
Number of transactions: 1 Total time for transactions(ms): 0 Number of
transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 9
2014-12-03 15:55:51,385 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Starting
CacheReplicationMonitor with interval 3 milliseconds
2014-12-03 15:55:51,385 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning because of pending operations
2014-12-03 15:55:51,678 INFO org.apache.hadoop.fs.TrashPolicyDefault: Namenode
trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0
minutes.
2014-12-03 15:55:51,679 INFO org.apache.hadoop.fs.TrashPolicyDefault: The
configured checkpoint interval is 0 minutes. Using an interval of 1440 minutes
that is used for deletion instead
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total number of
blocks= 179
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of invalid
blocks = 0
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
under-replicated blocks = 0
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
over-replicated blocks = 0
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of blocks
being written= 4
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Replication Queue initialization scan for invalid, over- and under-replicated
blocks completed in 386 msec
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 308 millisecond(s).
2014-12-03 15:56:21,385 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:56:21,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:56:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 30001 milliseconds
2014-12-03 15:56:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:57:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:57:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 1 millisecond(s).
2014-12-03 15:57:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:57:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:58:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:58:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 1 millisecond(s).
2014-12-03 15:58:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:58:51,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:59:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 30001 milliseconds
2014-12-03 15:59:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:59:51,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMoni
Re: Question about the QJM HA namenode
What is your Hadoop version?
On Wed, Dec 3, 2014 at 12:55 PM, mail list wrote:
> hi all,
>
> Attach log again!
>
> The failover happened at about time: 2014-12-03 12:01:
>
>
>
>
>
> On Dec 3, 2014, at 14:55, mail list wrote:
>
>> Sorry forget the log, the failover time at about 2014-12-03 12:01:
>>
>>
>> On Dec 3, 2014, at 14:48, mail list wrote:
>>
>>> Hi all,
>>>
>>> I deploy the hadoop with 3 machines:
>>>
>>> l-hbase1.dba.dev.cn0 (namenode active and QJM)
>>> l-hbase2.dba.dev.cn0 (namenode standby and datanode and QJM)
>>> l-hbase3.dba.dev.cn0 (datanode and QJM)
>>>
>>> Above the hadoop, i deploy a hbase:
>>> l-hbase1.dba.dev.cn0 (HMaster active)
>>> l-hbase2.dba.dev.cn0 (HMaster standby)
>>> l-hbase3.dba.dev.cn0 (RegionServer)
>>>
>>>
>>> I write a program which put data into hbase one row every seconds in a loop.
>>> Then I use iptables to simulate l-hbase1.dba.dev.cn0 offline，and after
>>> that , the program hang and can not
>>> write to hbase. After about 15 mins, the program can write again.
>>>
>>> The time 15mins for the HA failover is too long for me!
>>> And I’ve no idea about the reason.
>>>
>>> Then I check the l-hbase2.dba.dev.cn0 namenode logs, and find many retry
>>> like below:
>>> {code}
>>> 2014-12-03 12:13:35,165 INFO org.apache.hadoop.ipc.Client: Retrying connect
>>> to server: l-hbase1.dba.dev.cn0/10.86.36.217:8485. Already tried 1 time(s);
>>> retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
>>> sleepTime=1000 MILLISECONDS)
>>> {code}
>>>
>>> I have the QJM on l-hbase1.dba.dev.cn0, does it matter?
>>>
>>> I am a newbie, Any idea will be appreciated!!
>>
>
>
--
Harsh J
QJM cost 5 minutes for failover
Hi,all
we are testing QJM NAMENODE HA, and when the active name node down,it cost
about 5 mins to work normal.
Log on the standby name node as below:
{log}
014-12-03 15:55:51,307 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Will take over writing
edit logs at txnid 6797
2014-12-03 15:55:51,313 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog:
Starting log segment at 6797
2014-12-03 15:55:51,373 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog:
Number of transactions: 1 Total time for transactions(ms): 0 Number of
transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0 9
2014-12-03 15:55:51,385 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Starting
CacheReplicationMonitor with interval 3 milliseconds
2014-12-03 15:55:51,385 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning because of pending operations
2014-12-03 15:55:51,678 INFO org.apache.hadoop.fs.TrashPolicyDefault: Namenode
trash configuration: Deletion interval = 1440 minutes, Emptier interval = 0
minutes.
2014-12-03 15:55:51,679 INFO org.apache.hadoop.fs.TrashPolicyDefault: The
configured checkpoint interval is 0 minutes. Using an interval of 1440 minutes
that is used for deletion instead
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total number of
blocks= 179
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of invalid
blocks = 0
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
under-replicated blocks = 0
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of
over-replicated blocks = 0
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of blocks
being written= 4
2014-12-03 15:55:51,693 INFO org.apache.hadoop.hdfs.StateChange: STATE*
Replication Queue initialization scan for invalid, over- and under-replicated
blocks completed in 386 msec
2014-12-03 15:55:51,693 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 308 millisecond(s).
2014-12-03 15:56:21,385 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:56:21,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:56:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 30001 milliseconds
2014-12-03 15:56:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:57:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:57:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 1 millisecond(s).
2014-12-03 15:57:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:57:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:58:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:58:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 1 millisecond(s).
2014-12-03 15:58:51,386 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:58:51,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:59:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 30001 milliseconds
2014-12-03 15:59:21,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 15:59:51,387 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 3 milliseconds
2014-12-03 15:59:51,388 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 0 millisecond(s).
2014-12-03 16:00:14,295 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock: caught retry for allocation of a new block in
/hbase/testnn/WALs/l-hbase3.dba.dev.cn0.qunar.com,60020,1417585992012/l-hbase3.dba.dev.cn0.qunar.com%2C60020%2C1417585992012.1417593301483.
Returning previously allocated block
blk_107374345
