date:20140928

Hadoop shutdown scripts failing

2014-09-28 Thread Susheel Kumar Gadalay

How to redirect the storing of the following files from /tmp to some
other location.
hadoop--namenode.pid
hadoop--datanode.pid
yarn--resourcemanager.pid
yarn--nodemanager.pid

In /tmp, these files are cleared by OS sometime back and I am unable
to shutdown by standard scripts stop-dfs.sh or stop-yarn.sh

In core-site.xml I have given property hadoop.tmp.dir but still these
pid files are present in /tmp only,

Re: No space when running a hadoop job

2014-09-28 Thread Susheel Kumar Gadalay

You mean if multiple directory locations are given, Hadoop will
balance the distribution of files across these different directories.

But normally we start with 1 directory location and once it is
reaching the maximum, we add new directory.

In this case how can we balance the distribution of files?

One way is to list the files and move.

Will start balance script will work?

On 9/27/14, Alexander Pivovarov  wrote:
> It can read/write in parallel to all drives. More hdd more io speed.
>  On Sep 27, 2014 7:28 AM, "Susheel Kumar Gadalay" 
> wrote:
>
>> Correct me if I am wrong.
>>
>> Adding multiple directories will not balance the files distributions
>> across these locations.
>>
>> Hadoop will add exhaust the first directory and then start using the
>> next, next ..
>>
>> How can I tell Hadoop to evenly balance across these directories.
>>
>> On 9/26/14, Matt Narrell  wrote:
>> > You can add a comma separated list of paths to the
>> “dfs.datanode.data.dir”
>> > property in your hdfs-site.xml
>> >
>> > mn
>> >
>> > On Sep 26, 2014, at 8:37 AM, Abdul Navaz  wrote:
>> >
>> >> Hi
>> >>
>> >> I am facing some space issue when I saving file into HDFS and/or
>> >> running
>> >> map reduce job.
>> >>
>> >> root@nn:~# df -h
>> >> Filesystem   Size  Used Avail Use%
>> >> Mounted on
>> >> /dev/xvda2   5.9G  5.9G 0 100%
>> >> /
>> >> udev  98M  4.0K   98M   1%
>> >> /dev
>> >> tmpfs 48M  192K   48M   1%
>> >> /run
>> >> none 5.0M 0  5.0M   0%
>> >> /run/lock
>> >> none 120M 0  120M   0%
>> >> /run/shm
>> >> overflow 1.0M  4.0K 1020K   1%
>> >> /tmp
>> >> /dev/xvda4   7.9G  147M  7.4G   2%
>> >> /mnt
>> >> 172.17.253.254:/q/groups/ch-geni-net/Hadoop-NET  198G  108G   75G  59%
>> >> /groups/ch-geni-net/Hadoop-NET
>> >> 172.17.253.254:/q/proj/ch-geni-net   198G  108G   75G  59%
>> >> /proj/ch-geni-net
>> >> root@nn:~#
>> >>
>> >>
>> >> I can see there is no space left on /dev/xvda2.
>> >>
>> >> How can I make hadoop to see newly mounted /dev/xvda4 ? Or do I need
>> >> to
>> >> move the file manually from /dev/xvda2 to xvda4 ?
>> >>
>> >>
>> >>
>> >> Thanks & Regards,
>> >>
>> >> Abdul Navaz
>> >> Research Assistant
>> >> University of Houston Main Campus, Houston TX
>> >> Ph: 281-685-0388
>> >>
>> >
>> >
>>
>

Re: Hadoop UI - Unable to connect to the application master from the Hadoop UI.

2014-09-28 Thread Susheel Kumar Gadalay

I also faced some issue like this.

It shows the URL in :

Copy paste the link in browser and expand the host name.

I set up the host names in windows user etc/hosts file but still it
could not resolve.

On 9/29/14, S.L  wrote:
> Hi All,
>
> I am running a 3 node Apache Hadoop YARN 2.3.0  cluster , after a job is
> submitted , when I ascess the application master from the UI I get the
> following exception  and I am unable to connect to the Application Master
> from the UI, can someone let me know , what I need  to look at?
>
> HTTP ERROR 500
>
> Problem accessing /proxy/application_1411841629814_0032/. Reason:
>
> Connection refused
>
> Caused by:
>
> java.net.ConnectException: Connection refused
>   at java.net.PlainSocketImpl.socketConnect(Native Method)
>   at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>   at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>   at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>   at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>   at java.net.Socket.connect(Socket.java:579)
>   at java.net.Socket.connect(Socket.java:528)
>   at java.net.Socket.(Socket.java:425)
>   at java.net.Socket.(Socket.java:280)
>   at
> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
>   at
> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
>   at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
>   at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
>   at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
>   at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
>   at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
>   at
> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185)
>   at
> org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:336)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>   at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>   at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>   at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
>   at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
>   at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
>   at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
>   at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
>   at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
>   at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
>   at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
>   at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
>   at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183)
>   at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>   at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>   at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>   at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>   at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>   at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>   at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>   at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>   at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>   at org.mortbay.jetty.Server.handle(Server.java:326)
>   at 
> org.mortbay.jetty.HttpConnection.h

Re: How can I get the number files on HDFS have been visited and who have visited them?

2014-09-28 Thread Yu Yang

Do you mean the namenode logs in $HADOOP_LOG_DIR  ?



2014-09-29 10:28 GMT+08:00 Bing Jiang :

> Could you turn on hdfs audit log? It tracks the access to hdfs system.
>
> 2014-09-29 10:22 GMT+08:00 Yu Yang :
>
>> Hey Guys,
>>
>> I want to count the number files on HDFS have been visited and who have
>> visited them.
>> Is there an API which I can use to do this ?
>>
>> thank you.
>>
>
>
>
> --
> Bing Jiang
> weibo: http://weibo.com/jiangbinglover
> BLOG: www.binospace.com
> BLOG: http://blog.sina.com.cn/jiangbinglover
> Focus on distributed computing, HDFS/HBase
>

Re: How can I get the number files on HDFS have been visited and who have visited them?

2014-09-28 Thread Bing Jiang

Could you turn on hdfs audit log? It tracks the access to hdfs system.

2014-09-29 10:22 GMT+08:00 Yu Yang :

> Hey Guys,
>
> I want to count the number files on HDFS have been visited and who have
> visited them.
> Is there an API which I can use to do this ?
>
> thank you.
>



-- 
Bing Jiang
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase

How can I get the number files on HDFS have been visited and who have visited them?

2014-09-28 Thread Yu Yang

Hey Guys,

I want to count the number files on HDFS have been visited and who have
visited them.
Is there an API which I can use to do this ?

thank you.

Re: Re: Regarding HDFS and YARN support for S3

2014-09-28 Thread Takenori Sato

Hi,

You may want to check HADOOP-10400
 for the overhaul of S3
filesystem fixed in 2.6.

The subclass of AbstractFileSystem was filed as HADOOP-10643
, but which was not
included in HADOOP-10400 though I made a comment

.

I suggest not to use S3 as defaultFS as commented in "Why you cannot use S3
as a replacement for HDFS " to
avoid all sorts of these issues.

The best practice is to use S3 as a supplementary solution to Hadoop in
order to bring life cycle management(expiration and tiering), and
source/destination over the internet.

Thanks,
Takenori


On Sun, Sep 28, 2014 at 5:23 PM, Naganarasimha G R (Naga) <
garlanaganarasi...@huawei.com> wrote:

>  Hi Jay,
> Thanks a lot for replying and it clarifies most of it, but still some
> parts are not so clear .
> Some clarifications from my side :
> *| When you say "HDFS does not support fs.AbstractFileSystem.s3.impl"
> That is true.  If your file system is configured using HDFS, then s3 urls
> will not be used, ever.*
> :) i think i am not doing this basic mistake . What we have done is we
> have configured *"viewfs://nsX" for "fs.defaultFS"* and one of the mount
> is S3 i.e. *"fs.viewfs.mounttable.nsX.link./uds" to "s3://hadoop/test1/"*
> .
> So it fails to even create YARNRunner instance as there is no mapping for
> *"**fs.AbstractFileSystem.s3.impl" *if run "./yarn jar"*. *But as per the
> code even if set *"fs.defaultFS"* to s3 it will not work as there is no
> mapping for S3's impl of AbstractFileSystem interface.
>
>  These are my further queries
>
>1. Whats the purpose of *AbstractFileSystem *and *FileSystem *
>interfaces?
>2. Does HDFS default package(code) support configuration of S3 ? I see
>S3 implementation of *FileSystem* interface(
>*org.apache.hadoop.fs.s3.S3FileSystem*) *but not for **AbstractFileSystem
>**!. *So i presume it doesn't support S3 completely. Whats the reason
>for not supporting both ?
>3. Suppose if i need to support Amazon S3 do i need to extend and
>implement *AbstractFileSystem *and configure  
> *"**fs.AbstractFileSystem.s3.impl"
>*or some thing more than this i need to take care*?*
>
>Regards,
>
> Naga
>
>
>
> Huawei Technologies Co., Ltd.
> Phone:
> Fax:
> Mobile:  +91 9980040283
> Email: naganarasimh...@huawei.com
> Huawei Technologies Co., Ltd.
> Bantian, Longgang District,Shenzhen 518129, P.R.China
> http://www.huawei.com
>
>
>--
> *From:* jay vyas [jayunit100.apa...@gmail.com]
> *Sent:* Saturday, September 27, 2014 02:41
> *To:* common-u...@hadoop.apache.org
> *Subject:* Re:
>
>  See https://wiki.apache.org/hadoop/HCFS/
>
> YES Yarn is written to the FileSystem interface.  It works on S3FileSystem
> and GlusterFileSystem and any other HCFS.
>
>  We have run , and continue to run, the many tests in apache bigtop's test
> suite against our hadoop clusters running on alternative file system
> implementations,
>  and it works.
>
>  When you say "HDFS does not support fs.AbstractFileSystem.s3.impl"
> That is true.  If your file system is configured using HDFS, then s3 urls
> will not be used, ever.
>
>  When you create a FileSystem object in hadoop, it reads the uri (i.e.
> "glusterfs:///") and then finds the file system binding in your
> core-site.xml (i.e. fs.AbstractFileSystem.glusterfs.impl).
>
>  So the URI must have a corresponding entry in the core-site.xml.
>
>  As a reference implementation, you can see
> https://github.com/gluster/glusterfs-hadoop/blob/master/conf/core-site.xml
>
>
>
>
> On Fri, Sep 26, 2014 at 10:10 AM, Naganarasimha G R (Naga) <
> garlanaganarasi...@huawei.com> wrote:
>
>>   Hi All,
>>
>>  I have following doubts on pluggable FileSystem and YARN
>> 1. If all the implementations should extend FileSystem then why there is
>> a parallel class AbstractFileSystem. which ViewFS extends ?
>> 2. Is YARN supposed to run on any of the pluggable
>> org.apache.hadoop.fs.FileSystem like s3 ?
>> if its suppose to run then when submitting a job in the client side
>>  YARNRunner is calling FileContext.getFileContext(this.conf);
>> which is further calling FileContext.getAbstractFileSystem() which throws
>> exception for S3.
>> So i am not able to run YARN job with ViewFS with S3 as mount. And based
>> on the code even if i configure only S3 then also its going to fail.
>> 3. HDFS does not support "fs.AbstractFileSystem.s3.impl" with some
>> default class similar to org.apache.hadoop.fs.s3.S3FileSystem ?
>>
>>Regards,
>>
>> Naga
>>
>>
>>
>> Huawei Technologies Co., Ltd.
>> Phone:
>> Fax:
>> Mobile:  +91 9980040283
>> Email: naganarasimh...@huawei.com
>> Huawei Technologies Co., Ltd.
>> http://www.huawei.com
>>
>>
>>

Re: Failed to active namenode when config HA

2014-09-28 Thread Matt Narrell

I’m pretty sure HDFS HA is relegated to two name nodes (not four), designated 
active and standby.  Secondly, I believe these properties should be in 
hdfs-site.xml NOT core-site.xml.

Furthermore, I think your HDFS nameservices are misconfigured.  Consider the 
following:



  
dfs.replication
3
  
  
dfs.namenode.name.dir
file:/var/data/hadoop/hdfs/nn
  
  
dfs.datanode.data.dir
file:/var/data/hadoop/hdfs/dn
  


  dfs.ha.automatic-failover.enabled
  true


  dfs.nameservices
  hdfs-cluster



  dfs.ha.namenodes.hdfs-cluster
  nn1,nn2

  
dfs.namenode.rpc-address.hdfs-cluster.nn1
namenode1:8020
  
  
dfs.namenode.http-address.hdfs-cluster.nn1
namenode1:50070
  
  
dfs.namenode.rpc-address.hdfs-cluster.nn2
namenode2:8020
  
  
dfs.namenode.http-address.hdfs-cluster.nn2
namenode2:50070
  


  dfs.namenode.shared.edits.dir
  
qjournal://journalnode1:8485;journalnode2:8485;journalnode3:8485/hdfs-cluster



  dfs.client.failover.proxy.provider.hdfs-cluster
  
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider



  dfs.ha.fencing.methods
  sshfence


  dfs.ha.fencing.ssh.private-key-files
  /home/hadoop/.ssh/id_rsa



mn

On Sep 28, 2014, at 12:56 PM, 清如许 <475053...@qq.com> wrote:

> Hi,
> 
> I'm new to hadoop and meet some problems when config HA.
> Below are some important configuration in core-site.xml
> 
>   
> dfs.nameservices
> ns1,ns2
>   
>   
> dfs.ha.namenodes.ns1
> nn1,nn3
>   
>   
> dfs.ha.namenodes.ns2
> nn2,nn4
>   
>   
> dfs.namenode.rpc-address.ns1.nn1
> namenode1:9000
>   
>   
> dfs.namenode.rpc-address.ns1.nn3
> namenode3:9000
>   
>   
> dfs.namenode.rpc-address.ns2.nn2
> namenode2:9000
>   
>   
> dfs.namenode.rpc-address.ns2.nn4
> namenode4:9000
>   
>   
> dfs.namenode.shared.edits.dir
> qjournal://datanode2:8485;datanode3:8485;datanode4:8485/ns1
>   
>   
> dfs.client.failover.proxy.provider.ns1
> 
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>   
>   
> dfs.ha.fencing.methods
> sshfence
>   
>   
> dfs.ha.fencing.ssh.private-key-files
> /home/hduser/.ssh/id_rsa
>   
>   
> dfs.ha.fencing.ssh.connect-timeout
> 3
>   
>   
> dfs.journalnode.edits.dir
> /home/hduser/mydata/hdfs/journalnode
>   
> 
> (two nameservice ns1,ns2 is for configuring federation later. In this step, I 
> only want launch ns1 on namenode1,namenode3)
> 
> After configuration, I did the following steps
> firstly,  I start jornalnode on datanode2,datanode3,datanode4
> secondly I format datanode1 and start namenode on it
> then i run 'hdfs namenode -bootstrapStandby' on the other namenode and start 
> namenode on it
> 
> Everything seems fine unless no namenode is active now, then i tried to 
> active one by running 
> hdfs haadmin -transitionToActive nn1 on namenode1
> but strangely it says "Illegal argument: Unable to determine the nameservice 
> id."
> 
> Could anyone tell me why it cannot determine nn1 from my configuration?
> Is there something wrong in my configuraion?
> 
> Thanks a lot!!!
> 
>

Hadoop UI - Unable to connect to the application master from the Hadoop UI.

2014-09-28 Thread S.L

Hi All,

I am running a 3 node Apache Hadoop YARN 2.3.0  cluster , after a job is
submitted , when I ascess the application master from the UI I get the
following exception  and I am unable to connect to the Application Master
from the UI, can someone let me know , what I need  to look at?

HTTP ERROR 500

Problem accessing /proxy/application_1411841629814_0032/. Reason:

Connection refused

Caused by:

java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at 
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.(Socket.java:425)
at java.net.Socket.(Socket.java:280)
at 
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
at 
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
at 
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at 
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at 
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at 
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at 
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at 
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.proxyLink(WebAppProxyServlet.java:185)
at 
org.apache.hadoop.yarn.server.webproxy.WebAppProxyServlet.doGet(WebAppProxyServlet.java:336)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:66)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
at 
com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
at 
com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
at 
com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at 
org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at 
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1183)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

Re: Doubt Regarding QJM protocol - example 2.10.6 of Quorum-Journal Design document

2014-09-28 Thread Ulul


Hi

A developer should answer that but a quick look to an edit file with od 
suggests that record are not fixed length. So maybe the likeliness of 
the situation you suggest is so low that there is no need to check more 
than file size


Ulul

Le 28/09/2014 11:17, Giridhar Addepalli a écrit :

Hi All,

I am going through Quorum Journal Design document.

It is mentioned in Section 2.8 - In Accept Recovery RPC section
"
If the current on-disk log is missing, or a /different length /than 
the proposed recovery, the JN downloads the log from the provided URI, 
replacing any current copy of the log segment.

"

I can see it that the code follows above design

Source :: Journal.java
 

  public synchronized void acceptRecovery(RequestInfo reqInfo,
  SegmentStateProto segment, URL fromUrl)
  throws IOException {

  
  if (currentSegment == null ||
currentSegment.getEndTxId() != segment.getEndTxId()) {
  
  } else {
  LOG.info("Skipping download of log " +
  TextFormat.shortDebugString(segment) +
  ": already have up-to-date logs");
  }
  
  }


My question is what if on-disk log is present and is of /same length 
/as the proposed recovery


If JournalNode is skipping download because the logs are of same 
length, then we could end up in a situation where finalized log 
segments contain different data !


This could happen if we follow example 2.10.6

As per that example we write transactions (151-153 ) on JN1
then when recovery proceeded with only JN2 & JN3 let us assume that we 
write again /different transactions/ as (151-153) . Then after the 
crash when we run recovery , JN1 will skip downloading correct segment 
from JN2/JN3 as it thinks it has correct segment( as per the code 
pasted above). This will result in a situation where finalized segment 
( edits_151-153 ) on JN1 is different from finalized segment 
edits_151-153 on JN2/JN3.


Please let me know if i have gone wrong some where, and this situation 
is taken care of.


Thanks,
Giridhar.

Failed to active namenode when config HA

2014-09-28 Thread ??????

Hi,

I'm new to hadoop and meet some problems when config HA.
Below are some important configuration in core-site.xml

  
dfs.nameservices
ns1,ns2
  
  
dfs.ha.namenodes.ns1
nn1,nn3
  
  
dfs.ha.namenodes.ns2
nn2,nn4
  
  
dfs.namenode.rpc-address.ns1.nn1
namenode1:9000
  
  
dfs.namenode.rpc-address.ns1.nn3
namenode3:9000
  
  
dfs.namenode.rpc-address.ns2.nn2
namenode2:9000
  
  
dfs.namenode.rpc-address.ns2.nn4
namenode4:9000
  
  
dfs.namenode.shared.edits.dir
qjournal://datanode2:8485;datanode3:8485;datanode4:8485/ns1
  
  
dfs.client.failover.proxy.provider.ns1

org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
  
  
dfs.ha.fencing.methods
sshfence
  
  
dfs.ha.fencing.ssh.private-key-files
/home/hduser/.ssh/id_rsa
  
  
dfs.ha.fencing.ssh.connect-timeout
3
  
  
dfs.journalnode.edits.dir
/home/hduser/mydata/hdfs/journalnode
  

(two nameservice ns1,ns2 is for configuring federation later. In this step, I 
only want launch ns1 on namenode1,namenode3)

After configuration, I did the following steps
firstly,  I start jornalnode on datanode2,datanode3,datanode4
secondly I format datanode1 and start namenode on it
then i run 'hdfs namenode -bootstrapStandby' on the other namenode and start 
namenode on it

Everything seems fine unless no namenode is active now, then i tried to active 
one by running 
hdfs haadmin -transitionToActive nn1 on namenode1
but strangely it says "Illegal argument: Unable to determine the nameservice 
id."

Could anyone tell me why it cannot determine nn1 from my configuration?
Is there something wrong in my configuraion?

Thanks a lot!!!

Doubt Regarding QJM protocol - example 2.10.6 of Quorum-Journal Design document

2014-09-28 Thread Giridhar Addepalli

Hi All,

I am going through Quorum Journal Design document.

It is mentioned in Section 2.8 - In Accept Recovery RPC section
"
If the current on-disk log is missing, or a *different length *than the
proposed recovery, the JN downloads the log from the provided URI,
replacing any current copy of the log segment.
"

I can see it that the code follows above design

Source :: Journal.java
 

  public synchronized void acceptRecovery(RequestInfo reqInfo,
  SegmentStateProto segment, URL fromUrl)
  throws IOException {

  
  if (currentSegment == null ||
currentSegment.getEndTxId() != segment.getEndTxId()) {
  
  } else {
  LOG.info("Skipping download of log " +
  TextFormat.shortDebugString(segment) +
  ": already have up-to-date logs");
  }
  
  }


My question is what if on-disk log is present and is of *same length *as
the proposed recovery

If JournalNode is skipping download because the logs are of same length,
then we could end up in a situation where finalized log segments contain
different data !

This could happen if we follow example 2.10.6

As per that example we write transactions (151-153 ) on JN1
then when recovery proceeded with only JN2 & JN3 let us assume that we
write again *different transactions* as (151-153) . Then after the crash
when we run recovery , JN1 will skip downloading correct segment from
JN2/JN3 as it thinks it has correct segment( as per the code pasted above).
This will result in a situation where finalized segment ( edits_151-153 )
on JN1 is different from finalized segment edits_151-153 on JN2/JN3.

Please let me know if i have gone wrong some where, and this situation is
taken care of.

Thanks,
Giridhar.

RE: Re: Regarding HDFS and YARN support for S3

2014-09-28 Thread Naganarasimha G R (Naga)

Hi Jay,
Thanks a lot for replying and it clarifies most of it, but still some parts are 
not so clear .
Some clarifications from my side :
| When you say "HDFS does not support fs.AbstractFileSystem.s3.impl" That 
is true.  If your file system is configured using HDFS, then s3 urls will not 
be used, ever.
:) i think i am not doing this basic mistake . What we have done is we have 
configured "viewfs://nsX" for "fs.defaultFS" and one of the mount is S3 i.e. 
"fs.viewfs.mounttable.nsX.link./uds" to "s3://hadoop/test1/".
So it fails to even create YARNRunner instance as there is no mapping for 
"fs.AbstractFileSystem.s3.impl" if run "./yarn jar". But as per the code even 
if set "fs.defaultFS" to s3 it will not work as there is no mapping for S3's 
impl of AbstractFileSystem interface.

These are my further queries

  1.  Whats the purpose of AbstractFileSystem and FileSystem interfaces?
  2.  Does HDFS default package(code) support configuration of S3 ? I see S3 
implementation of FileSystem interface(org.apache.hadoop.fs.s3.S3FileSystem) 
but not for AbstractFileSystem !. So i presume it doesn't support S3 
completely. Whats the reason for not supporting both ?
  3.  Suppose if i need to support Amazon S3 do i need to extend and implement 
AbstractFileSystem and configure  "fs.AbstractFileSystem.s3.impl" or some thing 
more than this i need to take care?

Regards,

Naga



Huawei Technologies Co., Ltd.
Phone:
Fax:
Mobile:  +91 9980040283
Email: naganarasimh...@huawei.com
Huawei Technologies Co., Ltd.
Bantian, Longgang District,Shenzhen 518129, P.R.China
http://www.huawei.com



From: jay vyas [jayunit100.apa...@gmail.com]
Sent: Saturday, September 27, 2014 02:41
To: common-u...@hadoop.apache.org
Subject: Re:

See https://wiki.apache.org/hadoop/HCFS/

YES Yarn is written to the FileSystem interface.  It works on S3FileSystem and 
GlusterFileSystem and any other HCFS.

We have run , and continue to run, the many tests in apache bigtop's test suite 
against our hadoop clusters running on alternative file system implementations,
and it works.

When you say "HDFS does not support fs.AbstractFileSystem.s3.impl" That is 
true.  If your file system is configured using HDFS, then s3 urls will not be 
used, ever.

When you create a FileSystem object in hadoop, it reads the uri (i.e. 
"glusterfs:///") and then finds the file system binding in your core-site.xml 
(i.e. fs.AbstractFileSystem.glusterfs.impl).

So the URI must have a corresponding entry in the core-site.xml.

As a reference implementation, you can see 
https://github.com/gluster/glusterfs-hadoop/blob/master/conf/core-site.xml




On Fri, Sep 26, 2014 at 10:10 AM, Naganarasimha G R (Naga) 
mailto:garlanaganarasi...@huawei.com>> wrote:
Hi All,

I have following doubts on pluggable FileSystem and YARN
1. If all the implementations should extend FileSystem then why there is a 
parallel class AbstractFileSystem. which ViewFS extends ?
2. Is YARN supposed to run on any of the pluggable 
org.apache.hadoop.fs.FileSystem like s3 ?
if its suppose to run then when submitting a job in the client side  YARNRunner 
is calling FileContext.getFileContext(this.conf);
which is further calling FileContext.getAbstractFileSystem() which throws 
exception for S3.
So i am not able to run YARN job with ViewFS with S3 as mount. And based on the 
code even if i configure only S3 then also its going to fail.
3. HDFS does not support "fs.AbstractFileSystem.s3.impl" with some default 
class similar to org.apache.hadoop.fs.s3.S3FileSystem ?


Regards,

Naga



Huawei Technologies Co., Ltd.
Phone:
Fax:
Mobile:  +91 9980040283
Email: naganarasimh...@huawei.com
Huawei Technologies Co., Ltd.
http://www.huawei.com




--
jay vyas

Hadoop shutdown scripts failing

Re: No space when running a hadoop job

Re: Hadoop UI - Unable to connect to the application master from the Hadoop UI.

Re: How can I get the number files on HDFS have been visited and who have visited them?

Re: How can I get the number files on HDFS have been visited and who have visited them?

How can I get the number files on HDFS have been visited and who have visited them?

Re: Re: Regarding HDFS and YARN support for S3

Re: Failed to active namenode when config HA

Hadoop UI - Unable to connect to the application master from the Hadoop UI.

Re: Doubt Regarding QJM protocol - example 2.10.6 of Quorum-Journal Design document

Failed to active namenode when config HA

Doubt Regarding QJM protocol - example 2.10.6 of Quorum-Journal Design document

RE: Re: Regarding HDFS and YARN support for S3

13 matches

Site Navigation

Mail list logo

Footer information