I am using hadoop trunk (forked from github). It supports RESTful APIs as I am able to retrieve JSON objects for RM (cluster/nodes info)+ Historyserver. The only issue is with AppMaster REST API.
Regards, Prajakta On Fri, Jul 6, 2012 at 10:55 PM, Robert Evans <ev...@yahoo-inc.com> wrote: > What version of hadoop are you using? It could be that the version you > have does not have the RESTful APIs in it yet, and the proxy is working > just fine. > > --Bobby Evans > > On 7/6/12 12:06 PM, "Prajakta Kalmegh" <pkalm...@gmail.com> wrote: > > >Robert , Thanks for the response. If I do not provide any explicit > >configuration for the proxy server, do I still need to start it using the > >'yarn start proxy server'? I am currently not doing it. > > > >Also, I am able to access the html page for proxy using the > ><http://localhost:8088/proxy/{appid}/mapreduce/jobs> URL. (Note this url > >does not have the '/ws/v1/ part in it. I get the html response when I > >query > >for this URL in runtime. > > > >So I assume the proxy server must be starting fine since I am able to > >access this URL. I will try logging more details tomorrow from my office > >machine and will let you know the result. > > > >Regards, > >Prajakta > > > > > > > >On Fri, Jul 6, 2012 at 10:22 PM, Robert Evans <ev...@yahoo-inc.com> > wrote: > > > >> Sorry I did not respond sooner. The default behavior is to have the > >>proxy > >> server run as part of the RM. I am not really sure why it is not doing > >> this in your case. If you set the config yourself to be a URI that is > >> different from that of the RM then you need to launch a standalone proxy > >> server. You can do this by running > >> > >> yarn start proxy server > >> > >> Without sitting down with you it is going to be somewhat difficult to > >> debug why this is happening. However, in retrospect it would be nice to > >> add in some extra logging to help indicate why the proxy server is not > >> functioning as desired. If you could file a JIRA to add in the logging > >>I > >> would be happy to provide a patch to you and we can try and debug the > >> issue further. Please file it under the MAPREDUCE JIRA project. > >> > >> --Bobby > >> > >> On 7/6/12 3:29 AM, "Prajakta Kalmegh" <pkalm...@gmail.com> wrote: > >> > >> >Re-posting as I haven't got a solution yet. Sorry for spamming. I > >>won't be > >> >able to proceed in my code until I get a JSON response using AppMaster > >> >REST > >> >URL. :( > >> > > >> >Thanks, > >> >Prajakta > >> > > >> > > >> >On Wed, Jul 4, 2012 at 5:55 PM, Prajakta Kalmegh <pkalm...@gmail.com> > >> >wrote: > >> > > >> >> Hi Robert/Harsh > >> >> > >> >> Thanks for your reply. > >> >> > >> >> My RM is starting just fine. The problem is with the use of > >> >>http://<proxy httpddress:port>/proxy/{appid}/ws/v1/mapreduce > >> >> to get the JSON response. > >> >> > >> >> As I said before, I had not configured the yarn.web-proxy.address > >> >>property in yarn-site.xml. I assumed it will use the RM's > >> >>yarn.resourcemanager.webapp.address property value as default. > >>However, > >> >>it gives me a '404-Page not found error'. Today I tried specifying a > >> >>value explicitly for the yarn.web-proxy.address property. > >> >> > >> >> On running the wordcount example, it even gives a url > >> >><http://localhost:8090>/proxy/{appid}/> to track the App Mast info. > >> >>However, I am still not able to get a json response. > >> >> > >> >> Also, I tried to get the data from historyserver instead of runtime > >> >>using the instructions given on page > >> >>< > >> > http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yar > >> >>n-site/HistoryServerRest.html> > >> >> > >> >> HistoryServer REST response does not give me jobids corresponding to > >>an > >> >>application. It just lists all the jobs run until now. By the way, the > >> >>documentation does say > >> >> > >> >> ---------- > >> >> > >> >> "Both of the following URI's give you the history server information, > >> >>from an application id identified by the appid value. > >> >> * http://<history server http address:port>/ws/v1/history > >> >> * http://<history server http address:port>/ws/v1/history/info" > >> >> --------- > >> >> > >> >> But there is no provision to specify the application id with these > >>REST > >> >>URLs. > >> >> > >> >> Any idea how I can get the Application Master REST working and also > >> >>linking jobids to application id using the HistoryServerREST API? > >> >> > >> >> Any help is appreciated. Thanks in advance. > >> >> Regards, > >> >> Prajakta > >> >> > >> >> > >> >> > >> >> > >> >> On Fri, Jun 29, 2012 at 8:55 PM, Robert Evans <ev...@yahoo-inc.com> > >> >>wrote: > >> >> > >> >>> Please don't file that JIRA. The proxy server is intended to front > >>the > >> >>> web server for all calls to the AM. This is so you only have to go > >>to > >> >>>a > >> >>> single location to get to any AM's web service. The proxy server > >>is a > >> >>> very simple proxy and just forwards the extra part of the path on to > >> >>>the > >> >>> AM. > >> >>> > >> >>> If you are having issues with this please include the version you > >>are > >> >>> having problems with. Also please look at the logs for the RM on > >> >>>startup > >> >>> to see if there is anything there indicating why it is not starting > >>up. > >> >>> > >> >>> --Bobby Evans > >> >>> > >> >>> On 6/28/12 9:46 AM, "Harsh J" <ha...@cloudera.com> wrote: > >> >>> > >> >>> >As far as I can tell, the MR WebApp, as the name itself indicates > >>on > >> >>> >its doc page, starts only at the MR AM (which may be running at any > >> >>> >NM), and it starts as an ephemeral port logged at in the AM logs > >> >>> >usually as: > >> >>> > > >> >>> >INFO Web app /mapreduce started at [PORT] > >> >>> > > >> >>> >That it starts its own server with an ephemeral access point makes > >> >>> >sense, since each job uses its own AM and having a common location > >>may > >> >>> >not work with the form of REST API documented at your link. Can you > >> >>> >please file a JIRA to fix the doc and remove the proxy server refs, > >> >>> >which are misleading? > >> >>> > > >> >>> >Do correct me if I'm wrong. > >> >>> > > >> >>> >On Thu, Jun 28, 2012 at 6:13 PM, Prajakta Kalmegh > >><pkalm...@gmail.com > >> > > >> >>> >wrote: > >> >>> >> Hi > >> >>> >> > >> >>> >> I am trying to get the ApplicationMaster info using the > >> >>><http://<proxy > >> >>> >>http > >> >>> >> address:port>/proxy/{appid}/ws/v1/mapreduce/info> link as > >>described > >> >>>on > >> >>> >>the < > >> >>> >> > >> >>> >> > >> >>> > >> >>> > >> > http://hadoop.apache.org/common/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yar > >> >>>n > >> >>> >>-site/MapredAppMasterRest.html> > >> >>> >> page. > >> >>> >> > >> >>> >> I am able to access and retrieve JSON response for other modules > >> >>> >> (ResourceManager, NodeManager and HistoryServer). However, I am > >> >>>getting > >> >>> >> 'Page not found' when I try to use my ResourceManager Http > >>address > >> >>>to > >> >>> >> access the ApplicationMaster info. I am using < > >> >>> >> http://localhost:8088/proxy/{appid}/ws/v1/mapreduce/info> to > >> >>>retrieve > >> >>> >>JSON > >> >>> >> response. > >> >>> >> > >> >>> >> The instructions say "The application master should be accessed > >>via > >> >>>the > >> >>> >> proxy. This proxy is configurable to run either on the resource > >> >>>manager > >> >>> >>or > >> >>> >> on a separate host." > >> >>> >> > >> >>> >> My yarn-default.xml contains: > >> >>> >> <property> > >> >>> >> <description>The address for the web proxy as HOST:PORT, if > >>this > >> >>>is > >> >>> >>not > >> >>> >> given then the proxy will run as part of the RM</description> > >> >>> >> <name>yarn.web-proxy.address</name> > >> >>> >> <value/> > >> >>> >> </property> > >> >>> >> > >> >>> >> and I did not set a value explicitly in yarn-site.xml. Any idea > >> >>>how I > >> >>> >>can > >> >>> >> get this working? Thanks in advance. > >> >>> >> > >> >>> >> Regards, > >> >>> >> Prajakta > >> >>> > > >> >>> > > >> >>> > > >> >>> >-- > >> >>> >Harsh J > >> >>> > >> >>> > >> >> > >> > >> > >
