Here is a tested step by step.
curl -i -k -u guest:guest-password -X DELETE \
'https://localhost:8443/gateway/sandbox/webhdfs/v1/user/guest/test?op=DELETE&recursive=true'
curl -i -k -u guest:guest-password -X PUT \
'https://localhost:8443/gateway/sandbox/webhdfs/v1/user/guest/test/input?op=MKDIRS&permission=777'
curl -i -k -u guest:guest-password -X PUT \
"https://localhost:8443/gateway/sandbox/webhdfs/v1/user/guest/test/lib?op=MKDIRS&permission=777";
curl -i -k -u guest:guest-password -X PUT \
'https://localhost:8443/gateway/sandbox/webhdfs/v1/user/guest/test/lib/hadoop-examples.jar?op=CREATE'
curl -i -k -u guest:guest-password -T samples/hadoop-examples.jar -X PUT \
'{Value of Location header from command above}'
curl -i -k -u guest:guest-password -X PUT \
'https://localhost:8443/gateway/sandbox/webhdfs/v1/user/guest/test/input/README?op=CREATE'
curl -i -k -u guest:guest-password -T README -X PUT \
'{Value of Location header from command above}'
curl -i -k -u guest:guest-password -X GET \
'https://localhost:8443/gateway/sandbox/templeton/v1/jobs'
curl -i -k -u guest:guest-password -X POST \
-d arg=/user/guest/test/input -d arg=/user/guest/test/output \
-d jar=/user/guest/test/lib/hadoop-examples.jar \
-d class=wordcount \
'https://localhost:8443/gateway/sandbox/templeton/v1/mapreduce/jar'
curl -i -k -u guest:guest-password -X GET \
'https://localhost:8443/gateway/sandbox/templeton/v1/queue/{JobID from
command above}'
# Look for "jobComplete":false
curl -i -k -u guest:guest-password -X GET \
'https://localhost:8443/gateway/sandbox/webhdfs/v1/user/guest/test/output?op=LISTSTATUS'
curl -i -k -u guest:guest-password -X GET \
'https://localhost:8443/gateway/sandbox/webhdfs/v1/user/guest/test/output/part-r-00000?op=OPEN'
curl -i -k -u guest:guest-password -X GET \
'{Value of Location header from command above}'
On 11/22/13 4:12 PM, Dilli Arumugam wrote:
Yes, you should see out like the following:
curl -k -i -v -u guest:guest-password -X POST -d
arg=/user/guest/test/input -d arg=/user/guest/test/output -d
jar=/user/guest/test/lib/hadoop-examples.jar -d class=wordcount
https://localhost:8443/gateway/sandbox/templeton/v1/mapreduce/jar
* About to connect() to localhost port 8443 (#0)
* Trying ::1...
* connected
* Connected to localhost (::1) port 8443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using EDH-RSA-DES-CBC3-SHA
* Server certificate:
* subject: C=US; ST=Test; L=Test; O=Hadoop; OU=Test; CN=localhost
* start date: 2013-11-22 20:29:42 GMT
* expire date: 2014-11-22 20:29:42 GMT
* common name: localhost (matched)
* issuer: C=US; ST=Test; L=Test; O=Hadoop; OU=Test; CN=localhost
* SSL certificate verify result: self signed certificate (18), continuing
anyway.
* Server auth using Basic with user 'guest'
POST /gateway/sandbox/templeton/v1/mapreduce/jar HTTP/1.1
Authorization: Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=
User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0
OpenSSL/0.9.8x zlib/1.2.5
Host: localhost:8443
Accept: */*
Content-Length: 115
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 115 out of 115 bytes
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Set-Cookie:
JSESSIONID=1stfwywf2gt7wkrwv7iakl8q0;Path=/gateway/sandbox;Secure
Set-Cookie:
JSESSIONID=1stfwywf2gt7wkrwv7iakl8q0;Path=/gateway/sandbox;Secure
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Server: Jetty(7.6.0.v20120127)
Server: Jetty(7.6.0.v20120127)
< Content-Type: application/json
Content-Type: application/json
< Content-Length: 31
Content-Length: 31
<
* Connection #0 to host localhost left intact
*{"id":"job_1384985924282_0016"}** Closing connection #0
* SSLv3, TLS alert, Client hello (1):
On Fri, Nov 22, 2013 at 12:50 PM, Vinay Shukla <[email protected]>wrote:
Thanks Dilli,
When I submit the job through Knox with curl -k -i -v -u
guest:guest-password -X POST -d arg=/user/guest/test/input -d
arg=/user/guest/test/output -d
jar=/user/guest/test/lib/hadoop-examples.jar -d class=wordcount
https://localhost:8443/gateway/sandbox/templeton/v1/mapreduce/jar
I don't see job id in the output. Shouldn't the output have it?
W/o the job id how is the user going to query the job status or get the job
output?
On Fri, Nov 22, 2013 at 11:51 AM, Dilli Arumugam
<[email protected]>wrote:
curl -k -i -v -u guest:guest-password -X POST -d
arg=/user/guest/test/input -d arg=/user/guest/test/output -d
jar=/user/guest/test/lib/hadoop-examples.jar -d class=wordcount
https://localhost:8443/gateway/sandbox/templeton/v1/mapreduce/jar
On Fri, Nov 22, 2013 at 11:41 AM, Dilli Arumugam
<[email protected]>wrote:
Sorry, I sent you example for curl access to hadoop cluster direct.
Would send you another example that goes through knox shortly
Dilli
On Fri, Nov 22, 2013 at 11:26 AM, Vinay Shukla <[email protected]
wrote:
Dilli,
This was a great example and help a lot. Unfortunately in the last
step
cURL directly cURL calls templeton and bypasses Knox. How can I do it
through Knox?
-Thanks,
Vinay
On Fri, Nov 22, 2013 at 10:40 AM, Kevin Minder <
[email protected]
wrote:
Well that certainly is a whole lot easier than what I provided.
Thanks
Dilli!
On 11/22/13 1:37 PM, Dilli Arumugam wrote:
curl -i -X POST -d arg=/user/guest/test/input -d
arg=/user/guest/test/output -d
jar=/user/guest/test/lib/hadoop-examples.jar -d class=wordcount
http://localhost:50111/templeton/v1/mapreduce/jar?user.name=guest
Pre req*usites on** HDFS*
/user/guest/test/output shoud NOT exisit
/user/guest/test/input should exist with some text files
/user/guest/test/lib/hadoop-examples.jar should exisit
Thanks
Dilli
On Fri, Nov 22, 2013 at 9:15 AM, Vinay Shukla <
[email protected]
wrote:
This is go in the user guide. Currently I see instructions, for
submitting
a job (rather a multi steps process to submit Oozie workflow) with
Oozie
but don't see anything directly with Templeton.
Wonder what should we offer our users?
On Fri, Nov 22, 2013 at 9:09 AM, Kevin Minder
<[email protected]>wrote:
That is going to be enormously long but I'm sure I had one at one
time.
I'll look for it or otherwise re-figure out how to turn on
client
side
HTTP logging so that you can capture the URLs and bodies.
On 11/22/13 12:06 PM, Vinay Shukla wrote:
Is there a sample showing how to run this without using
ClientDSL
or
Groovy
script?
I am looking for a sample directly using cURL.
Thanks,
Vinay
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or
entity
to which it is addressed and may contain information that is
confidential,
privileged and exempt from disclosure under applicable law. If
the
reader
of this message is not the intended recipient, you are hereby
notified
that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you
have
received this communication in error, please contact the sender
immediately
and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or
entity
to which it is addressed and may contain information that is
confidential,
privileged and exempt from disclosure under applicable law. If the
reader
of this message is not the intended recipient, you are hereby
notified
that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately
and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity
to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified
that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately
and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
