I am trying to use WebHDFS to write two files to Hadoop. One is an image file
and the other is an index file. I am able to create both files, append twice to
both files, but when I try to append a 3rd time I get the following error:
"{"RemoteException":{"exception":"AlreadyBeingCreatedException","javaClassName":"org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException","message":"Failed
to APPEND_FILE /PSG/VideoVault/VideoStorage/VICADS-100/20160513.img for
DFSClient_NONMAPREDUCE_1168342942_28 on 192.168.1.211 because this file lease
is currently owned by DFSClient_NONMAPREDUCE_-2117428583_27 on
192.168.1.211\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2684)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2991)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2960)\n\tat
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:719)\n\tat
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:421)\n\tat
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)\n\tat
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)\n\tat
java.security.AccessController.doPrivileged(Native Method)\n\tat
javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)\n\tat
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)\n"}}"
WARN [net.psgglobal.videovault.recorder.JpegWriter] (WriterThread) Error
Writting JPeg: Error appending to file at Data Node
http://vv1.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020:
403 Forbidden
Any idea what I may be doing wrong? Is there something I need to know about the
file leases when using WebHDFS?
Here is the HTTP communication between the client and the WebHDFS server with
some annotations added.
--------------- CREATE IMG FILE ----------------
>> "PUT
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?user.name=hdfs&op=CREATE&overwrite=false
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: 192.168.1.201:50070[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Cookie:
>> hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1463185339054&s=QNjFxsvYO5jo1j5eSLM4rADC988="[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<< "HTTP/1.1 307 TEMPORARY_REDIRECT[\r][\n]"
<< "Cache-Control: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Content-Type: application/octet-stream[\r][\n]"
<< "Location:
http://vv2.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=CREATE&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020&createflag=&createparent=true&overwrite=false[\r][\n]";
<< "Content-Length: 0[\r][\n]"
<< "Server: Jetty(6.1.26.hwx)[\r][\n]"
<< "[\r][\n]"
>> "PUT
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=CREATE&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020&createflag=&createparent=true&overwrite=false
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: vv2.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 201 Created[\r][\n]"
<< "Location:
hdfs://vb1.localdomain:8020/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"
--------------- APPEND IMG FILE ----------------
>> "POST
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?user.name=hdfs&op=APPEND
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: 192.168.1.201:50070[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Cookie:
>> hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1463185339054&s=QNjFxsvYO5jo1j5eSLM4rADC988="[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<< "HTTP/1.1 307 TEMPORARY_REDIRECT[\r][\n]"
<< "Cache-Control: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Content-Type: application/octet-stream[\r][\n]"
<< "Location:
http://vv1.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020[\r][\n]";
<< "Content-Length: 0[\r][\n]"
<< "Server: Jetty(6.1.26.hwx)[\r][\n]"
<< "[\r][\n]"
>> "POST
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 52685[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv1.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>
<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"
--------------- APPEND IDX FILE ----------------
>> "PUT
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?user.name=hdfs&op=CREATE&overwrite=false
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: 192.168.1.201:50070[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Cookie:
>> hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1463185339054&s=QNjFxsvYO5jo1j5eSLM4rADC988="[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<< "HTTP/1.1 307 TEMPORARY_REDIRECT[\r][\n]"
<< "Cache-Control: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Content-Type: application/octet-stream[\r][\n]"
<< "Location:
http://vv2.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=CREATE&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020&createflag=&createparent=true&overwrite=false[\r][\n]";
<< "Content-Length: 0[\r][\n]"
<< "Server: Jetty(6.1.26.hwx)[\r][\n]"
<< "[\r][\n]"
>> "PUT
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=CREATE&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020&createflag=&createparent=true&overwrite=false
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: vv2.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 201 Created[\r][\n]"
<< "Location:
hdfs://vb1.localdomain:8020/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"
--------------- APPEND IDX FILE ----------------
>> "POST
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?user.name=hdfs&op=APPEND
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 0[\r][\n]"
>> "Host: 192.168.1.201:50070[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Cookie:
>> hadoop.auth="u=hdfs&p=hdfs&t=simple&e=1463185339054&s=QNjFxsvYO5jo1j5eSLM4rADC988="[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<< "HTTP/1.1 307 TEMPORARY_REDIRECT[\r][\n]"
<< "Cache-Control: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Expires: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Date: Fri, 13 May 2016 14:22:19 GMT[\r][\n]"
<< "Pragma: no-cache[\r][\n]"
<< "Content-Type: application/octet-stream[\r][\n]"
<< "Location:
http://vv2.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020[\r][\n]";
<< "Content-Length: 0[\r][\n]"
<< "Server: Jetty(6.1.26.hwx)[\r][\n]"
<< "[\r][\n]"
>> "POST
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 24[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv2.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>
<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"
--------------- APPEND IMG FILE ----------------
>> "POST
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 52685[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv1.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>
<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"
--------------- APPEND IDX FILE ----------------
>> "POST
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.idx?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 24[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv2.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>
<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 200 OK[\r][\n]"
<< "Content-Length: 0[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"
--------------- APPEND IMG FILE ----------------
>> "POST
>> /webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020
>> HTTP/1.1[\r][\n]"
>> "Content-Length: 54623[\r][\n]"
>> "Content-Type: application/octet-stream[\r][\n]"
>> "Host: vv1.localdomain:50075[\r][\n]"
>> "Connection: Keep-Alive[\r][\n]"
>> "User-Agent: Apache-HttpClient/4.5.1 (Java/1.8.0_91)[\r][\n]"
>> "Accept-Encoding: gzip,deflate[\r][\n]"
>> "[\r][\n]"
<<< DATA HERE >>>
<< "HTTP/1.1 100 Continue[\r][\n]"
<< "[\r][\n]"
<< "HTTP/1.1 403 Forbidden[\r][\n]"
<< "Content-Type: application/json; charset=utf-8[\r][\n]"
<< "Content-Length: 1756[\r][\n]"
<< "Connection: close[\r][\n]"
<< "[\r][\n]"
<<
"{"RemoteException":{"exception":"AlreadyBeingCreatedException","javaClassName":"org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException","message":"Failed
to APPEND_FILE /PSG/VideoVault/VideoStorage/VICADS-100/20160513.img for
DFSClient_NONMAPREDUCE_1168342942_28 on 192.168.1.211 because this file lease
is currently owned by DFSClient_NONMAPREDUCE_-2117428583_27 on
192.168.1.211\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2937)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:2684)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2991)\n\tat
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2960)\n\tat
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:719)\n\tat
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:421)\n\tat
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n\tat
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)\n\tat
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)\n\tat
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)\n\tat
java.security.AccessController.doPrivileged(Native Method)\n\tat
javax.security.auth.Subject.doAs(Subject.java:422)\n\tat
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)\n\tat
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)\n"}}"
WARN [net.psgglobal.videovault.recorder.JpegWriter] (WriterThread) Error
Writting JPeg: Error appending to file at Data Node
http://vv1.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020:
403 Forbidden
ERROR [stderr] (WriterThread) java.io.IOException: Error appending to file at
Data Node
http://vv1.localdomain:50075/webhdfs/v1/PSG/VideoVault/VideoStorage/VICADS-100/20160513.img?op=APPEND&user.name=hdfs&namenoderpcaddress=vb1.localdomain:8020:
403 Forbidden
ERROR [stderr] (WriterThread) at
net.psgglobal.hdfs.videovault.videostorage.WebHdfsFile.appendToFile(WebHdfsFile.java:252)
ERROR [stderr] (WriterThread) at
net.psgglobal.hdfs.videovault.videostorage.JpegImageFile.appendJpeg(JpegImageFile.java:43)
ERROR [stderr] (WriterThread) at
net.psgglobal.hdfs.videovault.videostorage.VideoVaultRecordFile.writeToFile(VideoVaultRecordFile.java:40)
ERROR [stderr] (WriterThread) at
net.psgglobal.videovault.recorder.JpegWriter$WriterThread.run(JpegWriter.java:95)
Thanks, David
David R Robison
Senior Systems Engineer
O. +1 512 247 3700
M. +1 512 608 3173
david.robi...@psgglobal.net<mailto:david.robi...@psgglobal.net>
www.psgglobal.net<http://www.psgglobal.net/>
Prometheus Security Group Global, Inc.
3019 Alvin Devane Boulevard
Building 4, Suite 450
Austin, TX 78741
