Makes sense, however, would it make sense to unpack to local file system, which 
would then avoid further HDFS ops?


> On 12 Apr 2017, at 13:24, Otto Fowler <[email protected]> wrote:
> 
> "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are 
> deployed
> if required.  So the writes for HDFS are the unpacking the bundles ( with the 
> unpacked jars being loaded into a classloader ).
> 
> If the unbundled is the same as the bundle, there is no op.  In this case, it 
> is first run.
> 
> So this is the parser bolt using the HDFS backed extension loader.”
> 
> 
> This is part of the parser side loading, also feeds into the stellar load 
> from hdfs stuff.
> 
> * metron has extensions
> * extensions are packaged with their configurations and their ‘bundle’ ( the 
> nar part )
> * extension bundles are deployed to hdfs *lib directory
> * parts of the system that use extensions discover and load them, with class 
> loaders setup for the bundle
> * part of that, the bundles are unpackaged into a ‘working’ area if required 
> ( so if you drop a new version into the lib, the next app will unpack the new 
> version )
> * thus, the bundle system needs to be able to write to the working areas in 
> hdfs
> * ???
> * profit
> 
> 
> On April 12, 2017 at 08:11:28, Simon Elliston Ball 
> ([email protected] <mailto:[email protected]>) wrote:
> 
>> I’m curious: Otto, what’s your use case for writing to HDFS from a parsers? 
>> 
>> Simon 
>> 
>> > On 12 Apr 2017, at 13:04, Justin Leet <[email protected] 
>> > <mailto:[email protected]>> wrote: 
>> >  
>> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and 
>> > with 775 will be able to write. 
>> >  
>> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <[email protected] 
>> > <mailto:[email protected]>> wrote: 
>> >  
>> >> It's curious to me that you're writing directly from parsing, but I 
>> >> suspect 
>> >> that your parsing topology is running as the storm user and it can't 
>> >> write 
>> >> to those directories. 
>> >>  
>> >> -D... 
>> >>  
>> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <[email protected] 
>> >> <mailto:[email protected]>> 
>> >> wrote: 
>> >>  
>> >>> The indexing dir is created: 
>> >>>  
>> >>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir, 
>> >>> type="directory", 
>> >>> action="create_on_execute", 
>> >>> owner=self.__params.metron_user, 
>> >>> group=self.__params.metron_group, 
>> >>> mode=0775, 
>> >>> ) 
>> >>>  
>> >>>  
>> >>>  
>> >>>  
>> >>> On April 12, 2017 at 07:49:16, Otto Fowler ([email protected] 
>> >>> <mailto:[email protected]>) 
>> >>> wrote: 
>> >>>  
>> >>>  
>> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the 
>> >> following 
>> >>> exception: 
>> >>>  
>> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied: 
>> >>> user=storm, access=WRITE, 
>> >>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. 
>> >>> check(FSPermissionChecker.java:319) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. 
>> >>> check(FSPermissionChecker.java:292) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. 
>> >>> checkPermission(FSPermissionChecker.java:213) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. 
>> >>> checkPermission(FSPermissionChecker.java:190) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. 
>> >>> checkPermission(FSDirectory.java:1827) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. 
>> >>> checkPermission(FSDirectory.java:1811) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess( 
>> >>> FSDirectory.java:1794) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs( 
>> >>> FSDirMkdirOp.java:71) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs( 
>> >>> FSNamesystem.java:4011) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. 
>> >>> mkdirs(NameNodeRpcServer.java:1102) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi 
>> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi 
>> >>> deTranslatorPB.java:630) 
>> >>> at 
>> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ 
>> >>> ClientNamenodeProtocol$2.callBlockingMethod( 
>> >> ClientNamenodeProtocolProtos. 
>> >>> java) 
>> >>> at 
>> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call( 
>> >>> ProtobufRpcEngine.java:640) 
>> >>>  
>> >>>  
>> >>> The HDFS directory is created as such: 
>> >>>  
>> >>> self.__params.HdfsResource(self.__params.hdfs_metron_ 
>> >>> apps_extensions_working, 
>> >>> type="directory", 
>> >>> action="create_on_execute", 
>> >>> owner=self.__params.metron_user, 
>> >>> mode=0775) 
>> >>>  
>> >>>  
>> >>> As the hdfs write handlers I am logging in as such: 
>> >>>  
>> >>> HdfsSecurityUtil.login(stormConfig, fsConf); 
>> >>> FileSystem fileSystem = FileSystem.get(fsConf); 
>> >>>  
>> >>> I am not sure what is different from the indexing hdfs writer setup 
>> >>> here, 
>> >>> but what I’m doing obviously is not working. 
>> >>>  
>> >>> Any ideas? 
>> >>>  
>> >>>  
>> >>> - the branch: 
>> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy 
>> >>> <https://github.com/ottobackwards/incubator-metron/tree/parser_deploy> 
>> >>>  
>> >>> I am not up to date with master. 
>> >>>  
>> >> 

Reply via email to