Makes sense, however, would it make sense to unpack to local file system, which would then avoid further HDFS ops?
> On 12 Apr 2017, at 13:24, Otto Fowler <[email protected]> wrote: > > "The parsers are packaged as ‘bundles’ ( our version of NIFI NARS ) and are > deployed > if required. So the writes for HDFS are the unpacking the bundles ( with the > unpacked jars being loaded into a classloader ). > > If the unbundled is the same as the bundle, there is no op. In this case, it > is first run. > > So this is the parser bolt using the HDFS backed extension loader.” > > > This is part of the parser side loading, also feeds into the stellar load > from hdfs stuff. > > * metron has extensions > * extensions are packaged with their configurations and their ‘bundle’ ( the > nar part ) > * extension bundles are deployed to hdfs *lib directory > * parts of the system that use extensions discover and load them, with class > loaders setup for the bundle > * part of that, the bundles are unpackaged into a ‘working’ area if required > ( so if you drop a new version into the lib, the next app will unpack the new > version ) > * thus, the bundle system needs to be able to write to the working areas in > hdfs > * ??? > * profit > > > On April 12, 2017 at 08:11:28, Simon Elliston Ball > ([email protected] <mailto:[email protected]>) wrote: > >> I’m curious: Otto, what’s your use case for writing to HDFS from a parsers? >> >> Simon >> >> > On 12 Apr 2017, at 13:04, Justin Leet <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Chown it to metron:hadoop and it'll work. Storm is in the hadoop group and >> > with 775 will be able to write. >> > >> > On Wed, Apr 12, 2017 at 7:56 AM, David Lyle <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> >> It's curious to me that you're writing directly from parsing, but I >> >> suspect >> >> that your parsing topology is running as the storm user and it can't >> >> write >> >> to those directories. >> >> >> >> -D... >> >> >> >> On Wed, Apr 12, 2017 at 7:51 AM, Otto Fowler <[email protected] >> >> <mailto:[email protected]>> >> >> wrote: >> >> >> >>> The indexing dir is created: >> >>> >> >>> self.__params.HdfsResource(self.__params.metron_apps_indexed_hdfs_dir, >> >>> type="directory", >> >>> action="create_on_execute", >> >>> owner=self.__params.metron_user, >> >>> group=self.__params.metron_group, >> >>> mode=0775, >> >>> ) >> >>> >> >>> >> >>> >> >>> >> >>> On April 12, 2017 at 07:49:16, Otto Fowler ([email protected] >> >>> <mailto:[email protected]>) >> >>> wrote: >> >>> >> >>> >> >>> I am trying to write to HDFS from ParserBolt, but I’m getting the >> >> following >> >>> exception: >> >>> >> >>> Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied: >> >>> user=storm, access=WRITE, >> >>> inode="/apps/metron/extension_working/framework":metron:hdfs:drwxrwxr-x >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. >> >>> check(FSPermissionChecker.java:319) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. >> >>> check(FSPermissionChecker.java:292) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. >> >>> checkPermission(FSPermissionChecker.java:213) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker. >> >>> checkPermission(FSPermissionChecker.java:190) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. >> >>> checkPermission(FSDirectory.java:1827) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory. >> >>> checkPermission(FSDirectory.java:1811) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess( >> >>> FSDirectory.java:1794) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs( >> >>> FSDirMkdirOp.java:71) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs( >> >>> FSNamesystem.java:4011) >> >>> at >> >>> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer. >> >>> mkdirs(NameNodeRpcServer.java:1102) >> >>> at >> >>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSi >> >>> deTranslatorPB.mkdirs(ClientNamenodeProtocolServerSi >> >>> deTranslatorPB.java:630) >> >>> at >> >>> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ >> >>> ClientNamenodeProtocol$2.callBlockingMethod( >> >> ClientNamenodeProtocolProtos. >> >>> java) >> >>> at >> >>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call( >> >>> ProtobufRpcEngine.java:640) >> >>> >> >>> >> >>> The HDFS directory is created as such: >> >>> >> >>> self.__params.HdfsResource(self.__params.hdfs_metron_ >> >>> apps_extensions_working, >> >>> type="directory", >> >>> action="create_on_execute", >> >>> owner=self.__params.metron_user, >> >>> mode=0775) >> >>> >> >>> >> >>> As the hdfs write handlers I am logging in as such: >> >>> >> >>> HdfsSecurityUtil.login(stormConfig, fsConf); >> >>> FileSystem fileSystem = FileSystem.get(fsConf); >> >>> >> >>> I am not sure what is different from the indexing hdfs writer setup >> >>> here, >> >>> but what I’m doing obviously is not working. >> >>> >> >>> Any ideas? >> >>> >> >>> >> >>> - the branch: >> >>> https://github.com/ottobackwards/incubator-metron/tree/parser_deploy >> >>> <https://github.com/ottobackwards/incubator-metron/tree/parser_deploy> >> >>> >> >>> I am not up to date with master. >> >>> >> >>
