RE: How to provide hbase-site.xml to Stellar Processor Java API
Hi All, FYI I am able to resolve the issue for my springboot application. I added a plugin as below to my pom.xml file under // and passing files to be loaded while starting the application as “-Dloader.path=/etc/hbase/conf/,/etc/hadoop/conf/” org.springframework.boot spring-boot-maven-plugin ZIP Thanks, Anil. From: Anil Donthireddy Sent: Friday, January 25, 2019 5:32 PM To: 'Otto Fowler' ; user@metron.apache.org; d...@metron.apache.org; Mohan Venkateshaiah Cc: Maxim Dashenko ; Satish Abburi ; James Sirota ; Christopher Berry Subject: RE: How to provide hbase-site.xml to Stellar Processor Java API Hi Mohan DV, I appreciate your time for sharing the much useful information. As per my understanding there seems to be no scope to pass my own build Hbase configuration object to execute Stellar queries in my extended API. I may need to re-write lot of things in my extended API in the way Stellar processor works to override the Hbase configuration. @Otto Fowler: I will raise a Jira case with the usecase and the issue. Thanks, Anil. From: Otto Fowler [mailto:ottobackwa...@gmail.com] Sent: Thursday, January 24, 2019 11:15 PM To: Anil Donthireddy mailto:anil.donthire...@sstech.us>>; user@metron.apache.org<mailto:user@metron.apache.org>; d...@metron.apache.org<mailto:d...@metron.apache.org>; Mohan Venkateshaiah mailto:mvenkatesha...@hortonworks.com>> Cc: Maxim Dashenko mailto:maxim.dashe...@sstech.us>>; Satish Abburi mailto:satish.abb...@sstech.us>>; James Sirota mailto:jsir...@hortonworks.com>>; Christopher Berry mailto:christopher.be...@sstech.us>> Subject: Re: How to provide hbase-site.xml to Stellar Processor Java API But still file a jira ;) On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (mvenkatesha...@hortonworks.com<mailto:mvenkatesha...@hortonworks.com>) wrote: Hi Anil, I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config. What I had done was in the global config, I set a property called hbase.provider.impl, it's the fully qualified class name for a class that implements the TableProvider interface which has one method: public HTableInterface getTable(Configuration config, String tableName) throws IOException if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work Thanks Mohan DV On 1/24/19, 8:56 PM, "Otto Fowler" mailto:ottobackwa...@gmail.com>> wrote: Hi Anil, Can you create a jira on this with these details and a general overview of your use case? It looks like the HbaseConfiguration we use in the HTableConnector is done using the create() method, which creates from resources. I think we would need to do some work to support the external file. On January 24, 2019 at 10:14:46, Anil Donthireddy ( anil.donthire...@sstech.us<mailto:anil.donthire...@sstech.us>) wrote: Hi, I have written a java application which uses Stellar processor and execute the stellar expressions. The issue I am facing is I am unable to connect Hbase unless I place hbase-site.xml in src/main/resources/ folder of the code. As it is not the proper way of packaging the hbase-site.xml with Jar, I would like to understand how the hbase-site.xml is being set to classpath while starting profiler topology. The ways I tried are 1) Setting the classpath to hbase conf folder using command “java -cp $CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar” 2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop classpath /usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf One more step I would like to try to fix the issue is to set property “ *zookeeper.znode.parent*” for configuration object while instantiating HbaseConnector. But it is in the scope of metron code to try this fix. I would like to know if any one able to provide hbase-site.xml to any Java Application or anyone able to extend Metron Stellar Processor and execute profile definitions successfully. Please provide any inputs to resolve the issue. Thanking you. Thanks, Anil.
RE: How to provide hbase-site.xml to Stellar Processor Java API
Please link it to https://issues.apache.org/jira/browse/METRON-1409. This is a jira around hosting stellar that I created a while ago. On January 25, 2019 at 07:02:30, Anil Donthireddy ( anil.donthire...@sstech.us) wrote: Hi Mohan DV, I appreciate your time for sharing the much useful information. As per my understanding there seems to be no scope to pass my own build Hbase configuration object to execute Stellar queries in my extended API. I may need to re-write lot of things in my extended API in the way Stellar processor works to override the Hbase configuration. @Otto Fowler: I will raise a Jira case with the usecase and the issue. Thanks, Anil. *From:* Otto Fowler [mailto:ottobackwa...@gmail.com] *Sent:* Thursday, January 24, 2019 11:15 PM *To:* Anil Donthireddy ; user@metron.apache.org; d...@metron.apache.org; Mohan Venkateshaiah *Cc:* Maxim Dashenko ; Satish Abburi < satish.abb...@sstech.us>; James Sirota ; Christopher Berry *Subject:* Re: How to provide hbase-site.xml to Stellar Processor Java API But still file a jira ;) On January 24, 2019 at 12:19:34, Mohan Venkateshaiah ( mvenkatesha...@hortonworks.com) wrote: Hi Anil, I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config. What I had done was in the global config, I set a property called hbase.provider.impl, it's the fully qualified class name for a class that implements the TableProvider interface which has one method: public HTableInterface getTable(Configuration config, String tableName) throws IOException if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work Thanks Mohan DV On 1/24/19, 8:56 PM, "Otto Fowler" wrote: Hi Anil, Can you create a jira on this with these details and a general overview of your use case? It looks like the HbaseConfiguration we use in the HTableConnector is done using the create() method, which creates from resources. I think we would need to do some work to support the external file. On January 24, 2019 at 10:14:46, Anil Donthireddy ( anil.donthire...@sstech.us) wrote: Hi, I have written a java application which uses Stellar processor and execute the stellar expressions. The issue I am facing is I am unable to connect Hbase unless I place hbase-site.xml in src/main/resources/ folder of the code. As it is not the proper way of packaging the hbase-site.xml with Jar, I would like to understand how the hbase-site.xml is being set to classpath while starting profiler topology. The ways I tried are 1) Setting the classpath to hbase conf folder using command “java -cp $CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar” 2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop classpath /usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf One more step I would like to try to fix the issue is to set property “ *zookeeper.znode.parent*” for configuration object while instantiating HbaseConnector. But it is in the scope of metron code to try this fix. I would like to know if any one able to provide hbase-site.xml to any Java Application or anyone able to extend Metron Stellar Processor and execute profile definitions successfully. Please provide any inputs to resolve the issue. Thanking you. Thanks, Anil.
RE: How to provide hbase-site.xml to Stellar Processor Java API
Hi Mohan DV, I appreciate your time for sharing the much useful information. As per my understanding there seems to be no scope to pass my own build Hbase configuration object to execute Stellar queries in my extended API. I may need to re-write lot of things in my extended API in the way Stellar processor works to override the Hbase configuration. @Otto Fowler: I will raise a Jira case with the usecase and the issue. Thanks, Anil. From: Otto Fowler [mailto:ottobackwa...@gmail.com] Sent: Thursday, January 24, 2019 11:15 PM To: Anil Donthireddy ; user@metron.apache.org; d...@metron.apache.org; Mohan Venkateshaiah Cc: Maxim Dashenko ; Satish Abburi ; James Sirota ; Christopher Berry Subject: Re: How to provide hbase-site.xml to Stellar Processor Java API But still file a jira ;) On January 24, 2019 at 12:19:34, Mohan Venkateshaiah (mvenkatesha...@hortonworks.com<mailto:mvenkatesha...@hortonworks.com>) wrote: Hi Anil, I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config. What I had done was in the global config, I set a property called hbase.provider.impl, it's the fully qualified class name for a class that implements the TableProvider interface which has one method: public HTableInterface getTable(Configuration config, String tableName) throws IOException if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work Thanks Mohan DV On 1/24/19, 8:56 PM, "Otto Fowler" mailto:ottobackwa...@gmail.com>> wrote: Hi Anil, Can you create a jira on this with these details and a general overview of your use case? It looks like the HbaseConfiguration we use in the HTableConnector is done using the create() method, which creates from resources. I think we would need to do some work to support the external file. On January 24, 2019 at 10:14:46, Anil Donthireddy ( anil.donthire...@sstech.us<mailto:anil.donthire...@sstech.us>) wrote: Hi, I have written a java application which uses Stellar processor and execute the stellar expressions. The issue I am facing is I am unable to connect Hbase unless I place hbase-site.xml in src/main/resources/ folder of the code. As it is not the proper way of packaging the hbase-site.xml with Jar, I would like to understand how the hbase-site.xml is being set to classpath while starting profiler topology. The ways I tried are 1) Setting the classpath to hbase conf folder using command “java -cp $CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar” 2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop classpath /usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf One more step I would like to try to fix the issue is to set property “ *zookeeper.znode.parent*” for configuration object while instantiating HbaseConnector. But it is in the scope of metron code to try this fix. I would like to know if any one able to provide hbase-site.xml to any Java Application or anyone able to extend Metron Stellar Processor and execute profile definitions successfully. Please provide any inputs to resolve the issue. Thanking you. Thanks, Anil.
Re: How to provide hbase-site.xml to Stellar Processor Java API
But still file a jira ;) On January 24, 2019 at 12:19:34, Mohan Venkateshaiah ( mvenkatesha...@hortonworks.com) wrote: Hi Anil, I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config. What I had done was in the global config, I set a property called hbase.provider.impl, it's the fully qualified class name for a class that implements the TableProvider interface which has one method: public HTableInterface getTable(Configuration config, String tableName) throws IOException if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work Thanks Mohan DV On 1/24/19, 8:56 PM, "Otto Fowler" wrote: Hi Anil, Can you create a jira on this with these details and a general overview of your use case? It looks like the HbaseConfiguration we use in the HTableConnector is done using the create() method, which creates from resources. I think we would need to do some work to support the external file. On January 24, 2019 at 10:14:46, Anil Donthireddy ( anil.donthire...@sstech.us) wrote: Hi, I have written a java application which uses Stellar processor and execute the stellar expressions. The issue I am facing is I am unable to connect Hbase unless I place hbase-site.xml in src/main/resources/ folder of the code. As it is not the proper way of packaging the hbase-site.xml with Jar, I would like to understand how the hbase-site.xml is being set to classpath while starting profiler topology. The ways I tried are 1) Setting the classpath to hbase conf folder using command “java -cp $CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar” 2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop classpath /usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf One more step I would like to try to fix the issue is to set property “ *zookeeper.znode.parent*” for configuration object while instantiating HbaseConnector. But it is in the scope of metron code to try this fix. I would like to know if any one able to provide hbase-site.xml to any Java Application or anyone able to extend Metron Stellar Processor and execute profile definitions successfully. Please provide any inputs to resolve the issue. Thanking you. Thanks, Anil.
Re: How to provide hbase-site.xml to Stellar Processor Java API
Hi Anil, I had done similar to this in the past . In the stellar to get the hbase configuration we call HBaseConfiguration.create() , in that call hbase adds the hbase-site and core-site as resources to the config we probably SHOULD let people specify a base config. What I had done was in the global config, I set a property called hbase.provider.impl, it's the fully qualified class name for a class that implements the TableProvider interface which has one method: public HTableInterface getTable(Configuration config, String tableName) throws IOException if you implement your own where you ignore the config argument and resolve the hbase table with your own injected config that will work Thanks Mohan DV On 1/24/19, 8:56 PM, "Otto Fowler" wrote: Hi Anil, Can you create a jira on this with these details and a general overview of your use case? It looks like the HbaseConfiguration we use in the HTableConnector is done using the create() method, which creates from resources. I think we would need to do some work to support the external file. On January 24, 2019 at 10:14:46, Anil Donthireddy ( anil.donthire...@sstech.us) wrote: Hi, I have written a java application which uses Stellar processor and execute the stellar expressions. The issue I am facing is I am unable to connect Hbase unless I place hbase-site.xml in src/main/resources/ folder of the code. As it is not the proper way of packaging the hbase-site.xml with Jar, I would like to understand how the hbase-site.xml is being set to classpath while starting profiler topology. The ways I tried are 1) Setting the classpath to hbase conf folder using command “java -cp $CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar” 2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop classpath /usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf One more step I would like to try to fix the issue is to set property “ *zookeeper.znode.parent*” for configuration object while instantiating HbaseConnector. But it is in the scope of metron code to try this fix. I would like to know if any one able to provide hbase-site.xml to any Java Application or anyone able to extend Metron Stellar Processor and execute profile definitions successfully. Please provide any inputs to resolve the issue. Thanking you. Thanks, Anil.
Re: How to provide hbase-site.xml to Stellar Processor Java API
Hi Anil, Can you create a jira on this with these details and a general overview of your use case? It looks like the HbaseConfiguration we use in the HTableConnector is done using the create() method, which creates from resources. I think we would need to do some work to support the external file. On January 24, 2019 at 10:14:46, Anil Donthireddy ( anil.donthire...@sstech.us) wrote: Hi, I have written a java application which uses Stellar processor and execute the stellar expressions. The issue I am facing is I am unable to connect Hbase unless I place hbase-site.xml in src/main/resources/ folder of the code. As it is not the proper way of packaging the hbase-site.xml with Jar, I would like to understand how the hbase-site.xml is being set to classpath while starting profiler topology. The ways I tried are 1) Setting the classpath to hbase conf folder using command “java -cp $CLASSPATH:/etc/hbase/conf:/etc/Hadoop/conf –jar myJar.jar” 2) Adding Hbase conf folder to HADOOP_CLASSPATH. Below is the Hadoop classpath /usr/hdp/2.6.1.0-129/hadoop/conf:/usr/hdp/2.6.1.0-129/hadoop/lib/*:/usr/hdp/2.6.1.0-129/hadoop/.//*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/./:/usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/*:/usr/hdp/2.6.1.0-129/hadoop-hdfs/.//*:/usr/hdp/2.6.1.0-129/hadoop-yarn/lib/*:/usr/hdp/2.6.1.0-129/hadoop-yarn/.//*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/lib/*:/usr/hdp/2.6.1.0-129/hadoop-mapreduce/.//*::mysql-connector-java.jar:postgresql-jdbc2ee.jar:postgresql-jdbc2.jar:postgresql-jdbc3.jar:postgresql-jdbc.jar:/etc/hbase/conf/:/usr/hdp/2.6.1.0-129/tez/*:/usr/hdp/2.6.1.0-129/tez/lib/*:/usr/hdp/2.6.1.0-129/tez/conf One more step I would like to try to fix the issue is to set property “ *zookeeper.znode.parent*” for configuration object while instantiating HbaseConnector. But it is in the scope of metron code to try this fix. I would like to know if any one able to provide hbase-site.xml to any Java Application or anyone able to extend Metron Stellar Processor and execute profile definitions successfully. Please provide any inputs to resolve the issue. Thanking you. Thanks, Anil.
