Processor example with dynamically fetched values for attributs

2018-03-26 Thread Dominique De Vito
Hi, I try to define a Processor such as for a given attribut: * the possible values are dynamically fetched from a REST WS (from time to time) * the UI offers to select one of these values from a drop-down list. Is there some processor implementing such a pattern? If so, then, I may take it as

Re: Store Hash data type in redis

2018-03-26 Thread Bryan Bende
If you setup the RedisDistributedMapCacheClient, you can then use the PutDistributedMapCache processor to store anything you want in Redis. You could use the HashContent processor and then store that in Redis. -Bryan On Fri, Mar 23, 2018 at 9:03 AM, Mike Thomsen wrote: > I don't think there are

Re: HDFS Processors and Native Compression Codecs

2018-03-26 Thread Bryan Bende
Shawn, You are right, the issue is that the JVM only allows a native library to be loaded once, and furthermore if you have native methods that need to be linked to the native library, the native code must be in the same class loader where the native library is loaded (at least from my testing).

Re: PutHDFS with mapr

2018-03-26 Thread Ravi Papisetti (rpapiset)
Hi Andre, I have tried with pointing puthdfs to lib class path with: /opt/mapr/lib,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common,/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib I have given this value for “Additional Classpath Resources” parameter of PutHDFS processor. Getting belo

Re: Processor example with dynamically fetched values for attributs

2018-03-26 Thread Bryan Bende
Hello, This pattern isn't really supported at the moment... You would not want to make calls to external services when opening the config window of a processor because then you risk blocking on an network call if the system is down or unresponsive, or just risking taking a really long time to loa

Unable to create HiveConnectionPool with kerberos.

2018-03-26 Thread mohit.jain
Hi, I am getting the following warning when I use HiveConnection pool with Kerberos : HiveConnectionPool[id=6e60258b-9e00-3bac-85ba-0dac8e22142f] Configuration does not have security enabled, Keytab and Principal will be ignored It also throws the following bulletin in my PutHiveQl proc

Re: Unable to create HiveConnectionPool with kerberos.

2018-03-26 Thread Juan Pablo Gardella
See https://issues.apache.org/jira/browse/NIFI-2575, the driver does not suppor that. I've put some workarounds in the ticket. On Mon, 26 Mar 2018 at 13:03 wrote: > Hi, > > > > I am getting the following warning when I use HiveConnection pool with > Kerberos : > > > > HiveConnectionPool[id=6e602

Re: Unable to create HiveConnectionPool with kerberos.

2018-03-26 Thread Juan Pablo Gardella
Sorry, the issue happens when a HA configuration is used. On Mon, 26 Mar 2018 at 13:03 Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > See https://issues.apache.org/jira/browse/NIFI-2575, the driver does not > suppor that. I've put some workarounds in the ticket. > > On Mon, 26 Mar 20

Re: Unable to create HiveConnectionPool with kerberos.

2018-03-26 Thread Pierre Villard
Mohit, I believe you need to change the JDBC url (even though you have the configuration files correctly set) to something like: jdbc:hive2://:/;principal= So it'd be something like: jdbc:hive2://localhost:1/default;principal=hive/my.fqdn.hive.ser...@example.com Pierre 2018-03-26 18:13 G

NiFi as a Web Services server

2018-03-26 Thread Boris Tyukin
I wonder how practical it is to use NiFi as a server to serve web service calls and return data over REST to consumers. It seems like a natural fit, given NiFi's scalability and redundancy and a host of data integration and transformation processors. I am thinking to create a quick pilot using Kaf

Re: AWS CloudWatch

2018-03-26 Thread Laurens Vets
Hi Andre, The "up to 12 hours" is from the Amazon documentation when you enable export from CloudWatch (NOT CloudTrail) to S3. I'm just trying to cover my bases and find a better/quicker way. I don't have anything specific set up right now... On 2018-03-25 14:06, Andre wrote: > Laurens, > >

Nifi processor stops picking up file from queue

2018-03-26 Thread af6140
Nifi processor(ReplaceText) stops processing file. I have a xml splitter to split into multiple flow files and then feed into the ReplaceText file, it always stops at the last split file and stops processing file from the upstream queue. stop/start, disable/enable, restart service etc did not hel

Re: Nifi processor stops picking up file from queue

2018-03-26 Thread af6140
This is nifi 1.2.0. -- Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Nifi processor stops picking up file from queue

2018-03-26 Thread Bryan Bende
Hello, Approximately how many flow files are generated by the split? -Bryan On Mon, Mar 26, 2018 at 4:26 PM, af6140 wrote: > This is nifi 1.2.0. > > > > -- > Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: NiFi as a Web Services server

2018-03-26 Thread Mike Thomsen
I think you would hit two big barriers in design: 1. NiFi just isn't designed to be an app server for additional service layer components a la Tomcat. 2. Synchronizing between the REST services and NiFi's highly asynchronous processing would be a logistical nightmare if your goal is to confine NiF

Re: NiFi Variables

2018-03-26 Thread scott
Daniel, That worked perfectly. Thank you for the help, and for creating this great tool. Scott On 03/25/2018 02:50 AM, Daniel Chaffelson wrote: Hi Scott, NiPyApi provides a python client for this purpose. There are calls to get/set variables in the canvas module. https://github.com/Chaffe

how to edit queue content

2018-03-26 Thread scott
Hi community, I've got a question about a feature I would find useful. I've been setting up a lot of new flows and testing various configurations, and I thought it would be really useful if I could edit the content of queues. For example, I can examine each file in the queue, then decide I wan

Re: how to edit queue content

2018-03-26 Thread Joe Witt
Scott Yep definitely something we've talked about [1]. We've not pursued it directly as of yet since it is indeed a queue and we're just letting you peak into it. We dont have facilities built in to really alter the queue in a particular position. Also, the complexity comes in when folks want to

RE: Unable to create HiveConnectionPool with kerberos.

2018-03-26 Thread mohit.jain
Hi, I have tried that URL but it gives me following error:- HiveConnectionPool[id=6e60258b-9e00-3bac-9590-543aec882280] Error getting Hive connection: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:

RE: Unable to create HiveConnectionPool with kerberos.

2018-03-26 Thread mohit.jain
When I try using my user prinicipal instead of hive it gives following error: SelectHiveQL[id=633d54ed-0162-1000--6fa47d56] org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$523/1312347477@1c34a7fa failed to process due to java.lang.IllegalArgumentException: Kerberos principal sho