Re: Processors on the fly for many sensor devices

Hi Andrew, We tested your 0.0.6 binary and it seemed to do what we want. Unfortunately it is not compatible with the current Nifi version. Do you have a binary available that would work with nifi 1.0.0 ? Thx. On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis wrote: > Davy, > The processor I hav

ReplaceText processor - $ symbol escaping done incorrectly

Hi, I'm using Replace text processor[1] to build a json in my flow file. Flow file has following attribute: *messageId: 001201d20dd3$7bcdc870$73695950$@xx.yy.co * Replace text processor "Replacement Value": *{* *"messageId" : "${messageId}"* *}* And all other properties are

RE: PutHiveQL and Hive Connection Pool with HDInsight

Tried couple of more connection options, but always got an error while setting up Hive Connection Pool. What’s strange is DBCP Connection Pool works fine with the same Hive JDBC connection string. Now I am writing a custom “PutSQL” like processor that uses standard DBCP Controller service and a

Best practice for querying table mid-flow

Hi All, I have a Nifi data flow that receives flowfiles each containing a JSON object. As part of the transformation of each flowfile I want to query a hive table using a property in the flowfile's JSON content to retrieve additional information that I then want to inject into the flowfile. The up

Re: Regarding ConsumeIMAP Processor.

Prabhu, There's a PR on github with a possible solution to this issue: Can you try running that code and let us know if it solves your problem? Also, be mindful of compiling NiFi using 'mvn -DskipTests=true clean install' Best regards On Wed, Sep 21, 2016 at 2:25 PM, prabhu Mahendran wrote:

Re: PutHiveQL and Hive Connection Pool with HDInsight

Manish, Sorry to hear you're having issues connecting. I should mention though that the fact that DBCPConnectionPool works with the Hive JDBC connection string doesn't imply that the processors that use DBCPConnectionPool will work for Hive. Notably there are three differences: 1) The Hive JDBC

Re: Best practice for querying table mid-flow

Hi Mike, This seems like the correct approach when using out-of-the-box processors. You could potentially create a custom processor that performed all three of those steps into one... take JSON as input and extract a value, query Hive, merge results and write to the flow file. Normally I would th

RE: PutHiveQL and Hive Connection Pool with HDInsight

Hi Matt, Thanks for the detailed response. Really appreciate it. Few comments – · “if loading a DBCPConnectionPool for a HiveQL processor happens to work” – it does not allow to pick a DBCPConnectionPool. Only Hive pool is possible. · “The non-HiveQL processors make calls to

Re: PutHiveQL and Hive Connection Pool with HDInsight

Manish, In NiFi 0.7, the Hive processors used Hive 2.0. Due to consistency and compatibility issues across the Hadoop-related processors/NARs, we downgraded Hive to 1.2.1 for NiFi 1.0.0 as it was the baseline against the Hadoop version (2.6.x). Is it possible for you to upgrade or try NiFi 1.0.0 a

Re: provenance

Brett, The default provenance store, PersistentProvenanceRepository, does require I/O in proportion to flowfile events. Flowfiles with many attributes, especially large attributes, are a frequent contributor to provenance overload because attribute state is tracked in provenance events. But this

RE: PutHiveQL Multiple Ordered Statements

Matt, I wanted to circle back on this conversation. What are your thoughts on a new property for this processor that would support this and do the split internally? I feel like a lot of people (or at least me) are coming from a world where they have HQL files with many statements back to back a

Re: ReplaceText processor - $ symbol escaping done incorrectly

In your ReplaceText processor, is the Replacement Strategy property set to the default "Regex Replace"? You might try "Always Replace" or changing the Search Value to ".*" (without quotes). I suspect it is escaping your $ as part of processing the regex. Thanks, James On Fri, Sep 30, 2016 at

Re: Processors on the fly for many sensor devices

Hi Davy, Sorry for the slow response, I am traveling today. However, I should be able to get a build of it for 1.0.0 out later today or tomorrow. Thanks, Andrew On Fri, Sep 30, 2016 at 6:54 PM, Davy De Waele wrote: > Hi Andrew, > > We tested your 0.0.6 binary and it seemed to do what we want. >