Re: ConvertJSONToAvro floating point / double errors

I think I had the same or similar problem (with earlier version) and couldn’t get a direct Json->Avro to work with any Json numerical type. I ended up working around by having the Avro schema data types being String only and then doing a ConvertAvroSchema to move from the string only schema to t

RE: nifi processor to parse+update the current json on the fly

Hello thx Thad , joe , all For different answers. I understand how to know…. Philippe Best regards De : Thad Guidry [mailto:thadgui...@gmail.com] Envoyé : jeudi 7 avril 2016 17:32 À : users@nifi.apache.org Objet : Re: nifi processor to parse+update the current json on the fly Philippe, I would e

ConvertJSONToAvro floating point / double errors

In Nifi 0.6.0 with Java 1.7, the conversion of JSON data to Avro appears to reject the floating point values (both float and double). Using the ConvertJSONToAvro processor and the following examples causes errors. Sample Avro Schema: {"name": " sample_float_avro", "type": "record", "fields": [

Re: ExecuteSQL to elasticsearch

Madhu, To achieve what Arpit is saying, you can add a SplitJson processor after your ConvertAvroToJson processor, using $[*] as the JSONPath expression. That should split the array into individual flow files, then you would proceed as Arpit describes. Regards, Matt On Thu, Apr 7, 2016 at 2:18 PM

Re: ExecuteSQL to elasticsearch

Each flowfile should contain a single document that gets written to elastic search. So if each flow file had content like > {"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName": > "AvgDiskSecTransfer", "InstanceName": "C:", "MetricValue": > 9.60508652497083E-4} I suspect it

Re: ExecuteSQL to elasticsearch

Here is the sample JSON output from ConvertAvroToJson Processor: [{"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName": "AvgDiskSecTransfer", "InstanceName": "C:", "MetricValue": 9.60508652497083E-4}, {"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName": "Avg

Re: ExecuteSQL to elasticsearch

Can you provide a sample JSON output from your ConvertAvroToJson processor? It could help identify the location of any mapping/parser exceptions. Thanks, Matt On Thu, Apr 7, 2016 at 1:31 PM, Madhukar Thota wrote: > I am able to construct the dataflow with the following processors > > ExecuteSQL

Re: ExecuteSQL to elasticsearch

I am able to construct the dataflow with the following processors ExecuteSQL--> ConvertAvrotoJson --> Elasticsearch. The problem i seeing is elasticsearch unable to index the data because of the Mapping parser exceptions. 13:27:37 EDT ERROR fc43fc28-215c-469a-9908-73d04d98d4c2 PutElasticsearch[

Re: NiFi Supports for Kerberized Kafka Cluster

Yep, just watch the JIRA for PR Sent from my iPhone On Apr 7, 2016, at 12:35, indus well mailto:indusw...@gmail.com>> wrote: Thank you, Oleg. I'll be testing it out once it is available next week. Regards, Indus On Thu, Apr 7, 2016 at 10:48 AM, Oleg Zhurakousky mailto:ozhurakou...@hortonwor

Re: NiFi Supports for Kerberized Kafka Cluster

Thank you, Oleg. I'll be testing it out once it is available next week. Regards, Indus On Thu, Apr 7, 2016 at 10:48 AM, Oleg Zhurakousky < ozhurakou...@hortonworks.com> wrote: > Not at the moment unless you are using HDP distribution of Kafka broker > 0.8 which has Kerberos support while ASF ve

Re: NiFi Supports for Kerberized Kafka Cluster

Not at the moment unless you are using HDP distribution of Kafka broker 0.8 which has Kerberos support while ASF version of 0.8 Kafka doesn't. That said, we are working on new pair of Kafka processors that will rely on Kafka 0.9 API and broker where such support is provided. It is slated for 0.7

Re: nifi processor to parse+update the current json on the fly

Here is an example of json to json conversion using Groovy with JsonSlurper. http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html On Thu, Apr 7, 2016 at 11:31 AM, Thad Guidry wrote: > Philippe, > > I would encourage you to just use Groovy with JsonSlurper in the > Exec

NiFi Supports for Kerberized Kafka Cluster

Hello All: Does current version of NiFi support kerberized Kafka cluster? I am getting time-out errors trying to use PutKafka and GetKafka processors. Please advise. Thanks, Indus

Re: nifi processor to parse+update the current json on the fly

Philippe, I would encourage you to just use Groovy with JsonSlurper in the ExecuteScript processor. Its a blazing fast parser actually. http://groovy-lang.org/json.html http://docs.groovy-lang.org/latest/html/gapi/groovy/json/JsonSlurper.html Thad +ThadGuidry

Re: nifi processor to parse+update the current json on the fly

Philippe As far as I know here is the state of affairs for this: 1) You can use EvaluateJSONPath and ReplaceText in a combination for some cases but it is more awkward and difficult than it should be. 2) You can use the execute script processors to write a groovy, javascript, or other type of scr

nifi processor to parse+update the current json on the fly

Hello I have a DF with a processor owning a json document that need to be transformed before sent to an ElasticSearch processor I have something like this $.element.attributes[0].value --> not to be changed $.element.attributes[0].type--> not to be changed ... $.element.attributes[10].value ->

Re: Approaches to Array in Json with Nifi?

Hong, Koji, There is a ticket to upgrade this processor to a new version [1] (although the ticket is showing its age by listing 2.1.0). I had looked into changing it, but the structure of the library validation process and visibility of some of the core classes changed a fair amount and was break

Re: problem with putElasticsearch processor

I think Elasticsearch processor uses transport client not http. So you should use 9300(transport port) port not 9200(http port). On Wed, Apr 6, 2016 at 12:41 PM, wrote: > Hello > > My context nifi 0.6.0 on Ubuntu 14.0 > > my small use case is : > > ‘sending JSON notifications arriving on ht

RE: problem with putElasticsearch processor

Hello thx arpit It works nicely with port 9300 :) Thx philippe Hey Philippe Couple of things we can do to see what might be going on. 1. Check the port being used by put elastic search. Port 9200 is usually the default http port and 9300 is the default transport port. In put elastic search

ExecuteSQL to elasticsearch

Friends, I am exploring ExecuteSQL processor in nifi and my goal to get sql data ingested in Elasticsearch. Can someone share or guide what's the flow looks like? Thanks in Advance.

Re: Approaches to Array in Json with Nifi?

Hi all, I got interested in EvaluateJsonPath, too. So I tried to figure out how to do that, and found there're few obstacles now. Input Json = {"company":"xyz", "rate":[0.02, 0.03, 0.04]} Then, applying following JsonPaths return: $.rate.min() => 0.2 $.rate.length() => 3 $.rate.avg() => 0.03 T