from:"l vic"

Re: jolt array name to attribute?

2020-01-16 Thread l vic

Hi Matt,
Is it possible with modify-default-beta? The problem with "shift" solution
- it doesn't work if i have several arrays within my struct:
{
 "HomeAddress": [
{
 "Street": "Washington St",
 "ZIP": "03420",
 "unit": "1"
}
],
"BusinessAddress": [
{
 "Street": "PO Box 22",
 "ZIP": "03420",
 "unit": "none"
}
]
}

Result:
{
  "type" : [ "HomeAddress", "BusinessAddress" ],
  "Street" : [ "Washington St", "PO Box 22" ],
  "ZIP" : [ "03420", "03420" ],
  "unit" : [ "1", "none" ]
}

Desired result:
{
 "HomeAddress": [
{
 "Street": "Washington St",
 "ZIP": "03420",
 "unit": "1",
"type": "HomeAddress"
}
],
"BusinessAddress": [
{
 "Street": "PO Box 22",
 "ZIP": "03420",
 "unit": "none",
"type": "BusinessAddress"
}
]
}

On Wed, Jan 15, 2020 at 3:29 PM Matt Burgess  wrote:

> Try the following chain spec:
>
> [
>   {
> "operation": "shift",
> "spec": {
>   "*": {
> "$": "type",
> "*": {
>   "*": "&"
> }
>   }
> }
>   }
> ]
>
> This will "flatten" any key with an array value into a "type" field
> with the name of the array key, and all its entries as top-level
> fields.
>
> Regards,
> Matt
>
>
> On Wed, Jan 15, 2020 at 2:52 PM l vic  wrote:
> >
> > i have input json such as the following example:
> > {
> >  "Address": [
> > {
> >  "Street": "Washington St",
> >  "ZIP": "03420",
> >  "unit": "1"
> > }
> > ]
> > }
> > How can i use Jolt transform to add array name as attribute value to
> > array members as following result:
> > {
> > "type": "Address",
> > "Street": "Washington St",
> >  "ZIP": "03420",
> >  "unit": "1"
> > }
> > Thank you,
>

jolt array name to attribute?

2020-01-15 Thread l vic

i have input json such as the following example:
{
 "Address": [
{
 "Street": "Washington St",
 "ZIP": "03420",
 "unit": "1"
}
]
}
How can i use Jolt transform to add array name as attribute value to
array members as following result:
{
*"type": "Address"*,
"Street": "Washington St",
 "ZIP": "03420",
 "unit": "1"
}
Thank you,

adding original attribute values to InvokeHttp response?

2019-12-26 Thread l vic

How can I add original attribute values into InvokeHttp response context
(Json)?
Thank you,

ReplaceText removes second '\' character?

2019-11-14 Thread l vic

I have ReplaceText 1.7.0 performing substitution with windows registry path
string value:
windows_path: "

"C:\\Users\\Administrator\\AppData\\Local\\Microsoft\\OneDrive\\19.070.0410.0007\\ck\\Controls\\Styles\\Flat\\qtquickextrasflatplugin.dll"

SearchValue: (?s)(^.*$)
ReplacementValue: upsert into mytable (fv) values ('${windows_path}')

The problem: after substitution ReplaceText removes '\' from "\\" string,
here's how the output queue content looks like:
upsert into mytable (fv) values
("C:\Users\Administrator\AppData\Local\Microsoft\OneDrive\19.070.0410.0007\ck\Controls\Styles\Flat\qtquickextrasflatplugin.dll")

This is causing the problems downstream... Is this a bug in ReplaceText?
Is there a way i can insert append another "\\" to transform  windows_path
to
"C:UsersAdministratorAppDataLocal\\MicrosoftOneDrive19.070.0410.0007ckControlsStylesFlatqtquickextrasflatplugin.dll"
as workaround?

problem with setting of Time Duration property from group variable

2019-09-16 Thread l vic

I am trying to set Time Duration property in ControlRate processor from
processor group variable.
I have INTERVAL variable set to 20 and I tried to set Time Duration as
${INTERVAL} mins
${INTERVAL:toNumber() mins}
I each case i have error like the following:
Time duration validated against '{INTERVAL mins}' is invalid because must
be of format   where duration is non-negative integer
and TimeUnit is supported Time Unit such as mins
Is there a solution to this problem?
Thank you

merging flowfile content and values?

2019-05-01 Thread l vic

Hello,

I have flow file with “root attributes” such as
id=‘9b892935-c0e7-42ba-958b-191138c71b39’, name=‘appliance1’ and json array
of user account info as content: [{“username”: “one”,
“create_date":1550500579036,
“role”: “tech lead”}, {“username”: “two”,  “create_date":1550500579100,
“role”: “user”}]

My question: how can i transform it into json object that uses both
attribute values as “root” values and content at “internal object”:

{ "id": "9b-892935-c0e7-42ba-958b-191138c71b39",

"name": "appliance1",

"account-info":  [{"username": "one",  "create_date\":1550500579036,
“role": "tech lead"}, {"username": "two",  "create_date\":1550500579100,
“role": "user"}]

}

Thank you,

-V

ISOChronology error in InvokeHTTP?

2019-04-30 Thread l vic

Have this error in ver. 1.5 of InvokeHTTP (see below), any idea how to fix
it?


WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunProcessorTask
Administratively Yielding
InvokeHTTP[id=2691affc-0034-3c06-3a1f-5a38cb1fc455] due to uncaught
Exception: java.lang.NoClassDefFoundError: Could not initialize class
org.joda.time.chrono.ISOChronology

2019-04-30 12:00:00,144 WARN [Timer-Driven Process Thread-8]
o.a.n.c.t.ContinuallyRunProcessorTask

java.lang.NoClassDefFoundError: Could not initialize class
org.joda.time.chrono.ISOChronology

at org.joda.time.DateTimeUtils.getChronology(DateTimeUtils.java:266)

at
org.joda.time.format.DateTimeFormatter.selectChronology(DateTimeFormatter.java:968)

at
org.joda.time.format.DateTimeFormatter.printTo(DateTimeFormatter.java:672)

at
org.joda.time.format.DateTimeFormatter.printTo(DateTimeFormatter.java:560)

at
org.joda.time.format.DateTimeFormatter.print(DateTimeFormatter.java:644)

at
org.apache.nifi.processors.standard.InvokeHTTP.setHeaderProperties(InvokeHTTP.java:938)

at
org.apache.nifi.processors.standard.InvokeHTTP.configureRequest(InvokeHTTP.java:905)

at
org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:698)

at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)

at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)

at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)

at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)

at
org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:161)

at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Re: PutKafka use with large quantity of data?

2019-04-04 Thread l vic

Actually, it's not Kafka topic but Nifi queue between "generateFlow" and
"PutKafka" gets overflown

On Thu, Apr 4, 2019 at 10:58 AM Joe Witt  wrote:

> Can you share screenshots, logs, and a more detailed description of what
> you're doing, observing with nifi and the system and what you expect it to
> be doing.
>
> Thanks
>
> On Thu, Apr 4, 2019 at 10:56 AM l vic  wrote:
>
>> No, actually what happens is - NiFi stops responding ( if I use it
>> without rate contol)
>>
>>
>> On Thu, Apr 4, 2019 at 10:42 AM Joe Witt  wrote:
>>
>>> Hello
>>>
>>> There isn't really a feedback mechanism based on load on the Kafka
>>> topic.  When you say overrunning the topic do you mean that you don't want
>>> there to be a large lag between consumers and their current offset and if
>>> that grows you want NiFi to slow down?  I dont believe there is anything
>>> inherent to the kafka producer protocol that would inform of us this.  We
>>> could periodically poll for this information and optional back-off.
>>>
>>> Is this what you have in mind?
>>>
>>> Thanks
>>>
>>> On Thu, Apr 4, 2019 at 10:34 AM l vic  wrote:
>>>
>>>> I have to ingest large (200,000 messages) data set into Kafka topic as
>>>> quickly as possible without overrunning topic... Right now I just use rate
>>>> limiter to do it but can be there some better "adaptive" way to do it?
>>>> Thank you...
>>>> -V
>>>>
>>>

Re: PutKafka use with large quantity of data?

2019-04-04 Thread l vic

What's this particular processing group does: writes large dataset to Kafka
topic, one consumer reads from topic and saves data to Hbase/PQS table,
another consumer writes to ES index

On Thu, Apr 4, 2019 at 10:58 AM Joe Witt  wrote:

> Can you share screenshots, logs, and a more detailed description of what
> you're doing, observing with nifi and the system and what you expect it to
> be doing.
>
> Thanks
>
> On Thu, Apr 4, 2019 at 10:56 AM l vic  wrote:
>
>> No, actually what happens is - NiFi stops responding ( if I use it
>> without rate contol)
>>
>>
>> On Thu, Apr 4, 2019 at 10:42 AM Joe Witt  wrote:
>>
>>> Hello
>>>
>>> There isn't really a feedback mechanism based on load on the Kafka
>>> topic.  When you say overrunning the topic do you mean that you don't want
>>> there to be a large lag between consumers and their current offset and if
>>> that grows you want NiFi to slow down?  I dont believe there is anything
>>> inherent to the kafka producer protocol that would inform of us this.  We
>>> could periodically poll for this information and optional back-off.
>>>
>>> Is this what you have in mind?
>>>
>>> Thanks
>>>
>>> On Thu, Apr 4, 2019 at 10:34 AM l vic  wrote:
>>>
>>>> I have to ingest large (200,000 messages) data set into Kafka topic as
>>>> quickly as possible without overrunning topic... Right now I just use rate
>>>> limiter to do it but can be there some better "adaptive" way to do it?
>>>> Thank you...
>>>> -V
>>>>
>>>

Re: PutKafka use with large quantity of data?

2019-04-04 Thread l vic

No, actually what happens is - NiFi stops responding ( if I use it without
rate contol)


On Thu, Apr 4, 2019 at 10:42 AM Joe Witt  wrote:

> Hello
>
> There isn't really a feedback mechanism based on load on the Kafka topic.
> When you say overrunning the topic do you mean that you don't want there to
> be a large lag between consumers and their current offset and if that grows
> you want NiFi to slow down?  I dont believe there is anything inherent to
> the kafka producer protocol that would inform of us this.  We could
> periodically poll for this information and optional back-off.
>
> Is this what you have in mind?
>
> Thanks
>
> On Thu, Apr 4, 2019 at 10:34 AM l vic  wrote:
>
>> I have to ingest large (200,000 messages) data set into Kafka topic as
>> quickly as possible without overrunning topic... Right now I just use rate
>> limiter to do it but can be there some better "adaptive" way to do it?
>> Thank you...
>> -V
>>
>

PutKafka use with large quantity of data?

2019-04-04 Thread l vic

I have to ingest large (200,000 messages) data set into Kafka topic as
quickly as possible without overrunning topic... Right now I just use rate
limiter to do it but can be there some better "adaptive" way to do it?
Thank you...
-V

Re: sensitive variable values ?

2019-03-21 Thread l vic

I don't see how to do it for service from NiFi REST api... Any suggestions?
Thank you


>
>

sensitive variable values ?

2019-03-21 Thread l vic

Is there a mechanism in Nifi to use sensitive variable values? For example:
would it be possible to store/set password value
in StandardRestrictedSSLContextService without exposing it? I am looking
for the way to set password in StandardRestrictedSSLContextService from the
automated deployment script
Thank you,
-V

Re: jolt transform spec ?

2019-03-06 Thread l vic

Hi Koji,
Turn out even simpler than that: EvaluateJsonPath to extract it MSG part
into attribute and AttibuteToJson produces normal Json.
Thank you,
Victor

On Wed, Mar 6, 2019 at 8:55 PM Koji Kawamura  wrote:

> Hello,
>
> I haven't tested myself, but using EvaluateJsonPath and ReplaceText
> with 'unescapeJson' EL function may be an alternative approach instead
> of Jolt.
>
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#unescapejson
>
> Idea is, use EvaluateJsonPath to extract the MSG part into a FlowFile
> attribute, e.g. 'message'.
> Then use ReplaceText with 'Replacement Value' as
> "${message:unescapeJson()}" to update FlowFile content with the normal
> JSON representation.
>
> Thanks,
> Koji
>
> On Thu, Mar 7, 2019 at 5:18 AM l vic  wrote:
> >
> > I have json message that contains another json message in textual form:
> > {
> > "one": "one".
> > "two":2,
> > "MSG": "{\"src\":\"my source\",\"priority\":\"0\"}"
> > }
> >
> > What would be transform spec to get "contained" message in json ?
> > {
> > "src": "my source",
> > "priority": "0"
> > }
> >
> > I've tried the following spec:
> > [
> >   {
> > "operation": "shift",
> > "spec": {
> >   "MSG": {
> > "*": "&"
> >   }
> > }
> >   }
> > ]
> > but ended up with just text message:
> > { "MSG": "{\"src\":\"my source\",\"priority\":\"0\"}"}
> > How should i change it?
> > Thank you
> >
>

jolt transform

2019-03-06 Thread l vic

I have json message that contains another json message in textual form:
{
"one": "one".
"two":2,
"MSG": "{\"src\":\"my source\",\"priority\":\"0\"}"
}

What would be transform spec to get "contained" message in json ?
{
"src": "my source",
"priority": "0"
}
Thank you

jolt transform spec ?

2019-03-06 Thread l vic

I have json message that contains another json message in textual form:
{
"one": "one".
"two":2,
"MSG": "{\"src\":\"my source\",\"priority\":\"0\"}"
}

What would be transform spec to get "contained" message in json ?
{
"src": "my source",
"priority": "0"
}

I've tried the following spec:
[
  {
"operation": "shift",
"spec": {
  "MSG": {
"*": "&"
  }
}
  }
]
but ended up with just text message:
{ "MSG": "{\"src\":\"my source\",\"priority\":\"0\"}"}
How should i change it?
Thank you

source build error?

2019-02-14 Thread l vic

Any idea what the problem is?


nifi-standard-processors: *Could not resolve dependencies for project
org.apache.nifi:nifi-standard-processors:jar:1.6.0: Failure to find
com.martiansoftware:macnificent:jar:0.2.0*

Re: PutSQL benchmarking ?

2019-02-12 Thread l vic

Would it be possible to work around this by passing "upsert" as attribute
to flowfile? If so: where can i find some examples of using
PutDatabaseRecord with RecordReader to extract/save Json array?
Thank you

On Thu, Feb 7, 2019 at 1:03 PM Matt Burgess  wrote:

> Yeah that's a gap that needs filling. I'm hopefully wrapping up some
> stuff shortly, and would like to take a crack at upsert for PDR.
>
> Regards,
> Matt
>
> On Thu, Feb 7, 2019 at 12:54 PM l vic  wrote:
> >
> > Sorry, I realize i do indeed perform record splitting, the problem with
> PutDatabaseRecord is that it doesn't seem to recognize "upsert"
> >
> > On Wed, Feb 6, 2019 at 4:10 PM Matt Burgess 
> wrote:
> >>
> >> If you don't do record splitting, how are you getting SQL to send to
> >> PutSQL? Can you describe your flow (processors, e.g.)?
> >>
> >> Thanks,
> >> Matt
> >>
> >> On Wed, Feb 6, 2019 at 3:41 PM l vic  wrote:
> >> >
> >> > Hi Matt,
> >> > No, I don't do record splitting, data looks like {
> "attr1":"val1",...[{}]}
> >> > where "parent" data is saved into 1 record in "parent" table and
> array data is saved into multiple records in "child" table...
> >> > What's "lineage duration"?
> >> > Event Duration
> >> > < 1ms
> >> > Lineage Duration
> >> > 00:00:00.070
> >> >
> >> > On Wed, Feb 6, 2019 at 2:59 PM Matt Burgess 
> wrote:
> >> >>
> >> >> In your flow, what does the data look like? Are you splitting it into
> >> >> individual records, then converting to SQL (probably via JSON) and
> >> >> calling PutSQL? If so, that's not going to be very performant; the
> >> >> PutDatabaseRecord processor combines all that together so you can
> >> >> leave your data in its original state (i.e. many records in one flow
> >> >> file). For benchmarking PutDatabaseRecord (PDR), you could provide
> >> >> sample data via GenerateFlowFile, run a few through PDR, and check
> the
> >> >> provenance events for fields such as durationMillis or calculations
> >> >> like (timestampMills - lineageStart).
> >> >>
> >> >> Regards,
> >> >> Matt
> >> >>
> >> >> On Wed, Feb 6, 2019 at 2:07 PM l vic  wrote:
> >> >> >
> >> >> > I have performance issues with PutSQL i my flow... Is there some
> way to benchmark time required to write certain number of records to table
> from GenerateFlowFile?
> >> >> > Thank you,
>

Re: PutSQL benchmarking ?

2019-02-07 Thread l vic

Sorry, I realize i do indeed perform record splitting, the problem with
PutDatabaseRecord is that it doesn't seem to recognize "upsert"

On Wed, Feb 6, 2019 at 4:10 PM Matt Burgess  wrote:

> If you don't do record splitting, how are you getting SQL to send to
> PutSQL? Can you describe your flow (processors, e.g.)?
>
> Thanks,
> Matt
>
> On Wed, Feb 6, 2019 at 3:41 PM l vic  wrote:
> >
> > Hi Matt,
> > No, I don't do record splitting, data looks like {
> "attr1":"val1",...[{}]}
> > where "parent" data is saved into 1 record in "parent" table and array
> data is saved into multiple records in "child" table...
> > What's "lineage duration"?
> > Event Duration
> > < 1ms
> > Lineage Duration
> > 00:00:00.070
> >
> > On Wed, Feb 6, 2019 at 2:59 PM Matt Burgess 
> wrote:
> >>
> >> In your flow, what does the data look like? Are you splitting it into
> >> individual records, then converting to SQL (probably via JSON) and
> >> calling PutSQL? If so, that's not going to be very performant; the
> >> PutDatabaseRecord processor combines all that together so you can
> >> leave your data in its original state (i.e. many records in one flow
> >> file). For benchmarking PutDatabaseRecord (PDR), you could provide
> >> sample data via GenerateFlowFile, run a few through PDR, and check the
> >> provenance events for fields such as durationMillis or calculations
> >> like (timestampMills - lineageStart).
> >>
> >> Regards,
> >> Matt
> >>
> >> On Wed, Feb 6, 2019 at 2:07 PM l vic  wrote:
> >> >
> >> > I have performance issues with PutSQL i my flow... Is there some way
> to benchmark time required to write certain number of records to table from
> GenerateFlowFile?
> >> > Thank you,
>

Re: PutSQL benchmarking ?

2019-02-06 Thread l vic

Hi Matt,
No, I don't do record splitting, data looks like {
"attr1":"val1",...[{}]}
where "parent" data is saved into 1 record in "parent" table and array data
is saved into multiple records in "child" table...
What's "lineage duration"?
Event Duration
< 1ms
Lineage Duration
00:00:00.070

On Wed, Feb 6, 2019 at 2:59 PM Matt Burgess  wrote:

> In your flow, what does the data look like? Are you splitting it into
> individual records, then converting to SQL (probably via JSON) and
> calling PutSQL? If so, that's not going to be very performant; the
> PutDatabaseRecord processor combines all that together so you can
> leave your data in its original state (i.e. many records in one flow
> file). For benchmarking PutDatabaseRecord (PDR), you could provide
> sample data via GenerateFlowFile, run a few through PDR, and check the
> provenance events for fields such as durationMillis or calculations
> like (timestampMills - lineageStart).
>
> Regards,
> Matt
>
> On Wed, Feb 6, 2019 at 2:07 PM l vic  wrote:
> >
> > I have performance issues with PutSQL i my flow... Is there some way to
> benchmark time required to write certain number of records to table from
> GenerateFlowFile?
> > Thank you,
>

PutSQL benchmarking ?

2019-02-06 Thread l vic

I have performance issues with PutSQL i my flow... Is there some way to
benchmark time required to write certain number of records to table from
GenerateFlowFile?
Thank you,

Re: expression failure in URL concatenation?

2019-02-01 Thread l vic

 "resolve the value of the REST_URL variable, then use that as the name of
a variable."
Doesn't seem to make much sense: the plan is to run "endsWith" on value of
REST_URL, doesn't your expression set to tun it on literal string
"REST_URL" within {...} ?
Thank you,

On Fri, Feb 1, 2019 at 8:58 AM Mark Payne  wrote:

> Hello,
>
> The issue appears to be that you're using `${${REST_URL}:endsWith`
> at the beginning, which is saying "resolve the value of the REST_URL
> variable, then use that as the name of a variable."
> So it's looking for a variable named "http://localhost:8080/nifi-api; and
> never finds it. So endsWith() evaluates to `false`
> in all cases.
>
> What you want is instead:
>
> *${REST_URL:endsWith('/'):ifElse('${REST_URL}resources','${REST_URL}/resources')}*
>
> I.e., check if the value of the REST_URL variable ends with a slash...
>
> Thanks
> -Mark
>
>
> On Jan 31, 2019, at 4:42 PM, l vic  wrote:
>
> I am using processor group variable as base part of my URL:
> REST_URL=http://localhost:8080/nifi-api
>
> I am trying to append second part of URL in InvokeHTTP regardless if
> REST_URL ends with '/', or not so that concatenation of "
> http://localhost:8080/nifi-api/;, or "http://localhost:8080/nifi-api;
> with "resources" return the same URL: "
> http://localhost:8080/nifi-api/resources":
>
> *${${REST_URL}:endsWith('/'):ifElse('${REST_URL}resources','${REST_URL}/resources')}*
>
> But the expression always appends '/', so in the first case I end up with *
> "http://localhost:8080/nifi-api//resources
> <http://localhost:8080/nifi-api//resources>"*... Any idea where the error
> is?
> Thank you,
>
>
>

Re: expression failure in URL concatenation?

2019-02-01 Thread l vic

No

On Thu, Jan 31, 2019 at 4:48 PM James Srinivasan 
wrote:

> Out of interest, does the URL containing the double slash work?
>
> On Thu, 31 Jan 2019, 21:42 l vic 
>> I am using processor group variable as base part of my URL:
>> REST_URL=http://localhost:8080/nifi-api
>>
>> I am trying to append second part of URL in InvokeHTTP regardless if
>> REST_URL ends with '/', or not so that concatenation of "
>> http://localhost:8080/nifi-api/;, or "http://localhost:8080/nifi-api;
>> with "resources" return the same URL: "
>> http://localhost:8080/nifi-api/resources":
>>
>>
>> *${${REST_URL}:endsWith('/'):ifElse('${REST_URL}resources','${REST_URL}/resources')}*
>>
>>
>> But the expression always appends '/', so in the first case I end up with 
>> *"http://localhost:8080/nifi-api//resources
>> <http://localhost:8080/nifi-api//resources>"*... Any idea where the
>> error is?
>>
>> Thank you,
>>
>>
>>

expression failure in URL concatenation?

2019-01-31 Thread l vic

I am using processor group variable as base part of my URL:
REST_URL=http://localhost:8080/nifi-api

I am trying to append second part of URL in InvokeHTTP regardless if
REST_URL ends with '/', or not so that concatenation of "
http://localhost:8080/nifi-api/;, or "http://localhost:8080/nifi-api; with
"resources" return the same URL: "http://localhost:8080/nifi-api/resources":

*${${REST_URL}:endsWith('/'):ifElse('${REST_URL}resources','${REST_URL}/resources')}*


But the expression always appends '/', so in the first case I end up
with *"http://localhost:8080/nifi-api//resources
"*... Any idea where the error
is?

Thank you,

Re: Unexpected char exception in InvokeHTTP

2019-01-30 Thread l vic

This template has been created on Nifi 1.6; It fails only on 1.5
environments: looks as InvokeHTTP 1.5 does not have this property

On Wed, Jan 30, 2019 at 1:29 PM Mark Payne  wrote:

> It looks like you are attempting to send a header named "Proxy Type". I
> don't believe that HTTP Headers
> are allowed to have spaces. So you'll want to check your configuration of
> the InvokeHTTP processor and see
> why it's trying to send that header.
>
> On Jan 30, 2019, at 12:01 PM, l vic  wrote:
>
> I am using InvokeHTTP with StandardRestrictedSSLContextService 1.6.0 to
> perform REST call with Proxy Type = https
>
>
> On some systems it just works, on other systems i have the following error:
>
> InvokeHTTP[id=d3f4b6c9-1204-3e1a-1373-431715543d24] Routing to Failure due
> to exception: Unexpected char 0x20 at 5 in header name: Proxy Type:
> java.lang.IllegalArgumentException: Unexpected char 0x20 at 5 in header
> name: Proxy Type
>
> This exception doesn't seem to appear anywhere in logs, any idea how to
> troubleshoot this problem, or what can cause it?
> Thank you
>
>
>

Re: [EXT] Unexpected char exception in InvokeHTTP

2019-01-30 Thread l vic

Can someone explain what's this property for in the first place? Is it
needed for SSLContextService? I didn't find it in documentation...

On Wed, Jan 30, 2019 at 12:57 PM Vincent, Mike  wrote:

> Are you sure you aren’t using “https “ (notice the space after the ‘s’)?
> Since 0x20 is an ASCII space, I wonder if that’s what’s causing your issue?
>
>
>
> Cheers,
>
>
>
> Michael J. Vincent
>
> Lead Network Systems Engineer | The MITRE Corporation | Network Technology
> & Security (T864) | +1 (781) 271-8381
>
>
>
> *From:* l vic 
> *Sent:* Wednesday, January 30, 2019 12:01 PM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] Unexpected char exception in InvokeHTTP
>
>
>
> I am using InvokeHTTP with StandardRestrictedSSLContextService 1.6.0 to
> perform REST call with Proxy Type = https
>
>
>
>
>
> On some systems it just works, on other systems i have the following error:
>
>
>
> InvokeHTTP[id=d3f4b6c9-1204-3e1a-1373-431715543d24] Routing to Failure due
> to exception: Unexpected char 0x20 at 5 in header name: Proxy Type:
> java.lang.IllegalArgumentException: Unexpected char 0x20 at 5 in header
> name: Proxy Type
>
>
>
> This exception doesn't seem to appear anywhere in logs, any idea how to
> troubleshoot this problem, or what can cause it?
>
> Thank you
>

Re: [EXT] Unexpected char exception in InvokeHTTP

2019-01-30 Thread l vic

I've been wondering about it myself but tweaking that value doesn't change
anything Yes I made sure it's "https", not "https "

On Wed, Jan 30, 2019 at 12:57 PM Vincent, Mike  wrote:

> Are you sure you aren’t using “https “ (notice the space after the ‘s’)?
> Since 0x20 is an ASCII space, I wonder if that’s what’s causing your issue?
>
>
>
> Cheers,
>
>
>
> Michael J. Vincent
>
> Lead Network Systems Engineer | The MITRE Corporation | Network Technology
> & Security (T864) | +1 (781) 271-8381
>
>
>
> *From:* l vic 
> *Sent:* Wednesday, January 30, 2019 12:01 PM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] Unexpected char exception in InvokeHTTP
>
>
>
> I am using InvokeHTTP with StandardRestrictedSSLContextService 1.6.0 to
> perform REST call with Proxy Type = https
>
>
>
>
>
> On some systems it just works, on other systems i have the following error:
>
>
>
> InvokeHTTP[id=d3f4b6c9-1204-3e1a-1373-431715543d24] Routing to Failure due
> to exception: Unexpected char 0x20 at 5 in header name: Proxy Type:
> java.lang.IllegalArgumentException: Unexpected char 0x20 at 5 in header
> name: Proxy Type
>
>
>
> This exception doesn't seem to appear anywhere in logs, any idea how to
> troubleshoot this problem, or what can cause it?
>
> Thank you
>

Unexpected char exception in InvokeHTTP

2019-01-30 Thread l vic

I am using InvokeHTTP with StandardRestrictedSSLContextService 1.6.0 to
perform REST call with Proxy Type = https


On some systems it just works, on other systems i have the following error:


InvokeHTTP[id=d3f4b6c9-1204-3e1a-1373-431715543d24] Routing to Failure due
to exception: Unexpected char 0x20 at 5 in header name: Proxy Type:
java.lang.IllegalArgumentException: Unexpected char 0x20 at 5 in header
name: Proxy Type


This exception doesn't seem to appear anywhere in logs, any idea how to
troubleshoot this problem, or what can cause it?

Thank you

problem with merging of failed response

2019-01-11 Thread l vic

I have to merge results of "original", REST call and sql query  responses
into one flowfile/json
The problem happens if REST call, or sql query result in empty resultset.
For example, suppose i have 3 incoming connections for Merge:
original->flowfile1/attribute1, REST->flowfile2/attribute2, SQL
query->flowfile3/attribute3
If REST call fails (error 404 - record not found), or sql call return empty
resultset - flowfiles never merged ( the origin gets stuck in input queue).
Can I found some merge strategy to solve this problem such as only origin
json would be let through? For example, is there something i can do so that
REST error would result in empty output flowfile?
Thank you...

Re: merging with default values?

2019-01-10 Thread l vic

Yes i do know them ahead of time, UpdateAttribute rules might work.
Thank you

On Thu, Jan 10, 2019 at 2:46 PM Russ Weiher 
wrote:

> Do you know the name of the attributes you are looking for ahead of time?
> If so, the UpdateAttribute processor has an “Advanced” section that lets
> you write conditional rules based on flowfile attributes and properties.
> If you haven’t already checked that out, it might be a good place to
> start.  There are some good examples in the NiFi documentation for that
> processor.
>
>
>
> *Russ Weiher*
>
> BI CI Solutions Architect
> Progenity, Inc.
>
>
>
> *From:* l vic [mailto:lvic4...@gmail.com]
> *Sent:* Thursday, January 10, 2019 2:18 PM
> *To:* users@nifi.apache.org
> *Subject:* merging with default values?
>
>
>
> I have to merge results of 3 different queries into one flowfile
>
> The problem happens if some of the queries result in empty resultset. For
> example, suppose i have 3 incoming connections for MergeContent:
>
> query1->flowfile1/attribute1, query2->flowfile2/attribute2,
> query3->flowfile3/attribute3
>
> And query2 returns empty resultset. In this case attribute2 never shows up
> in the merged flowfile.
>
> Can someone recommend merge solution where i would always have attribute2
> in the result but with some default value ( empty string assuming
> attribute2 is text)?
>
> Thank you...
> IMPORTANT: This email (and any attachments) is intended for the use of
> only the person or entity to which it is addressed, and may contain
> information that is privileged and confidential. You, the recipient, are
> obligated to maintain it in a safe, secure and confidential manner.
> Unauthorized redisclosure or failure to maintain confidentiality may
> subject you to federal and state penalties. If you are not the intended
> recipient, please immediately notify us by return email, and delete this
> message from your computer.
>

merging with default values?

2019-01-10 Thread l vic

I have to merge results of 3 different queries into one flowfile
The problem happens if some of the queries result in empty resultset. For
example, suppose i have 3 incoming connections for MergeContent:
query1->flowfile1/attribute1, query2->flowfile2/attribute2,
query3->flowfile3/attribute3
And query2 returns empty resultset. In this case attribute2 never shows up
in the merged flowfile.
Can someone recommend merge solution where i would always have attribute2
in the result but with some default value ( empty string assuming
attribute2 is text)?
Thank you...

Re: InvokeHTTP failure to schedule fro CRON?

2018-12-24 Thread l vic

Could you try using an explicit path to the cacerts provided by your
JDK/JRE, instead of referring to $JAVA_HOME?
Tried without success...
Were you able to successfully start the SSLContextService after configuring
it?
Yes
InvokeHTTP needs to present a certificate that is signed by a CA that is in
the default cacerts
Not sure how to identify one that is supposed to be presented


On Sun, Dec 23, 2018 at 1:32 PM Jeff  wrote:

> Could you try using an explicit path to the cacerts provided by your
> JDK/JRE, instead of referring to $JAVA_HOME?  Andy gave an example of
> "/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/cacerts",
> which you would update with the path to the JDK you are using.  Referencing
> an environment variable (without using EL) will not work for a NiFi
> property.  It does not appear that EL is supported for the keystore and
> truststore properties, as that could lead to security issues.  Those
> properties have validators that should also verify that the
> keystore/truststore exists and is readable.  Were you able to successfully
> start the SSLContextService after configuring it?
>
> Also, as Andy mentioned, the URL you are using in InvokeHTTP needs to
> present a certificate that is signed by a CA that is in the default
> cacerts.  Can you please verify this?  You can get a list of what is
> contained in cacerts by using keytool, and specifying the path to cacerts,
> the password, and the list command.  For example:
>
> keytool -storepass changeit -keystore
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/security/cacerts
> -list
>
> - Jeff
>
> On Fri, Dec 21, 2018 at 2:55 PM l vic  wrote:
>
>> I put "default" parameters for trust-store:
>> Path: $JAVA_HOME/jre/lib/security/cacerts
>> Password: changeit (default)
>> Type: JKS
>>  and got "invalid path" exception ( see below)
>> How does that missing cert file should look like?
>> Thanks again...
>>
>> 2018-12-21 14:46:00,021 ERROR [Timer-Driven Process Thread-1]
>> o.a.nifi.processors.standard.InvokeHTTP
>> InvokeHTTP[id=0929346d-d742-1fd9-e41a-8e4324b73349] Yielding processor due
>> to exception encountered as a source processor:
>> javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target: {}
>>
>> javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target
>>
>>at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>
>>at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964)
>>
>>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:328)
>>
>>at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
>>
>>at
>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
>>
>>at
>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
>>
>>at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
>>
>>at sun.security.ssl.Handshaker.process_record(Handshaker.java:987)
>>
>>at
>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072)
>>
>>at
>> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
>>
>>at
>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
>>
>>at
>> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
>>
>>at
>> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:267)
>>
>>at
>> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:237)
>>
>>at
>> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:148)
>>
>>at
>> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:186)
>>
>>at
>> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121)
>>
>>at
>> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100)
>>
>>at
>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
>>
>>at

Re: InvokeHTTP failure to schedule fro CRON?

2018-12-21 Thread l vic

tion: unable
to find valid certification path to requested target

   at
sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)

   at
sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)

   at sun.security.validator.Validator.validate(Validator.java:260)

   at
sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)

   at
sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)

   at
sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)

   at
sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596)

   ... 39 common frames omitted

Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target

   at
sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)

   at
sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)

   at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)

   at
sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)

   ... 45 common frames omitted






On Thu, Dec 20, 2018 at 4:14 PM Andy LoPresto  wrote:

> You need to configure the truststore properties in the SSLContextService —
> the keystore contains the private key and public certificate the service
> (NiFi) uses to identify itself, but the truststore contains the public
> certificate(s) of external services NiFi should trust. In this case, in
> order to connect to another service at https://service.external.com, you
> will need to have the public certificate (pub1) of the External Service or
> one of the public certificates in the chain that signed that pub1. If this
> is a site on the public internet, you can probably use the JVM defaults, as
> it will likely be signed by a known certificate authority. If not, you must
> obtain that public certificate independently, put it in a JKS truststore,
> and populate the controller service properties for it.
>
> JVM truststore:
>
> Path: $JAVA_HOME/jre/lib/security/cacerts (i.e.
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/cacerts)
> Password: changeit (default)
> Type: JKS
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Dec 20, 2018, at 2:31 PM, l vic  wrote:
>
> Hello,
> I am trying to perform "get" request over SSL from InvokeHTTP
> nifi-1.5.0-RC1;
> I configured SSL by the means of a StandardSSLContextService
> <https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.ssl.StandardSSLContextService/>
>  with
> jks certificate (see attached)
> When I try to execute processor, i see the following problem:
> *Caused by: java.lang.IllegalStateException: TrustManagerFactoryImpl is
> not initialized*
> *at
> sun.security.ssl.TrustManagerFactoryImpl.engineGetTrustManagers(TrustManagerFactoryImpl.java:100)*
> Do I have an error in my configuration, or is this bug? keystore
> file/password combination is valid - i can do that request from cli Can
> I do "insecure" SSL request ( like curl -k) with InvokeHTTP?
> Below is full stack trace
>
> 2018-12-20 14:53:41,116 ERROR [StandardProcessScheduler Thread-3]
> o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method
> due to java.lang.RuntimeException: Failed while executing one of
> processor's OnScheduled task.
> java.lang.RuntimeException: Failed while executing one of processor's
> OnScheduled task.
> at
> org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1504)
> at
> org.apache.nifi.controller.StandardProcessorNode.initiateStart(StandardProcessorNode.java:1330)
> at
> org.apache.nifi.controller.StandardProcessorNode.lambda$start$0(StandardProcessorNode.java:1315)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.reflect.Invoc

InvokeHTTP failure to schedule fro CRON?

2018-12-20 Thread l vic

Hello,
I am trying to perform "get" request over SSL from InvokeHTTP
nifi-1.5.0-RC1;
I configured SSL by the means of a StandardSSLContextService

with
jks certificate (see attached)
When I try to execute processor, i see the following problem:

*Caused by: java.lang.IllegalStateException: TrustManagerFactoryImpl is not
initialized*

*at
sun.security.ssl.TrustManagerFactoryImpl.engineGetTrustManagers(TrustManagerFactoryImpl.java:100)*
Do I have an error in my configuration, or is this bug? keystore
file/password combination is valid - i can do that request from cli Can
I do "insecure" SSL request ( like curl -k) with InvokeHTTP?
Below is full stack trace

2018-12-20 14:53:41,116 ERROR [StandardProcessScheduler Thread-3]
o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method
due to java.lang.RuntimeException: Failed while executing one of
processor's OnScheduled task.

java.lang.RuntimeException: Failed while executing one of processor's
OnScheduled task.

at
org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1504)

at
org.apache.nifi.controller.StandardProcessorNode.initiateStart(StandardProcessorNode.java:1330)

at
org.apache.nifi.controller.StandardProcessorNode.lambda$start$0(StandardProcessorNode.java:1315)

at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.ExecutionException:
java.lang.reflect.InvocationTargetException

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:206)

at
org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1487)

... 9 common frames omitted

Caused by: java.lang.reflect.InvocationTargetException: null

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)

at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)

at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)

at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)

at
org.apache.nifi.controller.StandardProcessorNode$1.call(StandardProcessorNode.java:1334)

at
org.apache.nifi.controller.StandardProcessorNode$1.call(StandardProcessorNode.java:1330)

... 6 common frames omitted

Caused by: java.lang.IllegalStateException: TrustManagerFactoryImpl is not
initialized

at
sun.security.ssl.TrustManagerFactoryImpl.engineGetTrustManagers(TrustManagerFactoryImpl.java:100)

at
javax.net.ssl.TrustManagerFactory.getTrustManagers(TrustManagerFactory.java:285)

at
org.apache.nifi.processors.standard.InvokeHTTP.setSslSocketFactory(InvokeHTTP.java:613)

at
org.apache.nifi.processors.standard.InvokeHTTP.setUpClient(InvokeHTTP.java:545)

... 16 common frames omitted

Re: setting topic name dynamically in PutKafka?

2018-11-20 Thread l vic

It does work, at least in the prototype

On Tue, Nov 20, 2018 at 2:08 PM Bryan Bende  wrote:

> That is correct for the ConsumeKafka processor, but this question was
> about the producer side :)
>
> I believe you can make a producer ahead of time, and then specify the
> topic when making a call to actually send a message, at least that
> what it looks like the code does in all the PublishKafka variants...
>
>
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-0-9-processors/src/main/java/org/apache/nifi/processors/kafka/pubsub/PublishKafka.java#L325
>
> On Tue, Nov 20, 2018 at 1:53 PM Boris Tyukin 
> wrote:
> >
> > I think they did in on purpose because it takes time (3-5 seconds) for a
> consumer group to connect to Kafka (joining consumer group) and whoever
> created processor did not want to do it for each flowfile.
> >
> > On Tue, Nov 20, 2018 at 10:09 AM l vic  wrote:
> >>
> >> Hi,
> >> I have to set topic name property of PutKafka processors from an
> attribute of the incoming flowfile...Is there any possible way of doing
> this?
> >> Thank you,
> >> -V
>

setting topic name dynamically in PutKafka?

2018-11-20 Thread l vic

Hi,
I have to set topic name property of PutKafka processors from an attribute
of the incoming flowfile...Is there any possible way of doing this?
Thank you,
-V

Re: UpdateAttributes use problems

2018-11-19 Thread l vic

"AttributesToJson" seems to work

On Mon, Nov 19, 2018 at 10:14 AM Nathan Gough  wrote:

> UpdateAttribute only updates flowfile attributes and will not change the
> content. Flowfile attributes and the actual content are two separate
> objects and are not linked. Updating one does not affect the other. If you
> want to manipulate your JSON content, look at the JoltTransformJSON
> processor:
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.8.0/org.apache.nifi.processors.standard.JoltTransformJSON/index.html
>
>
>
>
>
> *From: *l vic 
> *Reply-To: *
> *Date: *Saturday, November 17, 2018 at 8:29 AM
> *To: *
> *Subject: *Re: UpdateAttributes use problems
>
>
>
> Actually, if I check flowfile attributes after UpdateAttributes - it does
> work, eg 'value_filter' and 'name' are gone and 'ts' is added. But contents
> of the queue, or data provenance of the next processor doesn't seem to
> change, eg output claim still looks like {"value_filter":"*", "name":"me"}
>
> It's confusing: if I change attributes in flowfile, shouldn't it be
> reflected in the output json, or should i add another processor to make it
> happen? After all, it's really that json content i am seeking to change...
>
>
>
> On Fri, Nov 16, 2018 at 4:27 PM Nathan Gough  wrote:
>
> I set up an UpdateAttribute processor with the same configuration as the
> screenshot you sent and it worked for me.
>
>
>
> What version of NiFi are you using?
>
>
>
> *From: *l vic 
> *Reply-To: *
> *Date: *Friday, November 16, 2018 at 1:53 PM
> *To: *
> *Subject: *UpdateAttributes use problems
>
>
>
> Hello,
>
> I am trying to use UpdateAttributes remove 'value_filter" and "name"
> attributes from flowfile and add timestamp attribute, so I specified
> "value_filter | name" in "DeleteAttributesExpression" and added attribute
> "ts" with value '${now():toNumber()}'... No changes in resulting flowfile
> and no failures. What am i doing wrong?
>
> Thanks,
>
>
>
>

Re: UpdateAttributes use problems

2018-11-17 Thread l vic

Actually, if I check flowfile attributes after UpdateAttributes - it does
work, eg 'value_filter' and 'name' are gone and 'ts' is added. But contents
of the queue, or data provenance of the next processor doesn't seem to
change, eg output claim still looks like {"value_filter":"*", "name":"me"}
It's confusing: if I change attributes in flowfile, shouldn't it be
reflected in the output json, or should i add another processor to make it
happen? After all, it's really that json content i am seeking to change...

On Fri, Nov 16, 2018 at 4:27 PM Nathan Gough  wrote:

> I set up an UpdateAttribute processor with the same configuration as the
> screenshot you sent and it worked for me.
>
>
>
> What version of NiFi are you using?
>
>
>
> *From: *l vic 
> *Reply-To: *
> *Date: *Friday, November 16, 2018 at 1:53 PM
> *To: *
> *Subject: *UpdateAttributes use problems
>
>
>
> Hello,
>
> I am trying to use UpdateAttributes remove 'value_filter" and "name"
> attributes from flowfile and add timestamp attribute, so I specified
> "value_filter | name" in "DeleteAttributesExpression" and added attribute
> "ts" with value '${now():toNumber()}'... No changes in resulting flowfile
> and no failures. What am i doing wrong?
>
> Thanks,
>
>
>

Re: UpdateAttributes use problems

2018-11-17 Thread l vic

1.7.1

On Fri, Nov 16, 2018 at 4:27 PM Nathan Gough  wrote:

> I set up an UpdateAttribute processor with the same configuration as the
> screenshot you sent and it worked for me.
>
>
>
> What version of NiFi are you using?
>
>
>
> *From: *l vic 
> *Reply-To: *
> *Date: *Friday, November 16, 2018 at 1:53 PM
> *To: *
> *Subject: *UpdateAttributes use problems
>
>
>
> Hello,
>
> I am trying to use UpdateAttributes remove 'value_filter" and "name"
> attributes from flowfile and add timestamp attribute, so I specified
> "value_filter | name" in "DeleteAttributesExpression" and added attribute
> "ts" with value '${now():toNumber()}'... No changes in resulting flowfile
> and no failures. What am i doing wrong?
>
> Thanks,
>
>
>

Re: Getting a timestamp for today at midnight?

2018-11-16 Thread l vic

That returns timestamp corresponding to 19:00PM
For example:

*long* myTimeAsLong=154232640L; // from NiFi

DateFormat sdf = *new* SimpleDateFormat("/MM/dd HH:mm:ss");

System.*out*.println(sdf.format(*new* Date(myTimeAsLong)));


prints out 2018/11/15 19:00:00

On Wed, Nov 14, 2018 at 9:21 AM Mark Payne  wrote:

> You should be able to do something like:
>
> ${now():divide( 8640 ):multiply( 8640)}
>
> I.e., use integer division to divide by number of milliseconds in a day,
> which gives you
> the number of days since epoch. Then multiply by 86,400,000 again to
> convert from
> days back to milliseconds. While it looks immediately like it would do
> nothing, the thing
> to keep in mind is that the divide() function performs and Integer
> Division operation,
> not a Decimal Division, so all decimals would be dropped, which
> essentially results in
> all hours/mins/seconds/milliseconds getting dropped.
>
> Thanks
> -Mark
>
>
> > On Nov 14, 2018, at 9:01 AM, l vic  wrote:
> >
> > I have to retrieve "today's" records in ExecuteSQL, eg. with the query
> using timestamp "ts" column in table where ts type is "epoch" time as long
> integer :
> > select * from mytable where ts >= midnight-timestamp-value
> > Any idea how i can use "now()" function to get today's timestamp value
> at midnight?
> > Thank you,
>
>

UpdateAttributes use problems

2018-11-16 Thread l vic

Hello,
I am trying to use UpdateAttributes remove 'value_filter" and "name"
attributes from flowfile and add timestamp attribute, so I specified
"value_filter | name" in "DeleteAttributesExpression" and added attribute
"ts" with value '${now():toNumber()}'... No changes in resulting flowfile
and no failures. What am i doing wrong?
Thanks,

Getting a timestamp for today at midnight?

2018-11-14 Thread l vic

I have to retrieve "today's" records in ExecuteSQL, eg. with the query
using timestamp "ts" column in table where ts type is "epoch" time as long
integer :
select * from mytable where ts >= midnight-timestamp-value
Any idea how i can use "now()" function to get today's timestamp value at
midnight?
Thank you,

Re: [EXT] ExecuteSQL: convertToAvroStream failure with SQlite integer

2018-11-09 Thread l vic

Works if change column type to UNSIGNED BIG INT in SQLite

On Fri, Nov 9, 2018 at 9:33 AM Colin Dean  wrote:

> You could try dropping in the relevant NAR from 1.8.0 but otherwise, no.
> The 1.8.0 version will make it a lot easier to figure out what SQL type is
> causing the problem, though, and will probably result in a bug report.
>
>
>
>
>
> *From: *l vic 
> *Reply-To: *"users@nifi.apache.org" 
> *Date: *Friday, November 9, 2018 at 6:52 AM
> *To: *"users@nifi.apache.org" 
> *Subject: *Re: [EXT] ExecuteSQL: convertToAvroStream failure with SQlite
> integer
>
>
>
> Using ver. 1.7... Any way around it in existing version?
>
>
>
> On Thu, Nov 8, 2018 at 5:53 PM Colin Dean  wrote:
>
> What version of NiFi are you using? An error like this comes up every now
> and then; one was just fixed in NiFi 1.8.0 but it was related to JDBC
> drivers that return Long for unsigned ints. 1.8.0 also improved the error
> message so that it should show the type of the object that was passed into
> the unresolvable union.
>
>
>
> https://github.com/apache/nifi/pull/3032
>
>
>
>
>
> *From: *l vic 
> *Reply-To: *"users@nifi.apache.org" 
> *Date: *Thursday, November 8, 2018 at 5:43 PM
> *To: *"users@nifi.apache.org" 
> *Subject: *[EXT] ExecuteSQL: convertToAvroStream failure with SQlite
> integer
>
>
>
> Hi, I am trying to use ExecuteSQL to get "epoch time" value from SQLite
> table:
>
> select start_date from sched
>
> where start_time is defined as INTEGER
>
> If the start_date = 1536548297955 i see the following exception:
>
> failed to process due to
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]:
> 1536548297955; rolling back session: {}
>
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]:
> 1536548297955
>
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
>
> at
>
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null","int"]: 1536548297955
>
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:709)
>
>
>
> This is obviously Avro conversion issue as this works from sqlite3 CLI..
>
> If I try to define it as BIGINT i
> have org.apache.avro.UnresolvedUnionException: Not in union
> ["null","long"]: 1536548297955;
>
> Any idea how i can resolve this?
>
> Thanks,
>
> -V
>
>

Re: [EXT] ExecuteSQL: convertToAvroStream failure with SQlite integer

2018-11-09 Thread l vic

Using ver. 1.7... Any way around it in existing version?

On Thu, Nov 8, 2018 at 5:53 PM Colin Dean  wrote:

> What version of NiFi are you using? An error like this comes up every now
> and then; one was just fixed in NiFi 1.8.0 but it was related to JDBC
> drivers that return Long for unsigned ints. 1.8.0 also improved the error
> message so that it should show the type of the object that was passed into
> the unresolvable union.
>
>
>
> https://github.com/apache/nifi/pull/3032
>
>
>
>
>
> *From: *l vic 
> *Reply-To: *"users@nifi.apache.org" 
> *Date: *Thursday, November 8, 2018 at 5:43 PM
> *To: *"users@nifi.apache.org" 
> *Subject: *[EXT] ExecuteSQL: convertToAvroStream failure with SQlite
> integer
>
>
>
> Hi, I am trying to use ExecuteSQL to get "epoch time" value from SQLite
> table:
>
> select start_date from sched
>
> where start_time is defined as INTEGER
>
> If the start_date = 1536548297955 i see the following exception:
>
> failed to process due to
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]:
> 1536548297955; rolling back session: {}
>
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]:
> 1536548297955
>
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
>
> at
>
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null","int"]: 1536548297955
>
> at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:709)
>
>
>
> This is obviously Avro conversion issue as this works from sqlite3 CLI..
>
> If I try to define it as BIGINT i
> have org.apache.avro.UnresolvedUnionException: Not in union
> ["null","long"]: 1536548297955;
>
> Any idea how i can resolve this?
>
> Thanks,
>
> -V
>

ExecuteSQL: convertToAvroStream failure with SQlite integer

2018-11-08 Thread l vic

Hi, I am trying to use ExecuteSQL to get "epoch time" value from SQLite
table:
select start_date from sched
where start_time is defined as INTEGER
If the start_date = 1536548297955 i see the following exception:

failed to process due to
org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]:
1536548297955; rolling back session: {}

org.apache.avro.file.DataFileWriter$AppendWriteException:
org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]:
1536548297955

at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)

at

Caused by: org.apache.avro.UnresolvedUnionException: Not in union
["null","int"]: 1536548297955

at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:709)


This is obviously Avro conversion issue as this works from sqlite3 CLI..

If I try to define it as BIGINT i have
org.apache.avro.UnresolvedUnionException:
Not in union ["null","long"]: 1536548297955;
Any idea how i can resolve this?
Thanks,
-V

flowfile duplication?

2018-11-02 Thread l vic

Hello,
I need to  create 2 independent paths of execution for the same data (
message consumed from kafka topic) One path would save it to one db
table, another would be used to perform some computations on attributes and
save it to another table. I wonder if DuplicateFlowFile can be used for
that It looks as it creates several copies in each path of execution.
Would any other processor be a better option?
Thank you,
V

Re: schedule execution from relational database?

2018-10-30 Thread l vic

Hi Ed,
It usually would be days from the moment new value is captured

On Mon, Oct 29, 2018 at 6:29 PM Ed B  wrote:

> Hey Victor,
>
> If you already pulled the record and know new value - that won't really
> help you to determine a change in a schedule.
> In my opinion, the schedule determined by the acceptable data latency for
> given application, in other words, how soon you want your changed data be
> captured.
> The answer can be from "real-time" to "on-demand".
>
> For your particular case, you need to decide and then either schedule
> every X sec/min/hours/days, etc, or at given time (at minute 30 of each
> hour every day except for Saturday).
> If you don't know what your requirements for data availability and latency
> are, you could start with something like every "5 mins". And then adjust as
> needed.
>
> Regards,
> Ed.
>
>
> On Mon, Oct 29, 2018 at 3:58 PM l vic  wrote:
>
>> QDT works, eg it can detect change in MaximumValue column but how can I
>> use it to schedule cron job? I know it's possible to schedule cron from UI
>> but how can i do it based on the value of attribute?
>> Thank you again,
>> V.
>>
>> On Mon, Oct 29, 2018 at 12:39 PM Matt Burgess 
>> wrote:
>>
>>> Victor,
>>>
>>> Yes, both QDT and GTF would generate something like "SELECT * from
>>> myTable where event_time > X", and QDT will execute it and update X.
>>> So if event_time is always increasing, it will continue to pick up the
>>> same row(s).
>>>
>>> That's a curious use case, maybe NiFi could handle other parts of it
>>> so you wouldn't need to update a single row in an external database
>>> table?
>>>
>>> Regards,
>>> Matt
>>>
>>> On Mon, Oct 29, 2018 at 12:36 PM l vic  wrote:
>>> >
>>> > What if have only one row and update the values in it? Will QDT fetch
>>> updates?
>>> > Thank you,
>>> > Victor
>>> >
>>> >
>>> > On Mon, Oct 29, 2018 at 11:54 AM Matt Burgess 
>>> wrote:
>>> >>
>>> >> You can use QueryDatabaseTable (QDT) for this, you'd set your
>>> >> "event_time" column as the "Maximum Value Column(s)" property in the
>>> >> processor. The first time QDT executes, it will fetch all the rows
>>> >> (since it has not seen event_time before), then it will keep track of
>>> >> the largest value of event_time. As new rows are added (with larger
>>> >> event_time values), QDT will only fetch the rows whose event_time is
>>> >> greater than the largest one it's seen. Then it updates its "largest
>>> >> seen value" and so on.
>>> >>
>>> >> GenerateTableFetch (GTF) is another option, it works in a similar
>>> >> fashion, except that it does not fetch the rows itself, instead it
>>> >> generates flow files containing SQL statements that you can send
>>> >> downstream to perhaps ExecuteSQL in order to actually fetch the rows.
>>> >> GTF is often used in place of QDT if you'll be fetching a large number
>>> >> of rows in each statement, as you can distribute the SQL flow files
>>> >> among the nodes in a cluster, to do the fetch in parallel.
>>> >>
>>> >> Regards,
>>> >> Matt
>>> >>
>>> >> On Mon, Oct 29, 2018 at 11:13 AM l vic  wrote:
>>> >> >
>>> >> > Hi,
>>> >> > i have "event_time" field in SQLite database that means epoch time
>>> for triggering of external event. What processor(s) can i use to implement
>>> schedule monitoring/ execution based on change in "event_time" value?
>>> >> > Thanks,
>>>
>>

Re: schedule execution from relational database?

2018-10-30 Thread l vic

My purpose is to use new epoch milliseconds value in flow file to  schedule
spark job at corresponding date/time, I am asking how that can be done in
NiFi.
Thank you,
Victor


On Mon, Oct 29, 2018 at 4:30 PM Matt Burgess  wrote:

> Not sure I understand what you mean. Are you using the flow file to
> trigger ExecuteStreamCommand to schedule a cron job? Or do you mean
> scheduling a processor to run in NiFi? Or something else?
> On Mon, Oct 29, 2018 at 3:58 PM l vic  wrote:
> >
> > QDT works, eg it can detect change in MaximumValue column but how can I
> use it to schedule cron job? I know it's possible to schedule cron from UI
> but how can i do it based on the value of attribute?
> > Thank you again,
> > V.
> >
> > On Mon, Oct 29, 2018 at 12:39 PM Matt Burgess 
> wrote:
> >>
> >> Victor,
> >>
> >> Yes, both QDT and GTF would generate something like "SELECT * from
> >> myTable where event_time > X", and QDT will execute it and update X.
> >> So if event_time is always increasing, it will continue to pick up the
> >> same row(s).
> >>
> >> That's a curious use case, maybe NiFi could handle other parts of it
> >> so you wouldn't need to update a single row in an external database
> >> table?
> >>
> >> Regards,
> >> Matt
> >>
> >> On Mon, Oct 29, 2018 at 12:36 PM l vic  wrote:
> >> >
> >> > What if have only one row and update the values in it? Will QDT fetch
> updates?
> >> > Thank you,
> >> > Victor
> >> >
> >> >
> >> > On Mon, Oct 29, 2018 at 11:54 AM Matt Burgess 
> wrote:
> >> >>
> >> >> You can use QueryDatabaseTable (QDT) for this, you'd set your
> >> >> "event_time" column as the "Maximum Value Column(s)" property in the
> >> >> processor. The first time QDT executes, it will fetch all the rows
> >> >> (since it has not seen event_time before), then it will keep track of
> >> >> the largest value of event_time. As new rows are added (with larger
> >> >> event_time values), QDT will only fetch the rows whose event_time is
> >> >> greater than the largest one it's seen. Then it updates its "largest
> >> >> seen value" and so on.
> >> >>
> >> >> GenerateTableFetch (GTF) is another option, it works in a similar
> >> >> fashion, except that it does not fetch the rows itself, instead it
> >> >> generates flow files containing SQL statements that you can send
> >> >> downstream to perhaps ExecuteSQL in order to actually fetch the rows.
> >> >> GTF is often used in place of QDT if you'll be fetching a large
> number
> >> >> of rows in each statement, as you can distribute the SQL flow files
> >> >> among the nodes in a cluster, to do the fetch in parallel.
> >> >>
> >> >> Regards,
> >> >> Matt
> >> >>
> >> >> On Mon, Oct 29, 2018 at 11:13 AM l vic  wrote:
> >> >> >
> >> >> > Hi,
> >> >> > i have "event_time" field in SQLite database that means epoch time
> for triggering of external event. What processor(s) can i use to implement
> schedule monitoring/ execution based on change in "event_time" value?
> >> >> > Thanks,
>

Re: schedule execution from relational database?

2018-10-29 Thread l vic

QDT works, eg it can detect change in MaximumValue column but how can I use
it to schedule cron job? I know it's possible to schedule cron from UI but
how can i do it based on the value of attribute?
Thank you again,
V.

On Mon, Oct 29, 2018 at 12:39 PM Matt Burgess  wrote:

> Victor,
>
> Yes, both QDT and GTF would generate something like "SELECT * from
> myTable where event_time > X", and QDT will execute it and update X.
> So if event_time is always increasing, it will continue to pick up the
> same row(s).
>
> That's a curious use case, maybe NiFi could handle other parts of it
> so you wouldn't need to update a single row in an external database
> table?
>
> Regards,
> Matt
>
> On Mon, Oct 29, 2018 at 12:36 PM l vic  wrote:
> >
> > What if have only one row and update the values in it? Will QDT fetch
> updates?
> > Thank you,
> > Victor
> >
> >
> > On Mon, Oct 29, 2018 at 11:54 AM Matt Burgess 
> wrote:
> >>
> >> You can use QueryDatabaseTable (QDT) for this, you'd set your
> >> "event_time" column as the "Maximum Value Column(s)" property in the
> >> processor. The first time QDT executes, it will fetch all the rows
> >> (since it has not seen event_time before), then it will keep track of
> >> the largest value of event_time. As new rows are added (with larger
> >> event_time values), QDT will only fetch the rows whose event_time is
> >> greater than the largest one it's seen. Then it updates its "largest
> >> seen value" and so on.
> >>
> >> GenerateTableFetch (GTF) is another option, it works in a similar
> >> fashion, except that it does not fetch the rows itself, instead it
> >> generates flow files containing SQL statements that you can send
> >> downstream to perhaps ExecuteSQL in order to actually fetch the rows.
> >> GTF is often used in place of QDT if you'll be fetching a large number
> >> of rows in each statement, as you can distribute the SQL flow files
> >> among the nodes in a cluster, to do the fetch in parallel.
> >>
> >> Regards,
> >> Matt
> >>
> >> On Mon, Oct 29, 2018 at 11:13 AM l vic  wrote:
> >> >
> >> > Hi,
> >> > i have "event_time" field in SQLite database that means epoch time
> for triggering of external event. What processor(s) can i use to implement
> schedule monitoring/ execution based on change in "event_time" value?
> >> > Thanks,
>

Re: schedule execution from relational database?

2018-10-29 Thread l vic

Hi Matt,
Nifi does handle other parts of it, just different process group.
Regards,
Victor

On Mon, Oct 29, 2018 at 12:39 PM Matt Burgess  wrote:

> Victor,
>
> Yes, both QDT and GTF would generate something like "SELECT * from
> myTable where event_time > X", and QDT will execute it and update X.
> So if event_time is always increasing, it will continue to pick up the
> same row(s).
>
> That's a curious use case, maybe NiFi could handle other parts of it
> so you wouldn't need to update a single row in an external database
> table?
>
> Regards,
> Matt
>
> On Mon, Oct 29, 2018 at 12:36 PM l vic  wrote:
> >
> > What if have only one row and update the values in it? Will QDT fetch
> updates?
> > Thank you,
> > Victor
> >
> >
> > On Mon, Oct 29, 2018 at 11:54 AM Matt Burgess 
> wrote:
> >>
> >> You can use QueryDatabaseTable (QDT) for this, you'd set your
> >> "event_time" column as the "Maximum Value Column(s)" property in the
> >> processor. The first time QDT executes, it will fetch all the rows
> >> (since it has not seen event_time before), then it will keep track of
> >> the largest value of event_time. As new rows are added (with larger
> >> event_time values), QDT will only fetch the rows whose event_time is
> >> greater than the largest one it's seen. Then it updates its "largest
> >> seen value" and so on.
> >>
> >> GenerateTableFetch (GTF) is another option, it works in a similar
> >> fashion, except that it does not fetch the rows itself, instead it
> >> generates flow files containing SQL statements that you can send
> >> downstream to perhaps ExecuteSQL in order to actually fetch the rows.
> >> GTF is often used in place of QDT if you'll be fetching a large number
> >> of rows in each statement, as you can distribute the SQL flow files
> >> among the nodes in a cluster, to do the fetch in parallel.
> >>
> >> Regards,
> >> Matt
> >>
> >> On Mon, Oct 29, 2018 at 11:13 AM l vic  wrote:
> >> >
> >> > Hi,
> >> > i have "event_time" field in SQLite database that means epoch time
> for triggering of external event. What processor(s) can i use to implement
> schedule monitoring/ execution based on change in "event_time" value?
> >> > Thanks,
>

Re: schedule execution from relational database?

2018-10-29 Thread l vic

What if have only one row and update the values in it? Will QDT fetch
updates?
Thank you,
Victor


On Mon, Oct 29, 2018 at 11:54 AM Matt Burgess  wrote:

> You can use QueryDatabaseTable (QDT) for this, you'd set your
> "event_time" column as the "Maximum Value Column(s)" property in the
> processor. The first time QDT executes, it will fetch all the rows
> (since it has not seen event_time before), then it will keep track of
> the largest value of event_time. As new rows are added (with larger
> event_time values), QDT will only fetch the rows whose event_time is
> greater than the largest one it's seen. Then it updates its "largest
> seen value" and so on.
>
> GenerateTableFetch (GTF) is another option, it works in a similar
> fashion, except that it does not fetch the rows itself, instead it
> generates flow files containing SQL statements that you can send
> downstream to perhaps ExecuteSQL in order to actually fetch the rows.
> GTF is often used in place of QDT if you'll be fetching a large number
> of rows in each statement, as you can distribute the SQL flow files
> among the nodes in a cluster, to do the fetch in parallel.
>
> Regards,
> Matt
>
> On Mon, Oct 29, 2018 at 11:13 AM l vic  wrote:
> >
> > Hi,
> > i have "event_time" field in SQLite database that means epoch time for
> triggering of external event. What processor(s) can i use to implement
> schedule monitoring/ execution based on change in "event_time" value?
> > Thanks,
>

schedule execution from relational database?

2018-10-29 Thread l vic

Hi,
i have "event_time" field in SQLite database that means epoch time for
triggering of external event. What processor(s) can i use to implement
schedule monitoring/ execution based on change in "event_time" value?
Thanks,

Re: how to merge attributes?

2018-10-24 Thread l vic

Not sure I understand: "InvokeHttp" offers 5 relationships: "Failure", "No
Retry",  "Original", "Response", "Retry" with incoming FlowFile created for
"matched" relationship...Don't see any "Result" and "Response" flowfile
doesn't seem to contain input attributes. I work with ver. 1.7.0 of nifi
Thank you,
Victor

On Tue, Oct 23, 2018 at 10:16 PM Koji Kawamura 
wrote:

> Hello,
>
> InvokeHttp creates a new FlowFile for "Result" relationship from the
> incoming FlowFile. That means, the FlowFile for "Result" carries all
> attributes copied that the incoming one has. You just need to connect
> the "Result" relationship to ReplaceTest. "Original" can be
> auto-terminated in this case.
>
> Thanks,
> Koji
> On Wed, Oct 24, 2018 at 5:37 AM l vic  wrote:
> >
> > I have to create sql query from results of rest call (InvokeHttp ) and
> original request parameters...
> >
> > Any idea how i can merge attributes from both in single flowfile for
> passing it to "ReplaceText"? If I pass both "Original" and "Result" from
> "InvokeHttp" to "ReplaceText" I end up with one "valid" flowfile using
> attributes from both and one "junk" one with empty placeholders... If I try
> use "MergeContent" to  merge flows I end up only with junk only like the
> following
> >
> >
> > update joint_table set create_date = 1540321085127, filter = '',
> data_start_time =  + 1532016495 - , execute_date = 1532016495 + where id =
> '1749462c-ed2b-4a34-9332-3687a60c1e1c'
> >
> >
> > Thank you
>

how to merge attributes?

2018-10-23 Thread l vic

I have to create sql query from results of rest call (InvokeHttp ) and
original request parameters...

Any idea how i can merge attributes from both in single flowfile for
passing it to "ReplaceText"? If I pass both "Original" and "Result" from
"InvokeHttp" to "ReplaceText" I end up with one "valid" flowfile using
attributes from both and one "junk" one with empty placeholders... If I try
use "MergeContent" to  merge flows I end up only with junk only like the
following


update joint_table set create_date = 1540321085127, filter = '',
data_start_time =  + 1532016495 - , execute_date = 1532016495 + where id =
'1749462c-ed2b-4a34-9332-3687a60c1e1c'


Thank you

Re: upload template with REST problem

2018-09-27 Thread l vic

*Nice! Any idea how i can configure it programmatically? eg If i have some
template that consumes Kafka messages, *
*how can I create instance of processor group with the name of topic and
list of kafka brokers?*



On Thu, Sep 27, 2018 at 3:57 AM Daniel Chaffelson 
wrote:

> If you are looking to work with Templates programmatically, then I suggest
> you look at my Python client NiPyApi -
> https://github.com/Chaffelson/nipyapi
>
> There is a module for doing all the usual template commands:
>
> https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nipyapi.html#module-nipyapi.templates
> And modules for doing most other things, including working with
> NiFi-Registry.
>
> This way you do not have to worry about the mechanics of understanding how
> the NiFi REST Api works, you can just get on with your automation.
>
> On Wed, Sep 26, 2018 at 10:52 PM Andy LoPresto 
> wrote:
>
>> I would recommend you perform the actions you want to replicate in your
>> browser using the UI, and have the Developer Tools window open at the same
>> time. You can monitor all the requests to the API that way, and even copy
>> the request as a curl command, etc. This should make it much easier for you
>> to build the API requests you want.
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Sep 26, 2018, at 1:20 PM, l vic  wrote:
>>
>> Any idea how to actually instantiate the template after upload?
>>
>> On Thu, Sep 20, 2018 at 12:53 PM Bryan Bende  wrote:
>>
>>> The REST API documentation [1] for the method you are using says:
>>>
>>> NameLocation  Type  Description
>>> id   pathstring The process group id.
>>>
>>> [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>>> On Thu, Sep 20, 2018 at 12:49 PM l vic  wrote:
>>> >
>>> > yes, the "id" must actually be "groupId", then it works... As usual,
>>> Nifi documentation is vague or misleading
>>> >
>>> > On Thu, Sep 20, 2018 at 12:38 PM Bryan Bende  wrote:
>>> >>
>>> >> That means there is no process group with the id you specified.
>>> >> On Thu, Sep 20, 2018 at 12:36 PM l vic  wrote:
>>> >> >
>>> >> > yes, in this case i see another error:
>>> >> >
>>> >> > Unable to locate group with id
>>> '46f95fe5-3024-31d8--'.ma6219-mac:deploy
>>> >> >
>>> >> >
>>> >> > On Thu, Sep 20, 2018 at 9:58 AM Bryan Bende 
>>> wrote:
>>> >> >>
>>> >> >> The URL you specified is not the correct path, it is missing
>>> "process-groups".
>>> >> >>
>>> >> >>
>>> http://10.10.100.105:8080/nifi-api/process-groups/f2ae3d20-e105-37a1--/templates/upload
>>> >> >> On Wed, Sep 19, 2018 at 4:17 PM l vic  wrote:
>>> >> >> >
>>> >> >> > Hi,
>>> >> >> > I have template for the processing group "test-template.xml" with
>>> >> >> > id=f2ae3d20-e105-37a1--
>>> >> >> > that I am trying to upload from my mac
>>> >> >> >
>>> >> >> > curl -iv -F template=@test-template.xml -X POST
>>> http://10.10.100.105:8080/nifi-api/f2ae3d20-e105-37a1--/templates/upload
>>> >> >> >
>>> >> >> >
>>> >> >> > I see http error:
>>> >> >> >
>>> >> >> >
>>> >> >> > The specified resource could not be found.ma6219-mac:deploy
>>> >> >> >
>>> >> >> >
>>> >> >> > Any idea what the problem is with my command?
>>> >> >> >
>>> >> >> > Thanks ,
>>> >> >> >
>>> >> >> > -V
>>>
>>
>>

Re: upload template with REST problem

2018-09-26 Thread l vic

Any idea how to actually instantiate the template after upload?

On Thu, Sep 20, 2018 at 12:53 PM Bryan Bende  wrote:

> The REST API documentation [1] for the method you are using says:
>
> NameLocation  Type  Description
> id   pathstring The process group id.
>
> [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
> On Thu, Sep 20, 2018 at 12:49 PM l vic  wrote:
> >
> > yes, the "id" must actually be "groupId", then it works... As usual,
> Nifi documentation is vague or misleading
> >
> > On Thu, Sep 20, 2018 at 12:38 PM Bryan Bende  wrote:
> >>
> >> That means there is no process group with the id you specified.
> >> On Thu, Sep 20, 2018 at 12:36 PM l vic  wrote:
> >> >
> >> > yes, in this case i see another error:
> >> >
> >> > Unable to locate group with id
> '46f95fe5-3024-31d8--'.ma6219-mac:deploy
> >> >
> >> >
> >> > On Thu, Sep 20, 2018 at 9:58 AM Bryan Bende  wrote:
> >> >>
> >> >> The URL you specified is not the correct path, it is missing
> "process-groups".
> >> >>
> >> >>
> http://10.10.100.105:8080/nifi-api/process-groups/f2ae3d20-e105-37a1--/templates/upload
> >> >> On Wed, Sep 19, 2018 at 4:17 PM l vic  wrote:
> >> >> >
> >> >> > Hi,
> >> >> > I have template for the processing group "test-template.xml" with
> >> >> > id=f2ae3d20-e105-37a1--
> >> >> > that I am trying to upload from my mac
> >> >> >
> >> >> > curl -iv -F template=@test-template.xml -X POST
> http://10.10.100.105:8080/nifi-api/f2ae3d20-e105-37a1--/templates/upload
> >> >> >
> >> >> >
> >> >> > I see http error:
> >> >> >
> >> >> >
> >> >> > The specified resource could not be found.ma6219-mac:deploy
> >> >> >
> >> >> >
> >> >> > Any idea what the problem is with my command?
> >> >> >
> >> >> > Thanks ,
> >> >> >
> >> >> > -V
>

PutSQL jdbc error

2018-09-25 Thread l vic

I have PutSQL on host 10.10.105.214 trying to execute query on phoenix
query server on the host 10.10.105.202 and i see the following error:

Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC
driver of class 'org.apache.phoenix.jdbc.PhoenixDriver' for connect URL
'jdbc:10.10.105.202:2181:/hbase-unsecure
'
at
org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1452)
at
org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371)
at
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at
org.apache.nifi.dbcp.DBCPConnectionPool.getConnection(DBCPConnectionPool.java:272)
... 22 common frames omitted
Caused by: java.sql.SQLException: No suitable driver
at java.sql.DriverManager.getDriver(DriverManager.java:315)

For DBCPConnectionPool ( Enabled- no problem) I have the following settings:
jdbc:10.10.105.202:2181:/hbase-unsecure
org.apache.phoenix.jdbc.PhoenixDriver
file:usr/hdf/current/phoenix/phoenix-client.jar
Should the driver (/usr/hdf/current/phoenix/phoenix-client.jar) be on NiFi
or Phoenix host?
Seems as the problem is in the driver itself ( connection string works):
how can i get to the particular cause? Stack trace doesn't really help...
Any idea where the problem can be?
Thanks,

Re: upload template with REST problem

2018-09-21 Thread l vic

I can only see "id" in swagger doc, like the image attached, if i am
missing more verbose part I apologize

On Thu, Sep 20, 2018 at 12:53 PM Bryan Bende  wrote:

> The REST API documentation [1] for the method you are using says:
>
> NameLocation  Type  Description
> id   pathstring The process group id.
>
> [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
> On Thu, Sep 20, 2018 at 12:49 PM l vic  wrote:
> >
> > yes, the "id" must actually be "groupId", then it works... As usual,
> Nifi documentation is vague or misleading
> >
> > On Thu, Sep 20, 2018 at 12:38 PM Bryan Bende  wrote:
> >>
> >> That means there is no process group with the id you specified.
> >> On Thu, Sep 20, 2018 at 12:36 PM l vic  wrote:
> >> >
> >> > yes, in this case i see another error:
> >> >
> >> > Unable to locate group with id
> '46f95fe5-3024-31d8--'.ma6219-mac:deploy
> >> >
> >> >
> >> > On Thu, Sep 20, 2018 at 9:58 AM Bryan Bende  wrote:
> >> >>
> >> >> The URL you specified is not the correct path, it is missing
> "process-groups".
> >> >>
> >> >>
> http://10.10.100.105:8080/nifi-api/process-groups/f2ae3d20-e105-37a1--/templates/upload
> >> >> On Wed, Sep 19, 2018 at 4:17 PM l vic  wrote:
> >> >> >
> >> >> > Hi,
> >> >> > I have template for the processing group "test-template.xml" with
> >> >> > id=f2ae3d20-e105-37a1--
> >> >> > that I am trying to upload from my mac
> >> >> >
> >> >> > curl -iv -F template=@test-template.xml -X POST
> http://10.10.100.105:8080/nifi-api/f2ae3d20-e105-37a1--/templates/upload
> >> >> >
> >> >> >
> >> >> > I see http error:
> >> >> >
> >> >> >
> >> >> > The specified resource could not be found.ma6219-mac:deploy
> >> >> >
> >> >> >
> >> >> > Any idea what the problem is with my command?
> >> >> >
> >> >> > Thanks ,
> >> >> >
> >> >> > -V
>

Re: upload template with REST problem

2018-09-20 Thread l vic

yes, the "id" must actually be "groupId", then it works... As usual, Nifi
documentation is vague or misleading

On Thu, Sep 20, 2018 at 12:38 PM Bryan Bende  wrote:

> That means there is no process group with the id you specified.
> On Thu, Sep 20, 2018 at 12:36 PM l vic  wrote:
> >
> > yes, in this case i see another error:
> >
> > Unable to locate group with id
> '46f95fe5-3024-31d8--'.ma6219-mac:deploy
> >
> >
> > On Thu, Sep 20, 2018 at 9:58 AM Bryan Bende  wrote:
> >>
> >> The URL you specified is not the correct path, it is missing
> "process-groups".
> >>
> >>
> http://10.10.100.105:8080/nifi-api/process-groups/f2ae3d20-e105-37a1--/templates/upload
> >> On Wed, Sep 19, 2018 at 4:17 PM l vic  wrote:
> >> >
> >> > Hi,
> >> > I have template for the processing group "test-template.xml" with
> >> > id=f2ae3d20-e105-37a1--
> >> > that I am trying to upload from my mac
> >> >
> >> > curl -iv -F template=@test-template.xml -X POST
> http://10.10.100.105:8080/nifi-api/f2ae3d20-e105-37a1--/templates/upload
> >> >
> >> >
> >> > I see http error:
> >> >
> >> >
> >> > The specified resource could not be found.ma6219-mac:deploy
> >> >
> >> >
> >> > Any idea what the problem is with my command?
> >> >
> >> > Thanks ,
> >> >
> >> > -V
>

Re: upload template with REST problem

2018-09-20 Thread l vic

yes, in this case i see another error:

Unable to locate group with id
'46f95fe5-3024-31d8--'.ma6219-mac:deploy

On Thu, Sep 20, 2018 at 9:58 AM Bryan Bende  wrote:

> The URL you specified is not the correct path, it is missing
> "process-groups".
>
>
> http://10.10.100.105:8080/nifi-api/process-groups/f2ae3d20-e105-37a1--/templates/upload
> On Wed, Sep 19, 2018 at 4:17 PM l vic  wrote:
> >
> > Hi,
> > I have template for the processing group "test-template.xml" with
> > id=f2ae3d20-e105-37a1--
> > that I am trying to upload from my mac
> >
> > curl -iv -F template=@test-template.xml -X POST
> http://10.10.100.105:8080/nifi-api/f2ae3d20-e105-37a1--/templates/upload
> >
> >
> > I see http error:
> >
> >
> > The specified resource could not be found.ma6219-mac:deploy
> >
> >
> > Any idea what the problem is with my command?
> >
> > Thanks ,
> >
> > -V
>

upload template with REST problem

2018-09-19 Thread l vic

Hi,
I have template for the processing group "test-template.xml" with
id=f2ae3d20-e105-37a1--
that I am trying to upload from my mac

curl -iv -F template=@test-template.xml -X POST
http://10.10.100.105:8080/nifi-api/f2ae3d20-e105-37a1--/templates/upload


I see http error:


*The specified resource could not be found.ma6219-mac:deploy*


Any idea what the problem is with my command?

Thanks ,

-V

JsonSplit array into individual records?

2018-09-06 Thread l vic

I have array:
"mylist": [{
  "id": 10,
  "info": "2am-3am"
},
{
"id": 11,
"info": "3AM-4AM"
},
{
"id": 12,
"info": "4am-5am"
}]
How can i use combination of SplitJson / JsonPath to split it into multiple
flows per array element?
Thanks,
V

Re: saving record into different tables?

2018-09-05 Thread l vic

Ok, I've got "split" working but that still leaves me with the dilemma: i
have to write multiple rows for "mylist" into the same table... Note that I
am working with db (Phoenix Query Server)  that doesn't support "batching"
of multiple records in one insert as mysql. Any advise?


On Wed, Sep 5, 2018 at 10:04 PM Matt Burgess  wrote:

> V,
>
> Perhaps ironically but for the greater good, NiFi (IMHO, with record
> processors) performs better when you don't touch the content but just
> describe what you want from it. So in your case you could send the
> content to two different branches, one could be a PutDatabaseRecord
> with a reader schema that ignores the "mylist" element, and the other
> could use Jolt / SplitRecord / ForkRecord to "hoist" the fields from
> the "mylist" element into a root array/RecordSet, then use
> PutDatabaseRecord to process it. Then you don't need a "split" or "get
> into attribute" or any other operation that isn't in line with your
> business logic.
>
> Regards,
> Matt
>
> On Fri, Aug 31, 2018 at 7:34 PM l vic  wrote:
> >
> > Hi,
> > I have a json record that contains array "mylst":
> > {
> > "id": 0,
> > "name": "Root",
> >  "mylist": [{
> >   "id": 10,
> >   "info": "2am-3am"
> > },
> > {
> > "id": 11,
> > "info": "3AM-4AM"
> > },
> > {
> > "id": 12,
> > "info": "4am-5am"
> > }]
> >
> > }
> > I have to save root data into one db table and array into another... Can
> someone recommend an approach to "splitting" of record for 2 different
> database writers?
> > Thank you,
> > V
>

PutSQL with multiple upserts?

2018-09-05 Thread l vic

I need to save data with "root" object to be saved into "root" table and
array "mylist" of contained objects to be saved into "mylist" table:
{
"id": 3,
"name": "ROOT",
"mylist": [{
"id": 10,
"info": "2am-3am"
},
{
"id": 11,
"info": "3AM-4AM"
},
{
"id": 12,
"info": "4am-5am"
}]

}

 so I have to perform single "upsert" into "root" table and multiple
"upsert"s into "mylist" table from one flow... Does PutSQL support this?
Thank you

parsing json array in ExecuteScript?

2018-09-05 Thread l vic

I have json array attribute:
mylist
:[{"id":10,"info":"2am-3am"},{"id":11,"info":"3AM-4AM"},{"id":12,"info":"4am-5am"}]
How can I parse it to name/value pairs in groovy script?
Thanks,

Re: how to copy root fields in Jolt transform?

2018-09-04 Thread l vic

Thank you, works on the Jolt demo site. With real Nifi it doesn't. I am
trying to print those values in ExecuteScript with Groovy script iterating
flowfile attributets:
import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets
import org.apache.nifi.processor.io.StreamCallback

def flowFile = session.get()
def file1 = new File('/home/me/groovy/attributes.txt')
file1.write '##\n'
flowFile.getAttributes().each { key,value ->
  file1.append(key+' :')
  file1.append(value + '\n')
}
session.transfer(flowFile,REL_SUCCESS)
None of them show up. Is there some way to read them from ExecuteScript?

On Tue, Sep 4, 2018 at 12:56 PM Matt Burgess  wrote:

> Add the following to the end of your shift spec:
>
> "*": "&"
>
> This (at the root) matches any key not already matched and puts it in
> the output at the same location.
>
> Regards,
> Matt
>
> On Tue, Sep 4, 2018 at 11:39 AM l vic  wrote:
> >
> > Hi,
> > I want to "flatten" the following object:
> > {
> > "id": 0,
> > "name": "Root",
> >  "mylist": [{
> >   "id": 10,
> >   "info": "2am-3am"
> > },
> > {
> > "id": 11,
> > "info": "3AM-4AM"
> > },
> > {
> > "id": 12,
> > "info": "4am-5am"
> > }]
> > }
> > I figured how to "flatten" array, but "root" values are gone in
> transformed output:
> > {
> >"mylist-0-id":10,
> >" mylist-0-info":"2am-3am",
> > 
> > }
> > How can I retain "root" fields so that transformed output would like
> like one below?
> > {
> >  "id": 0,
> > "name": "Root",
> >  "mylist-0-id":10,
> >" mylist-0-info":"2am-3am",
> > 
> > }
> >
> >
>

Jolt doesn't work?

2018-09-04 Thread l vic

I have a json record that contains array "mylst":
{
"id": 0,
"name": "Root",
 "mylist": [{
  "id": 10,
  "info": "2am-3am"
},
{
"id": 11,
"info": "3AM-4AM"
}
   ]
}
I tested Jolt  spec:

[



  {

"operation": "shift",

"spec": {

  "mylist": {

"*": {

  "id": "mylist-&1-id",

  "info": "mylist-&1-info"

}

  },

  "*": "&"

}

  }

]
to transform it into "flat" version:
{
 "id": 0,
"name": "Root",
 "mylist-0-id":10,
   " mylist-0-info":"2am-3am",
   "mylist-1-id": 11,
   "mylist-1-info": "3AM-4AM"
}
However, when I put it all together to print into the file, only "root"
properties are printed:
GetFile->JoltTransformJson
I have the following exception in Jolt:
java.lang.NullPointerException: null
at java.util.ArrayList.addAll(ArrayList.java:581)
at com.bazaarvoice.jolt.defaultr.Key.applyChildren(Key.java:158)
at com.bazaarvoice.jolt.Defaultr.transform(Defaultr.java:242)
at
org.apache.nifi.processors.standard.util.jolt.TransformUtils.transform(TransformUtils.java:30)
at
org.apache.nifi.processors.standard.JoltTransformJSON.onTrigger(JoltTransformJSON.java:277)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-09-04 16:10:55,008 INFO [Flow Service Tasks Thread-1]
o.a.nifi.controller.StandardFlowService Saved flow controller
org.apache.nifi.controller.FlowController@4dd21cb1 // Another save pending
= false

Any idea what I do wrong?

how to copy root fields in Jolt transform?

2018-09-04 Thread l vic

Hi,
I want to "flatten" the following object:
{
"id": 0,
"name": "Root",
 "mylist": [{
  "id": 10,
  "info": "2am-3am"
},
{
"id": 11,
"info": "3AM-4AM"
},
{
"id": 12,
"info": "4am-5am"
}]
}
I figured how to "flatten" array, but "root" values are gone in transformed
output:
{
   "mylist-0-id":10,
   " mylist-0-info":"2am-3am",

}
How can I retain "root" fields so that transformed output would like like
one below?
{
 "id": 0,
"name": "Root",
 "mylist-0-id":10,
   " mylist-0-info":"2am-3am",

}

Re: saving record into different tables?

2018-09-03 Thread l vic

Would it be possible to have some example of such transform?


On Mon, Sep 3, 2018 at 4:18 AM DEHAY Aurelien 
wrote:

> Hello.
>
>
>
> I would flatten the records with jolt, with a kind of flag to know if it’s
> “root” data or not. Then I would extract the flag in an attribute with
> evaluatejson path, and at the end route with RouteOnAttribute.
>
>
>
> *Aurélien*
>
>
>
> *From:* l vic [mailto:lvic4...@gmail.com]
> *Sent:* samedi 1 septembre 2018 01:34
> *To:* users@nifi.apache.org
> *Subject:* saving record into different tables?
>
>
>
> Hi,
>
> I have a json record that contains array "mylst":
>
> {
>
> "id": 0,
>
> "name": "Root",
>
>  "mylist": [{
>
>   "id": 10,
>
>   "info": "2am-3am"
>
> },
>
> {
>
> "id": 11,
>
> "info": "3AM-4AM"
>
> },
>
> {
>
> "id": 12,
>
> "info": "4am-5am"
>
> }]
>
>
>
> }
>
> I have to save root data into one db table and array into another... Can
> someone recommend an approach to "splitting" of record for 2 different
> database writers?
>
> Thank you,
>
> V
>
> This electronic transmission (and any attachments thereto) is intended
> solely for the use of the addressee(s). It may contain confidential or
> legally privileged information. If you are not the intended recipient of
> this message, you must delete it immediately and notify the sender. Any
> unauthorized use or disclosure of this message is strictly prohibited.
> Faurecia does not guarantee the integrity of this transmission and shall
> therefore never be liable if the message is altered or falsified nor for
> any virus, interception or damage to your system.
>

saving record into different tables?

2018-08-31 Thread l vic

Hi,
I have a json record that contains array "mylst":
{
"id": 0,
"name": "Root",
 "mylist": [{
  "id": 10,
  "info": "2am-3am"
},
{
"id": 11,
"info": "3AM-4AM"
},
{
"id": 12,
"info": "4am-5am"
}]

}
I have to save root data into one db table and array into another... Can
someone recommend an approach to "splitting" of record for 2 different
database writers?
Thank you,
V

Re: Nifi RecordReader for Json?

2018-08-31 Thread l vic

What would be the right processor to use JsonTreeReader to get json
attributes as name/value pairs and to write them into the file?
Thanks again,

On Thu, Aug 30, 2018 at 2:09 PM Otto Fowler  wrote:

> What he said.
>
>
> On August 30, 2018 at 13:31:54, Joe Witt (joe.w...@gmail.com) wrote:
>
> Wow!
>
> My apologies for the really bad response I gave linking to the same
> article you mentioned. I should be more careful when
> reading/responding on the phone!
>
> Thanks
> On Thu, Aug 30, 2018 at 11:01 AM Matt Burgess 
> wrote:
> >
> > V,
> >
> > Currently NiFi does not support specifying a schema in JSONSchema
> > format, you'll want to convert that to an Avro schema for use in
> > JsonTreeReader. I don't know JSONSchema that well so I'm not sure if
> > that "stats" schema is supposed to be included in the outgoing object.
> > I ran it through a utility-under-development [1] and got the following
> > Avro schema out:
> >
> >
> {"type":"record","name":"record0","fields":[{"name":"create_date","type":"long"},{"name":"id","type":"string"}]}
>
> >
> > This doesn't add the "stats" record and specifies "create_date" as a
> > long versus a BigInteger. I think you might want to use an Avro
> > logical type of "decimal" [2] depending on what the value is in the
> > actual JSON object:
> >
> > {"type":"record","name":"record0","fields":[
> > {"name":"create_date","type": {"type": "bytes","logicalType":
> > "decimal","precision": 12,"scale": 0}},
> > {"name":"id","type":"string"}
> > ]}
> >
> > If you have a stats object present, this might work:
> >
> > {"type":"record","name":"record0","fields":[
> > {"name": "stats", "type" :
> >
> {"type":"record","name":"statsRecord","fields":[{"name":"id","type":"string"},{"name":"bin_qualifier","type":"string"}]}},
>
> > {"name":"create_date","type": {"type": "bytes","logicalType":
> > "decimal","precision": 12,"scale": 0}},
> > {"name":"id","type":"string"}
> > ]}
> >
> > I didn't validate or try these so there may be typos or other
> > (hopefully minor) mistakes.
> >
> > Regards,
> > Matt
> >
> > [1] https://github.com/fge/json-schema-avro
> > [2] https://avro.apache.org/docs/1.8.2/spec.html#Decimal
> >
> > On Thu, Aug 30, 2018 at 9:54 AM l vic  wrote:
> > >
> > > I have json file for the schema that looks like the following:
> > >
> > > {
> > > "$schema": "http://json-schema.org/draft-04/schema#;,
> > > "definitions": {
> > > "stats": {
> > > "type": "object",
> > > "additionalProperties": false,
> > > "properties": {
> > > "id": {
> > > "type": "string"
> > > },
> > > "bin_qualifier": {
> > > "type": "string"
> > > }
> > > }
> > > }
> > > },
> > > "additionalProperties": false,
> > > "description": "attributes",
> > > "type": "object",
> > > "properties": {
> > > "id": {
> > > "type": "string",
> > > "required": true,
> > > },
> > > "create_date": {
> > > "type": "integer",
> > > "javaType": "java.math.BigInteger",
> > > "required": true
> > > }
> > > }
> > > }
> > >
> > >
> > > How can I add this schema for JsonTreeReader?
> > >
> > > On Thu, Aug 30, 2018 at 9:02 AM Otto Fowler 
> wrote:
> > >>
> > >> The record readers are services, that processors use.
> > >> When you use a *Record* processor, you will have to select a Reader
> and a Writer Service, or create one ( which you can do through the UI ).
> > >> https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
> > >>
> > >>
> > >> On August 30, 2018 at 08:48:08, l vic (lvic4...@gmail.com) wrote:
> > >>
> > >> So, where's JsonTreeReader? I am on nifi-1.7.1-RC1 and i don't see it
> in the list of available processors...
> > >> Thanks,
> > >> V
> > >>
> > >> On Thu, Aug 30, 2018 at 5:31 AM Sivaprasanna <
> sivaprasanna...@gmail.com> wrote:
> > >>>
> > >>> Hi. Just like CSVRecordReader, we have record reader service for
> JSON. It's called JsonTreeReader. You can use AvroSchemaRegistry and
> provide an Avro schema (usually generated through InferAvroSchema
> processor) for your JSON. Refer:
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.7.1/org.apache.nifi.json.JsonTreeReader/index.html
> > >>>
> > >>> -
> > >>> Sivaprasanna
> > >>>
> > >>> On Thu, 30 Aug 2018 at 2:21 PM, l vic  wrote:
> > >>>>
> > >>>> I need to save two different json messages according to json
> schemas available for each to different relational database tables.
> > >>>> I saw this blog:
> > >>>> https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
> > >>>> with example using CSVRecordReader for csv->json transformation.
> > >>>> but what would be RecordReader for schema-based transformation from
> json? Is this a valid approach, or what would be best approach to solve
> this problem?
> > >>>> I am using: nifi-1.7.1-RC1...
> > >>>> Thank you,
>
>

Re: Nifi RecordReader for Json?

2018-08-30 Thread l vic

I have json file for the schema that looks like the following:

{
"$schema": "http://json-schema.org/draft-04/schema#;,
"definitions": {
"stats": {
"type": "object",
"additionalProperties": false,
"properties": {
"id": {
"type": "string"
},
"bin_qualifier": {
"type": "string"
}
}
}
},
"additionalProperties": false,
"description": "attributes",
"type": "object",
"properties": {
"id": {
"type": "string",
"required": true,
},
"create_date": {
"type": "integer",
"javaType": "java.math.BigInteger",
"required": true
}
}
}


How can I add this schema for JsonTreeReader?

On Thu, Aug 30, 2018 at 9:02 AM Otto Fowler  wrote:

> The record readers are services, that processors use.
> When you use a *Record* processor, you will have to select a Reader and a
> Writer Service, or create one ( which you can do through the UI ).
> https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
>
>
> On August 30, 2018 at 08:48:08, l vic (lvic4...@gmail.com) wrote:
>
> So, where's JsonTreeReader? I am on nifi-1.7.1-RC1 and i don't see it in
> the list of available processors...
> Thanks,
> V
>
> On Thu, Aug 30, 2018 at 5:31 AM Sivaprasanna 
> wrote:
>
>> Hi. Just like CSVRecordReader, we have record reader service for JSON.
>> It's called JsonTreeReader. You can use AvroSchemaRegistry and provide an
>> Avro schema (usually generated through InferAvroSchema processor) for your
>> JSON. Refer:
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.7.1/org.apache.nifi.json.JsonTreeReader/index.html
>>
>> -
>> Sivaprasanna
>>
>> On Thu, 30 Aug 2018 at 2:21 PM, l vic  wrote:
>>
>>> I need to save two different json messages according to json schemas
>>> available for each to different relational database tables.
>>> I saw this blog:
>>> https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
>>> with example using CSVRecordReader for csv->json transformation.
>>> but what would be RecordReader for schema-based transformation from
>>> json? Is this a valid approach, or what would be best approach to solve
>>> this problem?
>>> I am using: nifi-1.7.1-RC1...
>>> Thank you,
>>>
>>

Re: Nifi RecordReader for Json?

2018-08-30 Thread l vic

So, where's JsonTreeReader? I am on nifi-1.7.1-RC1 and i don't see it in
the list of available processors...
Thanks,
V

On Thu, Aug 30, 2018 at 5:31 AM Sivaprasanna 
wrote:

> Hi. Just like CSVRecordReader, we have record reader service for JSON.
> It's called JsonTreeReader. You can use AvroSchemaRegistry and provide an
> Avro schema (usually generated through InferAvroSchema processor) for your
> JSON. Refer:
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.7.1/org.apache.nifi.json.JsonTreeReader/index.html
>
> -
> Sivaprasanna
>
> On Thu, 30 Aug 2018 at 2:21 PM, l vic  wrote:
>
>> I need to save two different json messages according to json schemas
>> available for each to different relational database tables.
>> I saw this blog:
>> https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
>> with example using CSVRecordReader for csv->json transformation.
>> but what would be RecordReader for schema-based transformation from json?
>> Is this a valid approach, or what would be best approach to solve this
>> problem?
>> I am using: nifi-1.7.1-RC1...
>> Thank you,
>>
>

Nifi RecordReader for Json?

2018-08-30 Thread l vic

I need to save two different json messages according to json schemas
available for each to different relational database tables.
I saw this blog:
https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
with example using CSVRecordReader for csv->json transformation.
but what would be RecordReader for schema-based transformation from json?
Is this a valid approach, or what would be best approach to solve this
problem?
I am using: nifi-1.7.1-RC1...
Thank you,

Re: configuration variables?

2018-08-07 Thread l vic

What's about level higher - common variables for several process groups?
Thanks.

On Mon, Aug 6, 2018 at 7:06 PM, Charlie Meyer <
charlie.me...@civitaslearning.com> wrote:

> You might want to take a look at utilizing the variable registry for this:
> https://community.hortonworks.com/articles/155823/introduction-into-
> process-group-variables.html
>
> On Mon, Aug 6, 2018 at 4:07 PM lvic4...@gmail.com 
> wrote:
>
>> I have several processors in my flow using the same database url
>> parameter. Is there any method in Nifi to read configuration parameter(s)
>> from Zookeeper, or similar?
>> Thank you,
>>
>

75 matches

Mail list logo