RedisConnectionPoolService integration with Redis in AWS

[Vijay Chhipa]

Hey NiFi users,

Has anyone successfully set up a RedisConnectionPoolService to be used by 
Put/FetchDistributedMapCache to talk to Redis in AWS ?

What does your Redis configuration look like? 

AWS has a few options for Redis and would like to know how you made it work.

Upon searching found a couple of posts but none directly related to AWS 
https://www.cdata.com/kb/tech/redis-jdbc-apache-nifi.rst 

https://bryanbende.com/development/2017/10/09/apache-nifi-redis-integration

Thanks
Vijay 

Re: ECMAScript support missing from ExecuteScript

[Vijay Chhipa]

Ah, that makes perfect sense, we did upgrade to JDK 17 from the earlier JDK 11. 
I will try to add it explicitly until the next release is out. 

Thank you Matt. 


> On Jan 16, 2023, at 5:49 PM, Matt Burgess  wrote:
> 
> Did you upgrade the version of Java when you upgraded NiFi? Later versions of 
> Java don’t include the Nashorn (ECMAScript) library, but I believe we added 
> it explicitly, perhaps for the 1.20 release (not at computer right now)
> 
> Sent from my iPhone
> 
>> On Jan 16, 2023, at 6:28 PM, Vijay Chhipa  wrote:
>> 
>> Hi All, 
>> 
>> Has ECMAScript been removed from one of the scripting languages supported in 
>> the ExecuteScript processor ?
>> 
>> I had a dataflow use ECMAScript in ExecuteScript processor in NiFi 1.16.1 
>> and after the upgrade to 1.19.1 the processor became invalid. 
>> 
>> Please see the screenshot below 
>> 
>> 
>> 
>> 
>> The allowable values in the Script Engine properties are now these
>> 
>> 
>> 
>> 
>> 
>> I didn’t see any mention of ECMAScript removal in any release notes. 
>> 
>> Wondering if this was an oversight or should I choose a different scripting 
>> language for my dataflow. 
>> 
>> 
>> Thank you. 
>> Vijay
>> 
>> 

Re: Expected mergerecord performance

[Vijay Chhipa]

Hi Richard
Have you tried JoltTransformJSON or JoltTransformRecord

I believe you should be able to do this

Quick start here:  
https://community.cloudera.com/t5/Community-Articles/Jolt-quick-reference-for-Nifi-Jolt-Processors/ta-p/244350


> On Dec 20, 2022, at 4:13 AM, Richard Beare  wrote:
> 
> Hi Everyone,
> Still struggling to fix this issue and may need to try some different things.
> 
> What is the recommended way of transforming a record structure? At the moment 
> I have a groovy script doing this but the downstream processing is very slow, 
> as discussed in the preceding thread.
> 
> The transformation is very simple - starting structure is:
> 
> {
>  "result" : {
> "text" : " document text",
>   "metadata" : {
>  "X-TIKA:Parsed-By": [
>  "org.apache.tika.parser.pdf.PDFParser"
>  ],
> "X-OCR-Applied" : true,
> "dcterms:created": "2018;07-24T15:04:51Z",
> "Content-Type" : "application/pdf",
> "Page-Count" : 2,
>   },
> "success" : true,
> "timestamp" :  "2022-12-20T09:02:27.902Z",
> "processingElapsedTime" : 6
> }
> }
> 
> 
> final structure is':
> 
> [ {
> "doc_id" : 58,
> "doc_text" : "   ",
> "processing_timestamp" : "2022-12-20T09:02:27.902Z",
> "metadata_x_ocr_applies" : true,
> "metadata_x_parsed_by" : "org.apache.tika.parser.pdf.PDFParser",
> "metadata_content_type" : "application/pdf",
> "metadata_page_count" : 1
> "metadata_creation_date": null,
> "metadata_last_modified: nill
> }]
> 
> So a kind of flattening of the structure. Is there a processor I should be 
> using to do this instead of a groovy script?
> 
> Thanks
> 
> On Wed, Dec 14, 2022 at 7:57 AM Richard Beare  > wrote:
> Any thoughts on this? Are there some extra steps required when creating an 
> avro file from a user defined schema?
> 
> On Thu, Dec 8, 2022 at 2:56 PM Richard Beare  > wrote:
> Here's another result that I think suggests there's something wrong with the 
> avro files created by the groovy script, although I can't see what the 
> problem might be.
> 
> The test is as follows. Output of the groovy script creating avro files is 
> passed to convertrecord, configured with an avro reader and json writer. This 
> is slow. The json output is then converted back to avro with another 
> convertrecord processor, configured with a jsontreereader and an avro writer 
> - this is fast, instantly emptying the queue. The result of that is fed into 
> the previously problematic merge processor which works exactly as expected, 
> producing flowfiles with 100 records each.
> 
> The difference I can see between the two flow files is the way in which the 
> schema is specified. Perhaps some extras are required in the groovy file to 
> set that up?
> 
> The slow one has:
> 
> {"type":"record","name":"document", "fields":[{
> 
> The fast one 
> {"type":"record","name":"nifiRecord","namespace":"org.apache.nifi","fields":
> 
> 
> Initial characters are also slightly different.
> Slow one:
> 
> 000   O   b   j 001 002 026   a   v   r   o   .   s   c   h   e   m
> 020   a 346  \n   {   "   t   y   p   e   "   :   "   r   e   c   o
> 
> Fast one
> 
> 000   O   b   j 001 004 026   a   v   r   o   .   s   c   h   e   m
> 020   a 362  \b   {   "   t   y   p   e   "   :   "   r   e   c   o
> 
> 
> The groovy script is
> CogStack-NiFi/parse-tika-result-json-to-avro.groovy at master · 
> CogStack/CogStack-NiFi · GitHub 
> 
> 
> The schema is
> CogStack-NiFi/document.avsc at master · CogStack/CogStack-NiFi · GitHub 
> 
> 
> 
> On Thu, Dec 8, 2022 at 1:59 PM Richard Beare  > wrote:
> I'm diving into the convertrecord tests a bit deeper on the production server.
> 
> The first test case - 259 documents, total of 1M when in avro format in the 
> input queue to the convert record processor. These avro files were not 
> created by the groovy script - they start life as a database query and the 
> text field is in one of the columns. The convertrecord processor runs very 
> quickly - click start, press refresh and it is done. The avro ends up like 
> this:
> 
> [ {
>   "sampleid" : 1075,
>   "typeid" : 98,
>   "dct" : "2020-01-25T21:40:25.515Z",
>   "filename" : "__tmp/txt/mtsamples-type-98-sample-1075.txt",
>   "document" : "Text removed",
>   "docid" : "9"
> } ]
> 
> In the second test, where the text fields are extracted from pdf tika before 
> avro files are created by the groovy script (from the tika json output), the 
> total queue size for the 259 documents is larger - 1.77MB, and the 
> performance is very different - press start, click refresh and only two 
> flowfiles are processed.
> 
> [ {
>   "doc_id" : "70",
>   "doc_text" : "text removed",
>   "processing_timestamp" : "2022-12-07T23:09:52.354Z",
>   

Re: Migrate NiFi users

[Vijay Chhipa]

Hi HORNEMAN,

What I have done is the first option you mentioned, "move the users.xml and 
authorizations.xml files to the new instances and then add the instances 
manually” 
The reason of this is that the authorizations for instances are just a few 
while authorization for users could be a lot based on how granular your groups, 
and permissions are. 

Vijay


> On Nov 8, 2022, at 9:51 AM, Quentin HORNEMAN GUTTON 
>  wrote:
> 
> Hello
> 
> I am currently trying to migrate users and their rights from a NiFi 1.13.2 
> cluster to a NiFi 1.16.3 cluster on new servers. What are the steps to follow 
> to keep the old users with the new configuration ? 
> 
> Should I move the users.xml and authorizations.xml files to the new instances 
> and then add the instances manually ?
> 
> Or should I launch the new instances and manually add the old users?
> 
> Or is there a better solution?
> 
> Best regards,
> 
> HORNEMAN GUTTON Quentin

Flowfile disk space is not released from the content-repository until the entire dataflow is completed

[Vijay Chhipa]

Hi all, 

We have a use case where we list out the contents of a website and then 
download each item in the list and process it. 
What I expected is that when each item (a file) is downloaded, after processing 
is completed, and the flowfile is not in any of the queues the disk storage 
will be released. But what I see is the content-repo size continues to increase 
as the files are processed. If I pause the flow for several hours (over 24 
hours) the repo size stays at the increased level and does not go down. Only 
when I clear all the queues does the content-repo size goes down to the 
original size (before the flow started). 

I am not using provenance and have disabled it. 
Here is the relevant section of the properties file. 

I would have been okay with it but I need to process over 200K files each in 
size almost 1GB.   

What is holding reference to these processed flow files and how can I design 
the dataflow to not have the content repo filled up. 

nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowfile.repository.directory=/var/foo/bar/flowfile_repository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=false

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=1 MB
nifi.content.claim.max.flow.files=10
nifi.content.repository.directory.default=/var/foo/bar/content_repository
nifi.content.repository.archive.max.retention.period=6 hours
nifi.content.repository.archive.max.usage.percentage=40%
nifi.content.repository.archive.enabled=false
nifi.content.repository.always.sync=false
nifi.content.viewer.url=../nifi-content-viewer/

# Provenance Repository Properties
nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=

# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=/var/foo/bar/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false


nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, 
ProcessorID, Relationship

nifi.provenance.repository.indexed.attributes=

nifi.provenance.repository.index.shard.size=500 MB
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.concurrent.merge.threads=2

nifi.provenance.repository.warm.cache.frequency=1 hour
nifi.provenance.repository.buffer.size=10

Thanks
Vijay

smime.p7s
Description: S/MIME cryptographic signature

Re: Order of upgrading Nifi and Zookeeper clusters

[Vijay Chhipa]

Isha, 

When we migrated off of 1.9.1 we faced the same issue. You need to upgrade 
zookeeper to 3.5.x, better yet you should upgrade to 3.6+ as it allows the 
zookeeper nodes to be started in any order, otherwise they need to be started 
in the order of their id. 

Vijay




 


> On Jul 8, 2021, at 2:56 PM, Isha Lamboo  
> wrote:
> 
> Hi all,
>  
> I’m planning an upgrade of our NiFi 1.9.0 clusters with external Zookeeper 
> 3.4.6 clusters to NiFi 1.12 or 1.13.
>  
> After rolling out an upgrade of only the NiFi software in our test setup, I 
> found that NiFi 1.13.2 would not communicate correctly with the unchanged 
> Zookeeper cluster.
> Now I’m wondering if it is misconfiguration (ex. The zookeeper connection 
> string with the ports separate or combined as in 1.10+) or just 
> incompatibility.
>  
> So my question to those who have done this before: Should I upgrade Zookeeper 
> first (to 3.5.x), NiFi first (and fix config errors) or does it need to be 
> both of them at once?
>  
> As a side note, I haven’t found a compatibility matrix of NiFi and Zookeeper 
> versions, is this documented somewhere or do I just go with the embedded 
> Zookeeper version number of the NiFi release?
>  
> Regards,
> 
> Isha Lamboo



smime.p7s
Description: S/MIME cryptographic signature

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

[Vijay Chhipa]

Avinash, 

Did you try to tune Max Timer Driven Thread : 100

Set this value to  (number of cores per node) * (number of nodes) i.e. 48 *13 = 
624

https://docs.cloudera.com/cfm/2.1.1/nifi-tuning/topics/cfm-tuning-recommendations.html
 



HTH  



> On Jun 29, 2021, at 9:32 PM, Modepalli Venkata Avinash 
>  wrote:
> 
> Hi Joe,
>  
> Thanks for your feedback.
>  
> It’s deployed by using Hortonworks stack, so if we have to upgrade our NIFI 
> then we might have to upgrade entire stack. Also we have some integrations 
> done to NIFI using REST API along with some custom processors. 
> We have plans to upgrade down the lane, but not immediately due to above 
> challenges.
> We have almost 5-10M flow files in queues while bringing up NIFI. However, we 
> are observing this issue even with lesser number of flow files also.
>  
> Thanks & Regards,
> Avinash M V
>  
> From: Joe Witt mailto:joe.w...@gmail.com>> 
> Sent: 29 June 2021 17:15
> To: users@nifi.apache.org 
> Cc: Modepalli Venkata Avinash  >
> Subject: Re: Taking Huge Time for Connect all Nodes to NIFI cluster
>  
> CAUTION: This e-mail originated from outside of the organization. Do not 
> click links or open attachments unless you recognise the sender and know the 
> content is safe.
>  
> Hello
>  
> A cluster that size should be fine. We did make various improvement to 
> cluster behavior and startup times though.  What prevents you from moving to 
> 1.13?
>  
> How many flowfiles are in the repo when restarting is taking that long? 
>  
> thanks
>  
> On Tue, Jun 29, 2021 at 7:38 AM Joe Obernberger  > wrote:
> Impressive cluster size!  I do not have an answer for you, but could you 
> change your architecture so that instead of one large NiFi cluster you have 2 
> or 3 smaller clusters?  Very curious on the answer here as I have also 
> noticed UI slow-downs as the number of nodes increases.
> 
> -Joe
> 
> On 6/29/2021 3:45 AM, Modepalli Venkata Avinash wrote:
> Hi List,
>  
> We have 13 Nodes NIFI cluster in production & it’s taking huge time for 
> completing NIFI restart.
> According to our analysis, flow election & flow validation from other nodes 
> with coordinator is taking more time, approx. ~30 hours.
> Even after all 13 nodes gets connected, NIFI UI responds too slowly. Please 
> find below cluster details.
>  
> Apache NIFI Version : 1.9.0
> Flow.xml.gz size : 13MB (gz compressed)
> OS : RHEL 7.6
> JDK : jdk1.8.0_151
> GC : Default GC(Parallel GC) of JDK1.8 is in place. Commented out G1GC 
> because of Numerous bugs in JDK8 while using with WriteaHeadProvenance 
> Repository
> Min & Max Memory : 140GB
> Server Memory Per Node : 256GB
> CPU/CORE : 48
> Number of Nodes in Cluster : 13
> Max Timer Driven Thread : 100
> Running Processors Count : 12K
> Stopped Processors Count : 10K
> Disabled Processors Count : 25K
> Total Processor Count : 47K
>  
> We couldn’t find any abnormalities in app logs, bootstrap logs & GC logging. 
> Could you please share any input to identify & resolve this issue.
> Thanks for your help.
>  
> Thanks & Regards,
> Avinash M V
>  
>  
>  
> 
>  
> Virus-free. www.avg.com 
> 


smime.p7s
Description: S/MIME cryptographic signature

Re: Issue when extracting a large dataset into CSV files

[Vijay Chhipa]

Hi Vibhath, 

There is this  property on the processor 

Max Rows Per Flow File


Per docs:  If the value specified is zero, then all rows are returned in a 
single FlowFile.
Which seems to be what is happening in your case. 



> On May 9, 2021, at 12:11 PM, Vibhath Ileperuma  
> wrote:
> 
> executeSQLRecord



smime.p7s
Description: S/MIME cryptographic signature

Re: InvokeHTTP hangs after several successful calls

[Vijay Chhipa]

Jeanne, 

This is not a JDK issue. It is specific to HTTP. 
You can try to apply the fix for 
https://issues.apache.org/jira/browse/NIFI-8181 
 as a patch to your current 
version. 

HTH, 
Vijay

> On Apr 14, 2021, at 8:20 AM, jeanne-herndon 
>  wrote:
> 
> I am having this same issue and do not have access to log files.  Same
> pattern as the others.  I need to terminate the invokeHttp processor to
> release threads.   Is this a JDK issue or an HTTP2 issue?   I am not able to
> find any solutions that work.
> 
> 
> 
> 
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/



smime.p7s
Description: S/MIME cryptographic signature

Re: InvokeHTTP hangs after several successful calls

[Vijay Chhipa]

Mark, 

I changed the code to use HTTP 1.1 in InvokeHTTP, but it did not help. 

I looked into "Shutdown" part of this link 
https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/#okhttpclient
 
<https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/#okhttpclient>.
 After implementing this, the processor didn't hang. 

This defeats the purpose of the thread pool and connection reuse, but I was 
able to get past the issue. 

It is still a good idea to get  https://issues.apache.org/jira/browse/NIFI-8181 
<https://issues.apache.org/jira/browse/NIFI-8181>  implemented. 

Regards, 
Vijay



> On Jan 29, 2021, at 1:34 PM, Vijay Chhipa  wrote:
> 
> Joe, 
> 
> behavior is the same with the 1.12.1 version. 
> 
> Mark, 
> I will make the change to the OkHttp client and try. I do know that this flow 
> was working for a long time and there was possibly an upgrade done on the 
> source endpoint. Thank you for opening the Jira. 
> 
> Regards, 
> Vijay
> 
>> On Jan 28, 2021, at 4:37 PM, Mark Payne > <mailto:marka...@hotmail.com>> wrote:
>> 
>> Hey Vijay,
>> 
>> I’ve seen a few people lately running into issues with InvokeHTTP. The 
>> common thread for all of them is that they are hitting servers that are 
>> using HTTP 2. Reading threads from OkHttp (the underlying HTTP library that 
>> we use), I see that a lot of people are running into issues with it. In at 
>> least several of the cases, it ends up being the HTTP Server or a proxy/load 
>> balancer in the middle that is not properly supporting the HTTP 2 protocol. 
>> Unclear if there are HTTP 2.0 related bugs in OkHttp itself or not. In any 
>> case, users often comment that changing the protocol to HTTP 1.1 resolved 
>> the issue. I filed a Jira [1] to allow that to be exposed in the InvokeHTTP 
>> processor. Hopefully that will help.
>> 
>> Thanks
>> -Mark
>> 
>> [1] https://issues.apache.org/jira/browse/NIFI-8181 
>> <https://issues.apache.org/jira/browse/NIFI-8181>
>> 
>>> On Jan 28, 2021, at 4:46 PM, Joe Witt >> <mailto:joe.w...@gmail.com>> wrote:
>>> 
>>> The likely relevant thread is here
>>> 
>>> "Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms 
>>> elapsed=4403.88s tid=0x7fe44f16b000 nid=0xe103 in Object.wait()  
>>> [0x7fed4000]
>>>java.lang.Thread.State: WAITING (on object monitor)
>>> at java.lang.Object.wait(java.base@11.0.5/Native Method)
>>> - waiting on 
>>> at java.lang.Object.wait(java.base@11.0.5/Object.java:328)
>>> at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577)
>>> at 
>>> okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143)
>>> - waiting to re-lock in wait() <0x0007e007d818> (a 
>>> okhttp3.internal.http2.Http2Stream)
>>> at 
>>> okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
>>> at 
>>> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
>>> at 
>>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at 
>>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
>>> at 
>>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at 
>>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>> at 
>>> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
>>> at 
>>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at 
>>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>> at 
>>> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
>>> at 
>>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at 
>>> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
>>> at 
>>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>> at 
>>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
>>> at okhttp3.RealCall.execute(RealCall.java:69)
>>> at 
>>> org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793)
&

Re: InvokeHTTP hangs after several successful calls

[Vijay Chhipa]

Joe, 

behavior is the same with the 1.12.1 version. 

Mark, 
I will make the change to the OkHttp client and try. I do know that this flow 
was working for a long time and there was possibly an upgrade done on the 
source endpoint. Thank you for opening the Jira. 

Regards, 
Vijay

> On Jan 28, 2021, at 4:37 PM, Mark Payne  wrote:
> 
> Hey Vijay,
> 
> I’ve seen a few people lately running into issues with InvokeHTTP. The common 
> thread for all of them is that they are hitting servers that are using HTTP 
> 2. Reading threads from OkHttp (the underlying HTTP library that we use), I 
> see that a lot of people are running into issues with it. In at least several 
> of the cases, it ends up being the HTTP Server or a proxy/load balancer in 
> the middle that is not properly supporting the HTTP 2 protocol. Unclear if 
> there are HTTP 2.0 related bugs in OkHttp itself or not. In any case, users 
> often comment that changing the protocol to HTTP 1.1 resolved the issue. I 
> filed a Jira [1] to allow that to be exposed in the InvokeHTTP processor. 
> Hopefully that will help.
> 
> Thanks
> -Mark
> 
> [1] https://issues.apache.org/jira/browse/NIFI-8181 
> <https://issues.apache.org/jira/browse/NIFI-8181>
> 
>> On Jan 28, 2021, at 4:46 PM, Joe Witt > <mailto:joe.w...@gmail.com>> wrote:
>> 
>> The likely relevant thread is here
>> 
>> "Timer-Driven Process Thread-9" #70 prio=5 os_prio=31 cpu=12025.30ms 
>> elapsed=4403.88s tid=0x7fe44f16b000 nid=0xe103 in Object.wait()  
>> [0x7fed4000]
>>java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(java.base@11.0.5/Native Method)
>> - waiting on 
>> at java.lang.Object.wait(java.base@11.0.5/Object.java:328)
>> at okhttp3.internal.http2.Http2Stream.waitForIo(Http2Stream.java:577)
>> at 
>> okhttp3.internal.http2.Http2Stream.takeResponseHeaders(Http2Stream.java:143)
>> - waiting to re-lock in wait() <0x0007e007d818> (a 
>> okhttp3.internal.http2.Http2Stream)
>> at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:120)
>> at 
>> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at 
>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at 
>> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at 
>> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at 
>> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
>> at okhttp3.RealCall.execute(RealCall.java:69)
>> at 
>> org.apache.nifi.processors.standard.InvokeHTTP.onTrigger(InvokeHTTP.java:793)
>> at 
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at 
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>> at 
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>> at 
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at 
>> java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.5/Executors.java:515)
>> at 
>> java.util.concurrent.FutureTask.runAndReset(java.base@11.0.5/FutureTask.java:305)
>> at 
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.5/ScheduledThreadPoolExecutor.java:305)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.5/ThreadPoolExecutor.java:1128)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.5/ThreadPoolExecutor.java:628)
>> at java

Re: PrometheusReportingTask Metrics

[Vijay Chhipa]

Pavel
What problem did you run into?

If it is pulling an empty result set, you just need to add the trailing slash.  
See the solution here: 
https://stackoverflow.com/questions/63549400/prometheusreportingtask-not-emitting-metrics-in-nifi


> On Oct 28, 2020, at 7:18 PM, Pavel S  wrote:
> 
> I don't have an answer to your question. However I just wanted to share 
> something related to this. We have tried using prometheus reporting task on 
> my project and I couldn't get it to work. I have ended up creating my own 
> application that generates prometheus metrics.
> 
> On Wed, Oct 21, 2020, 7:44 PM Ritch, David  > wrote:
> Is there a description of each of the metrics provided by the 
> PrometheusReportingTask available somewhere?
> 
> Thank you!
> 
> -- 
> David Ritch
> Principal Software Engineer
> WaveStrike/Analytics Capabilities Division 
>  
>  
> 300 Sentinel Dr. Ste. 210
> Annapolis Junction, MD 20701
> Email dri...@novetta.com 
> Office (443) 661-4810x1173
> Mobile (443) 718-9327



smime.p7s
Description: S/MIME cryptographic signature

Re: Acks from Kafka not received (PutKafka, PublishKafka)

[Vijay Chhipa]

James, 

Port 9092 is the only port you need. 
See this post: 
https://stackoverflow.com/questions/38531054/kafka-and-firewall-rules 


Check that the firewall rules allow bidirectional traffic on that port. 

This is my setup: 

Kafka Brokers : a-b-c123:9092,a-b-c124:9092
Security protocol : SSL
SASL Mechanism : GSSAPI
SSLConextService : StandardSSLContextService

I am on NiFi 1.10  and Kafka 2.x

HTH
Vijay



> On Oct 30, 2020, at 5:57 AM, James McMahon  wrote:
> 
> We are attempting to send flowfiles from a NiFi clustered configuration to 
> Kafka services on remote hosts. We are getting errors that indicate no acks 
> are received from Kafka.
> 
> When Kafka acks a PutKafka or a PublishKafka, what port does it use to reach 
> back to my nifi cluster node hosts, and what configuration param tells it 
> that?
> 
> I have proven that I can ping my remote Kafka host from each of my four nifi 
> cluster nodes. I have also been able to telnet from each of my cluster nodes 
> to the Kafka host on port 9092, which is the port named in the Known Brokers 
> configuration in my PutKafka attempt and in my PublishKafka attempt. I 
> suspect the comms failure is in the other direction, when Kafka attempts to 
> ack back to my nifi cluster nodes.
> 
> My Known Brokers list appears like so (the name of my Kafka hosts changed to 
> a generic):
> Known Brokers  a-b-c123:9092,a-b-c124:9092
> 
> I am running nifi version 1.11.
> 
> Can anyone recommend other stepps to debug this and get it working? Thank you.



smime.p7s
Description: S/MIME cryptographic signature

Re: 2 Node Cluster High CPU usage

[Vijay Chhipa]

Cenk, 

Do you see anything interesting in the nifi-app.log or the nifi-bootstrap.log 
on the 2nd node?
You can also set the logging level to DEBUG on the second node to get a better 
idea. 

Vijay


> On Sep 11, 2020, at 11:58 AM, Cenk Aktas  wrote:
> 
> Hello all,
>  
> We have a 2 node cluster setup. Nearly all processors are cron scheduled and 
> configured to run on the primary node.
> Yes, we are utilizing 2nd node as a fail-over host. There is a weird CPU and 
> Memory usage on connected, not the primary node.
> Somehow 2nd node, which has nothing running, just checking primary node 
> status and hogging CPU and Memory.
> Any idea why? Is there something we miss configure?
>  
> Thanks in advance. 



smime.p7s
Description: S/MIME cryptographic signature

Re: Securing NiFI behind a proxy (NGINX).

[Vijay Chhipa]

Scotty, 

When you turn an unsecured cluster to secured one, authentication is turned on. 
You have to configure one of the authentication mechanism (Kerberos, LDAP, 
OIDC, ClientCertificates) in your nifi.properties. 


> On Sep 7, 2020, at 3:28 AM, scotty  wrote:
> 
> Hi,
> 
> 1. I had an usecured NiFi setup running: "Client > https > reverse  proxy >
> http > NiFi"
> 
> 2. I tried to secure it using the instructions at
> https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#tls_toolkit.
> Certificates were generate as follows:
> 
> a.  ./bin/tls-toolkit.sh standalone -n ''   
> 
> b. ./home/scotty/nifi-toolkit-1.9.1/bin/tls-toolkit.sh standalone -C
> 'CN=my_username,OU=NiFi'
> 
> 3. added CN=my_username,OU=NiFi in authorizers.xml  (users.xml and
> authorizations.xml appear correct)
> 
> 4. The response I'm getting at the NiFI URL is  'Insufficient Permissions:
> Unknown user with identity 'CN=localhost, OU=NIFI'. Contact the system
> administrator.'
> 
> 5. I'm getting the same message from whatever computer I try to access the
> UI from, whether it has the user certificate installed or not. So who's
> making the request to authenticate?
> 
> 6. nifi-user.log shows as follows:
> 
> INFO [NiFi Web Server-84] o.a.n.w.s.NiFiAuthenticationFilter Attempting
> request for (CN=localhost, OU=NIFI) GET https:// name>/nifi-api/flow/current-user (source ip: )
> 
> 2020-08-30 15:43:45,820 INFO [NiFi Web Server-84]
> o.a.n.w.s.NiFiAuthenticationFilter 
> Authentication success for CN=localhost, OU=NIFI
> 
> 2020-08-30 15:43:45,831 INFO [NiFi Web Server-84]
> o.a.n.w.a.c.AccessDeniedExceptionMapper identity[CN=localhost, OU=NIFI],
> groups [] does not have permission to access the requested resource. Unknown
> user with identity 'CN=localhost, OU=NIFI'. Returning Forbidden response
> 
> 7. My NiFI truststore.jks shows the following:
> 
> keystore contains 1 entry
> Alias name: nifi-cert
> Entry type: trustedCertEntry
> Owner: CN=localhost, OU=NIFI
> Issuer: CN=localhost, OU=NIFI
> 
> 
> 8. When I tried to use the X-ProxiedEntitiesChain header in my NGINX config,
> I got an "untrusted proxy" error when trying to access the NiFI UI.
> 
> Was I suppsed to secure NiFi without the reverse proxy and then setup the
> proxy through the NiFI UI (as discussed in the following)? 
> 
> "If NiFi is running securely, any proxy needs to be authorized to proxy user
> requests. These can be configured in the NiFi UI through the Global Menu.
> Once these permissions are in place, proxies can begin proxying user
> requests. The end user identity must be relayed in a HTTP header. For
> example, if the end user sent a request to the proxy, the proxy must
> authenticate the user. Following this the proxy can send the request to
> NiFi. In this request an HTTP header should be added as follows."
> 
> 
> 9. Was I supposed to generate a certificate for the proxy (as discussed in
> this Apache Knox tutorial)?
> https://risdenk.github.io/2018/03/18/apache-knox-proxying-apache-nifi.html
> 
> 
> Any guiidance appreciated.
> 
> Thanks.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/



smime.p7s
Description: S/MIME cryptographic signature

Re: In memoriam of Jeff Storck

[Vijay Chhipa]

I waited on pins and needles for the Java 11 support to come out. 
Little did I know that Jeff was the man behind it. 

Thanks for all of your efforts Jeff, because of you we were able to meet 
critical deadlines. 
RIP Jeff. 

Vijay


> On Jun 16, 2020, at 10:03 AM, Kevin Doran  wrote:
> 
> Jeff, you were a fantastic collaborator and friend. You will be dearly 
> missed. Thank you for all your contributions, and for all you’ve  shown and 
> taught me over the years. You’ve left behind a great legacy that will 
> continue to have a positive impact on the world for years to come, not just 
> your work but your way of working with others, and for that we are all 
> grateful. RIP.
> 
>> On Jun 15, 2020, at 3:30 PM, Pierre Villard > > wrote:
>> 
>> I can't say how much we will miss you Jeff. You were a great guy, always 
>> nice and helpful with everyone. You always went the extra mile to make 
>> things easier and more robust.
>> 
>> RIP Jeff
>> 
>> Le lun. 15 juin 2020 à 21:13, Jeremy Dyer > > a écrit :
>> This is shocking and heartbreaking news. Jeff was a great guy and will be 
>> deeply missed. 
>> 
>> The last time I saw Jeff in person was with Aldrin. We were eating at 
>> Bonchon chicken and he was mocking me for how little spice I could handle 
>> XD. I could always count on him for a good Dumb and Dumber reference and 
>> laugh. We also shared a common hatred for conference food.
>> 
>> RIP Jeff
>> 
>> On Mon, Jun 15, 2020 at 2:33 PM Joe Witt > > wrote:
>> You will be greatly missed.  Your impact to this community has been 
>> tremendous.  The items Andy summarizes were huge efforts that you drove over 
>> periods of many many months if not a year or more and they make NiFi so much 
>> more accessible than before.
>> 
>> RIP Jeff.
>> 
>> 
>> 
>> On Mon, Jun 15, 2020 at 11:24 AM Andy LoPresto > > wrote:
>> It is with a heavy heart that I write to the NiFi community today. Jeff 
>> Storck, a PMC member, committer, and genuine and helpful presence in the 
>> community, has passed away. 
>> 
>> I was lucky enough to know Jeff personally for many years, and his absence 
>> is a huge loss to all of us who did. Jeff was incredibly intelligent, but 
>> also kind and willing to share his experience with everyone. Whether playing 
>> volleyball (I am nowhere near as good but he humored me), discussing the 
>> best ramen and sushi spots, or evaluating a new exercise regime, Jeff 
>> brought passion to everything. A number of us are sharing stories of our 
>> favorite times with Jeff, and I am touched by how many people have a memory 
>> of Jeff reaching out and patiently helping them when they were new or 
>> struggling with a task. 
>> 
>> While other colleagues would happily transition to any topic _but_ work when 
>> we went to a nearby brewery at the end of a long day, Jeff would sit down 
>> next to me and say with a smile, "Ok Andy, work's done, now we can _really_ 
>> talk about Groovy unit testing." He never shied away from expressing his 
>> perspective and stood on conviction, but he was also open and genuinely 
>> wanted to hear other views to expand his mind. 
>> 
>> If you come across a Spock test in the NiFi codebase, that was most likely 
>> Jeff's work. He was intimately involved in much of the most challenging code 
>> - especially Kerberos integration, making the difficult but critical 
>> processes easier for our users. Anyone running NiFi on Java 11 should thank 
>> Jeff, as that was a labor of love, pushing against the headwinds of so many 
>> compatibility issues and language changes. The ease with which NiFi runs on 
>> multiple versions and platforms belies the immense amount of effort and 
>> dedication that he put into making this happen. 
>> 
>> There are so many aspects to Jeff that a note like this could never capture, 
>> but one that stands above the rest to me is Jeff's passion for learning and 
>> growth. He devoted himself to doing the best he could and constantly 
>> improving that. That is a noble philosophy that I know I will remember and 
>> admire moving forward. I’ve already started learning Kotlin because of 
>> Jeff’s enthusiasm and encouragement. 
>> 
>> Jeff’s family has created a GoFundMe page [1] and there they describe their 
>> intent to celebrate his life. I think that message is very positive and 
>> uplifting. To anyone wondering how they can honor Jeff's legacy, I suggest 
>> offering a helping hand to someone who needs it. Something as simple as 
>> responding to an extra "newbie" mailing list question at the end of a long 
>> day, or taking on a challenging task because your neighbor has their plate 
>> full. That's how Jeff lived, and he made the world a better place. 
>> 
>> 
>> 
>> Andy
>> 
>> [1] https://www.gofundme.com/f/in-memory-of-the-awesome-jeff-storck 
>> 
>> 
>> Andy 

NIFI-3065 Fine-grained logging in multi-tenant context

[Vijay Chhipa]

Hi Team, 

This is an old issue from 2019:  NIFI-3065  Fine-grained logging in 
multi-tenant context  : 
http://mail-archives.apache.org/mod_mbox/nifi-issues/201612.mbox/%3cjira.13021713.1479459766000.583688.1482431698...@atlassian.jira%3E

This is very useful feature. 

Do we know if there is any traction on getting this implemented ?

Thank you. 

Vijay
 



smime.p7s
Description: S/MIME cryptographic signature

Re: invokehttp timeout

[Vijay Chhipa]

Michael, 

Just to rule out timeouts set along the network hops, can you use cURL or 
Postman to ensure that you can make this call successfully from the host that's 
running NiFi?

Vijay


> On Oct 30, 2019, at 1:52 PM, Michael Di Domenico  
> wrote:
> 
> i'm trying to use invokehttp to post a file to a webserver.  for
> smaller files this seems to work just fine, but i try to send a larger
> file, i get snagged with this
> 
> 2019-10-30 14:47:59,188 ERROR [Timer-Driven Process Thread-10]
> o.a.nifi.processors.standard.InvokeHTTP
> InvokeHTTP[id=a2046302-016a-1000-940e-07f6f5992610] Routing to Failure
> due to exception:
> org.apache.nifi.processor.exception.ProcessException: IOException
> thrown from InvokeHTTP[id=a2046302-016a-1000-940e-07f6f5992610]:
> java.net.SocketTimeoutException: timeout:
> org.apache.nifi.processor.exception.ProcessException: IOException
> thrown from InvokeHTTP[id=a2046302-016a-1000-940e-07f6f5992610]:
> java.net.SocketTimeoutException: timeout
> org.apache.nifi.processor.exception.ProcessException: IOException
> thrown from InvokeHTTP[id=a2046302-016a-1000-940e-07f6f5992610]:
> java.net.SocketTimeoutException: timeout
> 
> i've lengthened the connect and read timeout in the properties to 1min
> and 1hrs respectively, but it doesn't seem to help.
> 
> is there a setting i'm missing?  I also semi-related, i need to ensure
> i get response from the webserver.  however, the web server might take
> up 30 mins to respond.  will invokehttp hold the connection open, if
> not is there a timeout setting somewhere?



smime.p7s
Description: S/MIME cryptographic signature

Re: InvokeHTTP with dynamic query parameters based on previous request response

[Vijay Chhipa]

Devraj, 

You will need to parse the response JSON produced by the first InvokeHTTP. You 
can do extract the timestamp you want using the EL syntax. 
If you only care when you made the previous request and want to store that 
timestamp somewhere, you can do so by adding a new attribute in the flowfiles 
attribute map. you can then use this value in the next InvokeHTTP cal as a 
value for 'since'

Vijay


> On Oct 10, 2019, at 10:42 AM, dkommineni  wrote:
> 
> Hi,
> 
> I am planning to pull data from github API URL 
> https://api.github.com/repos/kubernetes/kubernetes/issues?since=2018-01-01T00:00:00Z
> 
>   
> using InvokeHTTP processor. Once the processor fetch data, I would like to
> query the github API again with new timestamp based on the previous request
> results like 
> https://api.github.com/repos/kubernetes/kubernetes/issues?since=2018-01-02T00:00:00Z
> 
>  
> . 
> 
> How can I achieve this dynamic query parameters for each request based on
> previous request response?
> 
> Any help or directions would be great help.
> 
> Regards
> Devaraj
> 
> 
> 
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/



smime.p7s
Description: S/MIME cryptographic signature

Re: Getting the value of a variable defined in a processor group

[Vijay Chhipa]

Bryan, Andy

Thanks for the reply, 

I could use the PropertyDescriptor and have the user enter a value using EL. 

My use case is this:

I am constructing a topic name, to post data to. 
Part of the topic name, I would the user to enter in a field. But the other 
part I would like to pick up from a variable without having the user worry 
about it. 

Not a blocker issue, but would be nice. 

Thanks again. 





> On May 14, 2019, at 1:17 PM, Bryan Bende  wrote:
> 
> The intent is to get a variable through a property of a component
> using expression language. So you define a PropertyDescriptor that
> supports EL like:
> 
> public static final PropertyDescriptor DIRECTORY = new
> PropertyDescriptor.Builder()
>.name("Directory")
>.description("The HDFS directory from which files should be read")
>.required(true)
>
> .addValidator(StandardValidators.ATTRIBUTE_EXPRESSION_LANGUAGE_VALIDATOR)
>
> .expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
>.build();
> 
> Then you get the value and evaluate the expressions:
> 
> context.getProperty(DIRECTORY).evaluateAttributeExpressions(flow).getValue()
> 
> This will automatically substitute in any variables, so if the user
> entered ${directory} and there is a variable "directory", it will be
> substituted in. The order of precedence when evaluating expression
> language is the following:
> 
> 1) flow file attributes
> 2) variable registry at process group level
> 3) variable registry properties files
> 4) system properties
> 5) environment variables
> 
> So if there was a flow file attribute named "directory" it would take
> precedence over a process group variable named "directory".
> 
> Some properties don't support EL from flow file attributes and only
> have  expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
> and then don't pass in a flow file when evaluating:
> 
> context.getProperty(DIRECTORY).evaluateAttributeExpressions().getValue()
> 
> Is there a good use case for getting a variable value without having a
> PropertyDescriptor?
> 
> On Tue, May 14, 2019 at 2:05 PM Andy LoPresto  wrote:
>> 
>> Vijay,
>> 
>> I don’t believe there is a good way to get a complete list of all variables 
>> within processor code. If you want to explore this further, you can take a 
>> look at StandardProcessContext [1], which processors have access to in their 
>> #onTrigger() method, and MutableVariableRegistry [2], which is the only 
>> concrete implementation of the ComponentVariableRegistry interface which 
>> exposes a variable map (the StandardComponentVariableRegistry returns an 
>> empty map when accessed). The issue is that the context does not expose the 
>> ProcessorNode field, which is the intermediary to the variable registry. I 
>> believe this is by design. There may be a valid use case to provide an 
>> accessor method on the context to return a variable map, or at least the 
>> entry set.
>> 
>> I’d like to hear from Matt Gilman, Bryan Bende, and/or Mark Payne with their 
>> thoughts, since much of this code was written by them.
>> 
>> [1] 
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/processor/StandardProcessContext.java#L87
>> [2] 
>> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/registry/variable/MutableVariableRegistry.java#L27
>> 
>> 
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>> On May 14, 2019, at 10:48 AM, Vijay Chhipa  wrote:
>> 
>> Hello
>> 
>> I am writing a new NiFi processor. In the source code I would like to use a 
>> variable defined in the NiFi processor group using the UI.
>> Is there a way to get the map of all the variables available to this 
>> processor?
>> 
>> 
>> I am on NiFi 1.9.1
>> 
>> 
>> Thanks
>> 
>> 
>> 
>> 



smime.p7s
Description: S/MIME cryptographic signature

Getting the value of a variable defined in a processor group

[Vijay Chhipa]

Hello

I am writing a new NiFi processor. In the source code I would like to use a 
variable defined in the NiFi processor group using the UI. 
Is there a way to get the map of all the variables available to this processor?


I am on NiFi 1.9.1 


Thanks





smime.p7s
Description: S/MIME cryptographic signature

Re: InvokeHTTP POST

[Vijay Chhipa]

Martjin, 

Can you show what the HTTP error code and the message is?
Also, please list all the properties you have set on the InvokeHTTP processor. 
A screenshot works too,

Vijay


> On Apr 26, 2019, at 6:27 AM, Martijn Dekkers  wrote:
> 
> Hi,
> 
> I am working to implement a flow where we need to POST some data to another 
> system using multipart/form-data. This system requires that one of the form 
> fields acts as an attached file. Using curl, this works fine:
> 
> curl -F = -F = -F =@[filename] 
> 
> I created a flowfile that mimics what curl sends, including the boundaries 
> etc. Using InvokeHTTP POST I receive a web-server error from the receiving 
> end that mentions a missing parameter.
> 
> Running with curl --trace-ascii or POSTing to a running netcat with both curl 
> and nifi shows exactly the same results, with the exception of the actual 
> boundary marker (which is always different) and some minor header stuff (UA, 
> etc.) The payload is simple text. 
> 
> I understand that there is some difference between sending the data and 
> uploading a file, but am unable to figure out what this is. for example, 
> sending the same data with curl using -F=<[filename] (instead of 
> @[filename]) also results in an error. 
> 
> I have read a few different messages about POSTing multipart-form data where 
> files need to be uploaded, and most appear to end up with writing a scripted 
> processor or invoking shell. This is particularly undesirable for us given 
> the volume of files we must process, and the speed at which we should be 
> uploading those files. 
> 
> Is there any way in which to mimic the uploading/attaching of a file, instead 
> of sending the data as text?
> 
> Thanks
> 
> Martijn



smime.p7s
Description: S/MIME cryptographic signature

Re: When do I download the older releases of Apache NiFi

[Vijay Chhipa]

Thanks Bryan and Tomislav. 

Is it possible to put a link to that on the current download page:  
https://nifi.apache.org/download.html <https://nifi.apache.org/download.html> ?

Vijay


> On Feb 27, 2019, at 11:56 AM, Tomislav Novosel  wrote:
> 
> Hi Vijay,
> 
> here you can find older releases:
> https://archive.apache.org/dist/nifi/ <https://archive.apache.org/dist/nifi/>
> 
> BR,
> Tom
> 
> On Wed, 27 Feb 2019, 18:52 Vijay Chhipa,  <mailto:vchh...@apple.com>> wrote:
> Hi, 
> 
> Noticed that with the release of NiFi 1.9.0  there is not a link to download 
> version 1.7.1 and older, 
> 
> Is there a link to the archived versions?
> 
> Thanks
> 



smime.p7s
Description: S/MIME cryptographic signature

When do I download the older releases of Apache NiFi

[Vijay Chhipa]

Hi, 

Noticed that with the release of NiFi 1.9.0  there is not a link to download 
version 1.7.1 and older, 

Is there a link to the archived versions?

Thanks



smime.p7s
Description: S/MIME cryptographic signature

Re: running multiple commands from a single ExecuteStreamCommand processor

[Vijay Chhipa]

Thanks Mark, that was very helpful. 

I have chained the commands now and writing the final output to stdout.

Thanks again, 
Vijay


> On Feb 13, 2019, at 2:34 PM, Mark Payne  wrote:
> 
> Vijay,
> 
> No worries, this thread is fine. The processor will stream the contents of 
> the FlowFIle to the Standard Input (StdIn) of the process
> that is generated. So it will go to the bash script. The bash script can do 
> whatever it needs to do, pipe to another command, etc.
> Whatever is written to StdOut becomes the content of the FlowFile. So it 
> would be up to you to pipe the output of the first command
> to the input of the second. Does that make sense?
> 
> Thanks
> -Mark
> 
> 
> 
>> On Feb 13, 2019, at 3:26 PM, Vijay Chhipa  wrote:
>> 
>> Mark,
>> 
>> Thanks for your quick response, 
>> When calling bash script that has multiple commands, is there a single flow 
>> file generated after all commands are executed (accumulating output from 
>> each command) or multiple flow files generated per command line in the bash 
>> script. 
>> 
>> Sorry for tagging along another question on top of this, I can ask it as a 
>> separate thread if it makes more sense. 
>> 
>> Thanks
>> 
>> 
>>> On Feb 13, 2019, at 12:50 PM, Mark Payne  wrote:
>>> 
>>> Vijay,
>>> 
>>> This would be treated as arguments to a single command.
>>> 
>>> One option would be to create a simple bash script that executes the 
>>> desired commands and invoke
>>> that from the processor. Or, of course, you can chain together multiple 
>>> processors.
>>> 
>>> Thanks
>>> -Mark
>>> 
>>> 
>>>> On Feb 13, 2019, at 1:48 PM, Vijay Chhipa  wrote:
>>>> 
>>>> Hi, 
>>>> 
>>>> I have a ExecuteStreamCommand  processor running a single command, 
>>>> (executing a  -jar  ) and it runs fine, 
>>>> 
>>>> I need to run the same command but with different arguments. 
>>>> 
>>>> My question is: Can I put multiple lines as command arguments and still 
>>>> have a single instance of the ExecuteStreamCommand?
>>>> 
>>>> Would those be treated as arguments to a single command, or each line of 
>>>> arguments would be treated as separate command?
>>>> 
>>>> 
>>>> Thanks 
>>>> 
>>>> Vijay
>>>> 
>>>> 
>>>> 
>>> 
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: running multiple commands from a single ExecuteStreamCommand processor

[Vijay Chhipa]

Mark,

Thanks for your quick response, 
When calling bash script that has multiple commands, is there a single flow 
file generated after all commands are executed (accumulating output from each 
command) or multiple flow files generated per command line in the bash script. 

Sorry for tagging along another question on top of this, I can ask it as a 
separate thread if it makes more sense. 

Thanks


> On Feb 13, 2019, at 12:50 PM, Mark Payne  wrote:
> 
> Vijay,
> 
> This would be treated as arguments to a single command.
> 
> One option would be to create a simple bash script that executes the desired 
> commands and invoke
> that from the processor. Or, of course, you can chain together multiple 
> processors.
> 
> Thanks
> -Mark
> 
> 
>> On Feb 13, 2019, at 1:48 PM, Vijay Chhipa  wrote:
>> 
>> Hi, 
>> 
>> I have a ExecuteStreamCommand  processor running a single command, 
>> (executing a  -jar  ) and it runs fine, 
>> 
>> I need to run the same command but with different arguments. 
>> 
>> My question is: Can I put multiple lines as command arguments and still have 
>> a single instance of the ExecuteStreamCommand?
>> 
>> Would those be treated as arguments to a single command, or each line of 
>> arguments would be treated as separate command?
>> 
>> 
>> Thanks 
>> 
>> Vijay
>> 
>> 
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

running multiple commands from a single ExecuteStreamCommand processor

[Vijay Chhipa]

Hi, 

I have a ExecuteStreamCommand  processor running a single command, (executing a 
 -jar  ) and it runs fine, 

I need to run the same command but with different arguments. 

My question is: Can I put multiple lines as command arguments and still have a 
single instance of the ExecuteStreamCommand?
  
Would those be treated as arguments to a single command, or each line of 
arguments would be treated as separate command?


Thanks 

Vijay





smime.p7s
Description: S/MIME cryptographic signature

Re: How to remove lines from a flow file that don't start with a certain prefix?

[Vijay Chhipa]

Mark, 


Thanks for the tip, I set up the RouteText processor like below and it worked 
beautifully. First I thought that each matched line will become its own 
flowfile but thats not the case. All lines that matched become part of a single 
new flow file. 





Vijay



> On Dec 29, 2018, at 8:18 AM, Mark Payne  wrote:
> 
> Vijay,
> 
> I would recommend using RouteText for that use case. You can then use 
> Expression Language against line of text to do something like 
> ${line:startsWith(“{“)}
> 
> It avoids the complexities of regex and is much more efficient. 
> 
> Thanks
> -Mark
> 
> Sent from my iPhone
> 
> On Dec 29, 2018, at 7:56 AM, Vijay Chhipa  <mailto:vchh...@apple.com>> wrote:
> 
>> Hi All, 
>> 
>> I have an output from a processor that contains lines that are JSON 
>> structured, i.e. each line is a JSON by itself. 
>> In certain cases there are lines that are not JSON and I want to remove 
>> them. 
>> 
>> I tried the ReplaceText processor with the following patterns. 
>> 
>> ((.|\n)*?)(?={)
>> 
>> Contents of the input file:
>> foobar
>> sdkfskdl
>> ksdfjlsdj
>> {"key":"value"}
>> {"key2":"value2"}
>> 
>> Desired output file contents: 
>> {"key":"value"}
>> {"key2":"value2"}
>> 
>> 
>> In the https://www.regextester.com/ <https://www.regextester.com/>  site 
>> above pattern gives me the following matched string: 
>> 
>> 
>> 
>> 
>> When I put the above in the ReplaceText Processor like this
>> 
>> I get an error that this is not a valid Java regular expression: 
>> 
>> 
>> 
>> Why is this not valid, and is there an online reg ex checked that I can 
>> validate with before putting it in NiFi?
>> Also should I be using a different processor than the ReplaceText for this 
>> purpose?
>> 
>> 
>> Thank you for your help and Happy New Year. 
>> 
>> Vijay
>> 
>> 
>> 
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Unable to List Queue on a connection

[Vijay Chhipa]

Andy

Thank you for the pointers, that worked perfectly. 

Cheers!

Vijay


> On Dec 17, 2018, at 6:23 PM, Andy LoPresto  wrote:
> 
> Hi Vijay,
> 
> Apache NiFi 1.x doesn’t have “roles”, so the “administrators” group doesn’t 
> carry any special significance [1], and connections do not have policies 
> assigned to them. You’ll need to assign the “View the data” and “Modify the 
> data” policies to yourself on the specified resources (the components where 
> the connection originates and terminates) [2]. If you do this on the root 
> process group (or more granularly, the specific process group containing this 
> connection), all child process groups and components will inherit that 
> permission unless you override it [3]. There are some examples of this 
> process to help [4]. Hope this helps. 
> 
> [1] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policies
>  
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policies>
> [2] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#component-level-access-policies
>  
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#component-level-access-policies>
> [3] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policy-inheritance
>  
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policy-inheritance>
> [4] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policy-config-examples
>  
> <https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#access-policy-config-examples>
> 
> Andy LoPresto
> alopre...@apache.org <mailto:alopre...@apache.org>
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> 
>> On Dec 17, 2018, at 2:45 PM, Vijay Chhipa > <mailto:vchh...@apple.com>> wrote:
>> 
>> Hi 
>> 
>> I have a secure NiFi instance setup. 
>> 
>> I put my self in the Administrator group and created policies for the 
>> administrator. 
>> 
>> When I right click on  a connection and click on the List Queue, I get the 
>> following error:  
>> 
>> Insufficient Permissions:
>>  No Applicable policies could be found. Contact the system 
>> administrator. 
>>   OK
>> 
>> 
>> What permission I need to give myself to view the queue and may be empty the 
>> queue.
>> 
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Unable to List Queue on a connection

[Vijay Chhipa]

Hi 

I have a secure NiFi instance setup. 

I put my self in the Administrator group and created policies for the 
administrator. 

When I right click on  a connection and click on the List Queue, I get the 
following error:  

Insufficient Permissions:
 No Applicable policies could be found. Contact the system 
administrator. 
 OK


What permission I need to give myself to view the queue and may be empty the 
queue.



smime.p7s
Description: S/MIME cryptographic signature

Re: Can a nar bundle host a servlet outside of the NiFi web framework?

[Vijay Chhipa]

Hi Bryan

So we implemented it based on your suggestion, and it works, however, for 
concerns unrelated to NiFi  from the team, I  will need to go back to the 
original idea which is standup  a servlet from within my .nar file.

Is it possible and if so can you point me to an example?

Thank you, 

Vijay



> On Nov 6, 2018, at 4:35 PM, Vijay Chhipa  wrote:
> 
> Bryan
> 
> Thats perfect, I was planning to launch my own Jetty instance from within the 
> nar, but ListenHttp or HandleHttpRequest is even better 
> 
> Thank you, 
> 
> 
>> On Nov 6, 2018, at 4:24 PM, Bryan Bende  wrote:
>> 
>> Not sure if this is what you are asking for, but ListenHttp and
>> HandleHttpRequest both do this.
>> 
>> They start an embedded Jetty server on a different port, unrelated to
>> NiFi's web server, and they are not part of NiFi's security model,
>> although they can be secured with 2-way TLS, or basic auth (I tthink).
>> 
>> The incoming request gets turned into a flow file and be processed by
>> the rest of the flow, and a response can be sent with
>> HandleHttpResponse.
>> On Tue, Nov 6, 2018 at 4:52 PM Vijay Chhipa  wrote:
>>> 
>>> Hello
>>> 
>>> I would like to create a nar bundle that can serve up some content, I would 
>>> like this to be outside of the secure NiFI web context. Essentially it has 
>>> nothing to do with the "https://:/nifi"  and the 
>>> nifi.properties
>>> 
>>> This would be a unsecured endpoint  accessible from the host thats running 
>>> NiFi.
>>> http:/localhost:8089/blabla
>>> 
>>> Essentially, my nar  would  be running a Jetty server other than the NiFi 
>>> Jetty server
>>> 
>>> Any technical issues with this approach ( obviously there is a security 
>>> concerns but apart from that)
>>> 
>>> Thanks
>>> 
>>> Vijay
>>> 
>>> 
>>> 
>>> 
>>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Can a nar bundle host a servlet outside of the NiFi web framework?

[Vijay Chhipa]

Bryan

Thats perfect, I was planning to launch my own Jetty instance from within the 
nar, but ListenHttp or HandleHttpRequest is even better 

Thank you, 


> On Nov 6, 2018, at 4:24 PM, Bryan Bende  wrote:
> 
> Not sure if this is what you are asking for, but ListenHttp and
> HandleHttpRequest both do this.
> 
> They start an embedded Jetty server on a different port, unrelated to
> NiFi's web server, and they are not part of NiFi's security model,
> although they can be secured with 2-way TLS, or basic auth (I tthink).
> 
> The incoming request gets turned into a flow file and be processed by
> the rest of the flow, and a response can be sent with
> HandleHttpResponse.
> On Tue, Nov 6, 2018 at 4:52 PM Vijay Chhipa  wrote:
>> 
>> Hello
>> 
>> I would like to create a nar bundle that can serve up some content, I would 
>> like this to be outside of the secure NiFI web context. Essentially it has 
>> nothing to do with the "https://:/nifi"  and the 
>> nifi.properties
>> 
>> This would be a unsecured endpoint  accessible from the host thats running 
>> NiFi.
>> http:/localhost:8089/blabla
>> 
>> Essentially, my nar  would  be running a Jetty server other than the NiFi 
>> Jetty server
>> 
>> Any technical issues with this approach ( obviously there is a security 
>> concerns but apart from that)
>> 
>> Thanks
>> 
>> Vijay
>> 
>> 
>> 
>> 
>> 



smime.p7s
Description: S/MIME cryptographic signature

Can a nar bundle host a servlet outside of the NiFi web framework?

[Vijay Chhipa]

Hello

I would like to create a nar bundle that can serve up some content, I would 
like this to be outside of the secure NiFI web context. Essentially it has 
nothing to do with the "https://:/nifi"  and the nifi.properties  
   

This would be a unsecured endpoint  accessible from the host thats running NiFi.
http:/localhost:8089/blabla

Essentially, my nar  would  be running a Jetty server other than the NiFi Jetty 
server

Any technical issues with this approach ( obviously there is a security 
concerns but apart from that) 

Thanks

Vijay







smime.p7s
Description: S/MIME cryptographic signature

Cluster setup in multiple data centers

[Vijay Chhipa]

Hello 

We are putting NiFi in production. To ensure that there is no single point of 
failure we have two options. 

a) Setup a single NiFi cluster with nodes distributed across data centers. 
b) Setup a cluster in one data center and for failover have a hot stand-by 
cluster in another data center 

What are the pros and cons of each approach?

Are there other options?

Any feedback based on your experience will be highly useful. 

Thank you, 

Vijay

How do I logout of NiFi UI

[Vijay Chhipa]

Hello 

I can't find the 'logout' link on the canvas or under my profile name or in the 
hamburger menu. 
Whats the recommended way to logout of Apache NiFi 

Can you please point me to this link on a screenshot or if its somewhere in the 
docs a pointer to that is highly appreciated. 

Thanks

SAML based identity provider

[Vijay Chhipa]

Hello,

I am setting up NiFi in the company, but the out-of-the-box authentication 
modules are not an option for me.
I would like to write a SAML based login identity provider, 
Is there one out there already ?

I am on NiFi 1.7.1, with Java 8, SAML 2.0, 

What do I need to get started with writing a new  login identity provider? Any 
examples, sample, or pointers are highly appreciated

Vijay



smime.p7s
Description: S/MIME cryptographic signature

Re: Send data to ElasticSearch

[Vijay Chhipa]

For value of the date field, ensure that you have 'Z' at the end to indicate 
the timezone, even if your timezone is UTC. I ran into a similar issue working 
with ES  HTTP API from Python



> On Jul 16, 2018, at 1:59 PM, Mike Thomsen  wrote:
> 
> ISO8601



smime.p7s
Description: S/MIME cryptographic signature