Re: Template not saving parameter context assignment

[Andy LoPresto]

Parameter contexts were added after the use of templates was no longer 
recommended and substantially deprecated and NiFi Registry was selected as the 
managed flow definition version control and environment promotion system. There 
are future efforts to facilitate the export/import of flow segments directly 
to/from a NiFi instance without NiFi Registry but they are not released yet. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jul 28, 2020, at 2:00 PM, bsavard  wrote:
> 
> Hi,
> 
> I have a flow that uses parameter contexts.  I understand that the parameter
> context definitions themselves are persisted to flow.xml.gz.
> 
> However, when I save my flow as a template, delete the flow from the canvas,
> then add the template back onto a blank canvas, all of the processor context
> assignments I made are gone.  When I download and search the template file
> itself, there's no reference to the parameter context assignments.
> 
> So, where are the assignments stored?  And how would I go about deploying
> this template to many servers without someone having to manually go back
> into the flow on each server and adding the assignments all over the place? 
> We don't use NiFi Registry; we use a traditional Maven based deployment
> cycle.
> 
> TIA!
> 
> 
> 
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Template not saving parameter context assignment

[bsavard]

Hi,

I have a flow that uses parameter contexts.  I understand that the parameter
context definitions themselves are persisted to flow.xml.gz.

However, when I save my flow as a template, delete the flow from the canvas,
then add the template back onto a blank canvas, all of the processor context
assignments I made are gone.  When I download and search the template file
itself, there's no reference to the parameter context assignments.

So, where are the assignments stored?  And how would I go about deploying
this template to many servers without someone having to manually go back
into the flow on each server and adding the assignments all over the place? 
We don't use NiFi Registry; we use a traditional Maven based deployment
cycle.

TIA!



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Issue with Secured NiFi on kubernetes using Helm charts

[Atul Wankhade]

Thanks a million Sushil finally was able to get it up and running(With
SSL). You just made my day, even though it's night time now :).
After the tweaks you suggested, it worked all fine.

Thanks again,
Atul

On Mon, Jul 27, 2020 at 11:49 PM Atul Wankhade 
wrote:

> That's awesome, sure will take a look.
>
> Thanks,
> Atul
>
> On Mon, Jul 27, 2020, 10:52 PM Sushil Kumar  wrote:
>
>> Hello Atul
>>
>> You can have a look at the chart at
>> https://github.com/sushilkm/nifi-chart repository.
>> I used this chart to generate and use self-signed certificates, and
>> deploy it over azure.
>>
>> Couple of things to note before you use it:
>> storageclass -> you might need to change the value of storageclass
>> parameter as per where you trying to deploy
>> https://github.com/sushilkm/nifi-chart/blob/master/nifi/values.yaml#L58
>> You might need to tweak memory parameters as per your requirement, but it
>> should not matter to test deploy.
>> https://github.com/sushilkm/nifi-chart/blob/master/nifi/values.yaml#L49-L54
>> Do not forget to update the default sizes of volumes.
>> https://github.com/sushilkm/nifi-chart/blob/master/nifi/values.yaml#L57-L64
>>
>> There are helpful READMEs in project home as well as in the chart
>> directory.
>> FYI, this is not a production code, use at your own risk.
>>
>> Let me know if you have any questions.
>>
>> Thanks
>> Sushil Kumar
>>
>> On Mon, Jul 27, 2020 at 9:15 AM Atul Wankhade 
>> wrote:
>>
>>>
>>> Hi Sushil,
>>>
>>> Would you be kind enough to share the Helm chart unless it's
>>> confidential? I really want to make this setup work. As said above I am
>>> generating the certs in the init container, but I am not sure where they
>>> are getting stored, when I am redirecting to a volume mount I am getting
>>> below error.
>>>   initContainers:
>>> - name: nifi-init
>>>   image: "apache/nifi:1.11.1"
>>>   imagePullPolicy: "IfNotPresent"
>>>   command: [*'sh', '-c',
>>> '/opt/nifi/nifi-toolkit-current/bin/tls-toolkit.sh client -c nifi-ca-cs -t
>>> Mytesttoken12345 --dn "CN=$(hostname -f),
>>> OU=NIFI"','>','/opt/nifi/flowfile_repository/'*]
>>>   #volumeMounts:
>>>   # - mountPath: /opt/certs/
>>>   #   name: certs
>>>   volumeMounts:
>>> - name: "flowfile-repository"
>>>   mountPath: /opt/nifi/flowfile_repository
>>> [image: image.png]
>>> I even tried to write the whole command within the same quotes, no luck.
>>> Gave cetic helm chart a try after getting rid of that error, now nifi pods
>>> aren't coming up at all :(
>>> Will it possible for to connect personally? Below is my whatsapp number.
>>>
>>> Thanks again for the helping hand,
>>> Atul
>>> +91 9766545790
>>>
>>> On Sun, Jul 26, 2020 at 11:57 PM Sushil Kumar  wrote:
>>>
 Hello Atul

 I wrote the chart myself.
 Cert generation pattern was similar to what you are trying.
 I ran the server as a separate container, and generated client certs in
 init-container.

 Thanks
 Sushil

 On Sun, Jul 26, 2020, 9:46 AM Atul Wankhade 
 wrote:

> Hi Sushil,
>
> I am using Cetic helm chart only. May I know which did you use? Where
> did you generate the certs?
>
> Thanks,
> Atul
>
> On Sat, Jul 25, 2020 at 2:00 AM Sushil Kumar 
> wrote:
>
>> Hello Atul
>>
>> I have recently tried using self signed certificates generated using
>> nifi toolkit while using helm chart.
>> cetic helm chart is not written completely to accomplish this,
>> I may be able to help if you can share your helm chart.
>>
>> However, as of now the error is in your values.yaml file.
>>
>> Thanks
>> Sushil Kumar
>>
>> On Fri, Jul 24, 2020 at 9:14 AM Chris Sampson <
>> chris.samp...@naimuri.com> wrote:
>>
>>> I don't use our know much about helm, but that error suggests you've
>>> got something wrong on line 202 of your yaml, so what's on that line (or
>>> the lines immediately before/after)?
>>>
>>> Notice you're using nifi 1.11.1, might be worth considering 1.11.4
>>> if you can to take advantage of several high priority by fixes in nifi 
>>> (but
>>> that won't affect your helm chart). Also, suggest using the
>>> apache/nifi-toolkit image for running the toolkit in tls server mode 
>>> (much
>>> lighter weight), but again that's not likely to be causing you a problem
>>> here.
>>>
>>>
>>> Cheers,
>>>
>>> Chris Sampson
>>>
>>> On Fri, 24 Jul 2020, 15:05 Atul Wankhade, 
>>> wrote:
>>>
 Chris I am trying what you have suggested, while passing the init
 container params in values.yaml getting below error, can you please 
 help to
 get around this issue.
 *Error: cannot load values.yaml: error converting YAML to JSON:
 yaml: line 202: did not find expected ',' or '}'*
 I am adding below i

Nifi 1.8 to 1.11.4 migration issue

[sanjeet rath]

Hi Team,

I am facing a wired issue while doing migration from 1.8 to 1.11.4
I am bringing flow.xml.gz,user.xml,authorization.xml from a nifi
cluster(1.8 version) to a newly created 1.11.4 version cluster.

When i am starting the 1.11.4 cluster with this new files, i am getting
node disconnected error (this node is not conncted to cluster pop up )

However with only users.xml, authrization.xml from 1.8 and default 1.11.4
flow.xml.gz. i am getting no error all 3 nodes are getting conncted in the
new cluster.
Infact when i am bringing another 1.8 env's flow.xml.gz files it working
fine.

So specific for this flow.xml.gz file only my nodes are getting disconncted.

It will really helpfull if you can help where i need to check to
resolve this issue.

I am using external zookeeper 3.5 version

Thanks in advance,
Sanjeet

Re: DetectDuplicate Processor Doesn't Remove From Cache

[bsavard]

Hi,

I'm questioning my understanding of how the code is working.  I did a screen
shot while debugging and I'll attach it here.  If you look at the values of
'now' and 'originalCachevalue entryTimeMS', how could this ever work? 
Contrast the value of cacheValue.entryTimeMS vs
originalCacheValue.entryTimeMS.  

Stepping through the code, I think the value of entryTimeMS is derived from
the flowfile's value, not from a timestamp.  I'm assuming that I haven't had
enough coffee yet and I'm just being stupid. :)


 




--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: DetectDuplicate Processor Doesn't Remove From Cache

[Mark Payne]

Hello,

I’m not sure that I understand the concern. CacheValue.getEntryTimeMS() returns 
a Long (64-bit signed integer) value, which
represents the timestamp of when the entry was added. Are you experiencing 
problems with events not being evicted and resulting
in heap exhaustion, or is your concern just stemming from a lack of clarity 
around how the code is working?

Thanks
-Mark


> On Jul 28, 2020, at 9:28 AM, bsavard  wrote:
> 
> Hi,
> 
> I've developed a test where I'm caching a flowfile using
> PutDistributedMapCache, then checking for entries using a DeleteDuplicate
> processor.  I'm finding that the DeleteDuplicate is, in fact, finding the
> duplicate, but it's not removing it from the cache.
> The problem is the test in DeleteDuplicate on line 189:
> if (duplicate && durationMS != null && (now >=
> originalCacheValue.getEntryTimeMS() + durationMS)) {
> 
> If I dig into the code, originalCacheValue.getEntryTimeMS() doesn't look to
> me like it really represents a time value, so the comparison with "now"
> always fails.  I say that because the cache value seems to be a
> serialization of the flowfile.
> 
> Maybe I've misunderstood what the doc means when it says "determines if the
> cached value has already been seen".  Has anyone else had difficulty with
> this?
> 
> 
> 
> --
> Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

DetectDuplicate Processor Doesn't Remove From Cache

[bsavard]

Hi,

I've developed a test where I'm caching a flowfile using
PutDistributedMapCache, then checking for entries using a DeleteDuplicate
processor.  I'm finding that the DeleteDuplicate is, in fact, finding the
duplicate, but it's not removing it from the cache.
The problem is the test in DeleteDuplicate on line 189:
if (duplicate && durationMS != null && (now >=
originalCacheValue.getEntryTimeMS() + durationMS)) {

If I dig into the code, originalCacheValue.getEntryTimeMS() doesn't look to
me like it really represents a time value, so the comparison with "now"
always fails.  I say that because the cache value seems to be a
serialization of the flowfile.

Maybe I've misunderstood what the doc means when it says "determines if the
cached value has already been seen".  Has anyone else had difficulty with
this?



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/