Re: NiFi - How to Wait until all Inserts into Table are done, then Create/Analyze Indexes on the Table afterwards

[Koji Kawamura]

Hi Matt,

I posted my answer to your Stackoverflow question.
https://stackoverflow.com/questions/55483317/nifi-create-indexes-after-inserting-records-into-table/55486259#55486259

Thanks,
Koji

On Wed, Apr 3, 2019 at 8:04 AM matthewmich...@northwesternmutual.com
 wrote:
>
> NiFi Developers,
>
> I tried reaching out to Stack Overflow Apache-Nifi Tag, but am getting no 
> replies - so normally I wouldn't ask this email list this type of question, 
> but am not sure where else to turn?
>
> So to keep simple, my 1st Process Group (PG) Truncates then drops indexes in 
> table. Then that output port routes to input port in 2nd process group the 
> does the 500K Inserts into the table. After successfully inserting the 500K 
> rows, I want to create the indexes on the table and analyze it via 3rd 
> process group. This is typical Data Warehouse methodology. Can anyone please 
> give advice on how to do this?  I'd like to be able to not even start the 3rd 
> process group, until first 2 complete successfully, and the Insert Count 
> matches the Select Count.
>
> I've tried setting counters, then comparing total inserted count to match 
> initial select count, but it appears that I cannot reference Counters in the 
> Expression Language because this "${InsertCounter}"  syntax always returns 
> nulls??   So maybe instead I should be using the Wait & Notify Processors, so 
> could Wait until all the inserts are done, then Notify the next process (the 
> create & analyze analyze indexes process) to start after all the inserts are 
> done?  Here's example code on that from Stack Overflow:
>
> In the wait processor set the Target Signal Count to ${fragment.count}.
>
> Set the Release Signal Identifier in both the notify and wait processor to 
> ${fragment.identifier}
> Thanks for any help anyone can provide!
>
> Thanks & Regards,
>
> Matt Michala
> Senior Engineer
> Digital Workplace & Corp. Solutions
> Northwestern Mutual (T07-150)
> 414-661-4668
>
>
>
> This e-mail and any attachments may contain confidential information of 
> Northwestern Mutual. If you are not the intended recipient of this message, 
> be aware that any disclosure, copying, distribution or use of this e-mail and 
> any attachments is prohibited. If you have received this e-mail in error, 
> please notify Northwestern Mutual immediately by returning it to the sender 
> and delete all copies from your system. Please be advised that communications 
> with {SECURE MESSAGE} in the subject line have been sent using a secure 
> messaging system. Communications that do not have this tag may not be secure 
> and could be observed by a third party. Thank you for your cooperation.
>
>

Re: initial component access policies

[Bryan Bende]

The initial admin policies are created here:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-file-authorizer/src/main/java/org/apache/nifi/authorization/FileAccessPolicyProvider.java#L595
 


You can see where it will create the root group policies if rootGroupId is not 
null.

The rootGroupId comes from the parseFlow() method above which tries to read the 
flow.xml.gz from disk, using the location from nifi.properties.


> On Apr 2, 2019, at 9:57 PM, Mark Bean  wrote:
> 
> When NiFi is started for the first time, the Component Access Policies are
> not populated even for the Initial Admin or for legacy DFM_ROLE users in
> authorized-users.xml file.That is, not unless a flow.xml.gz file exists.
> The fact that the admin user does not have access to these policies has led
> to confusion for a great number of users.
> 
> I believe this came up before and an explanation was given that since the
> flow.xml.gz does not yet exist (nor the root process group's UUID), the
> Component Access Policies cannot be created. However, I have to believe
> there is a mechanism that can be employed to return to policy generation
> after the flow.xml.gz is created.
> 
> Can someone provide some pointers to where in the code I can look to see
> where the Global Policies are initially created and/or where Component
> Policies are created when migrating with an existing flow.xml.gz?
> 
> Thanks,
> Mark

initial component access policies

[Mark Bean]

When NiFi is started for the first time, the Component Access Policies are
not populated even for the Initial Admin or for legacy DFM_ROLE users in
authorized-users.xml file.That is, not unless a flow.xml.gz file exists.
The fact that the admin user does not have access to these policies has led
to confusion for a great number of users.

I believe this came up before and an explanation was given that since the
flow.xml.gz does not yet exist (nor the root process group's UUID), the
Component Access Policies cannot be created. However, I have to believe
there is a mechanism that can be employed to return to policy generation
after the flow.xml.gz is created.

Can someone provide some pointers to where in the code I can look to see
where the Global Policies are initially created and/or where Component
Policies are created when migrating with an existing flow.xml.gz?

Thanks,
Mark

NiFi - How to Wait until all Inserts into Table are done, then Create/Analyze Indexes on the Table afterwards

[matthewmich...@northwesternmutual.com]

NiFi Developers,

I tried reaching out to Stack Overflow Apache-Nifi Tag, but am getting no 
replies - so normally I wouldn't ask this email list this type of question, but 
am not sure where else to turn?

So to keep simple, my 1st Process Group (PG) Truncates then drops indexes in 
table. Then that output port routes to input port in 2nd process group the does 
the 500K Inserts into the table. After successfully inserting the 500K rows, I 
want to create the indexes on the table and analyze it via 3rd process group. 
This is typical Data Warehouse methodology. Can anyone please give advice on 
how to do this?  I'd like to be able to not even start the 3rd process group, 
until first 2 complete successfully, and the Insert Count matches the Select 
Count.

I've tried setting counters, then comparing total inserted count to match 
initial select count, but it appears that I cannot reference Counters in the 
Expression Language because this "${InsertCounter}"  syntax always returns 
nulls??   So maybe instead I should be using the Wait & Notify Processors, so 
could Wait until all the inserts are done, then Notify the next process (the 
create & analyze analyze indexes process) to start after all the inserts are 
done?  Here's example code on that from Stack Overflow:

In the wait processor set the Target Signal Count to ${fragment.count}.

Set the Release Signal Identifier in both the notify and wait processor to 
${fragment.identifier}
Thanks for any help anyone can provide!

Thanks & Regards,

Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668



This e-mail and any attachments may contain confidential information of 
Northwestern Mutual. If you are not the intended recipient of this message, be 
aware that any disclosure, copying, distribution or use of this e-mail and any 
attachments is prohibited. If you have received this e-mail in error, please 
notify Northwestern Mutual immediately by returning it to the sender and delete 
all copies from your system. Please be advised that communications with {SECURE 
MESSAGE} in the subject line have been sent using a secure messaging system. 
Communications that do not have this tag may not be secure and could be 
observed by a third party. Thank you for your cooperation. 

Outdated NiFi svn repo

[Andrew Lim]

Does anyone know why the following repo exists?

https://svn.apache.org/repos/infra/websites/production/nifi/

A new NiFi user came across the outdated documentation in that repo [1].  To 
prevent this from happening again, I was planning on filing a JIRA with the ASF 
INFRA team, but wanted to ask on this mailing list before doing so.

Thanks,

-Drew

[1] https://issues.apache.org/jira/browse/NIFI-6173