Re: Default NiFi registry port

2018-06-20 Thread Jagrut Sharma
Thanks Bryan! -- Jagrut On Wed, Jun 20, 2018 at 12:18 PM, Bryan Bende wrote: > Hello, > > Since the port is configurable and can easily be changed I don't think > we would plan to change it. > > There are also lots of people who are not running NiFi Registry on the > same server as Spark History

Re: Default NiFi registry port

2018-06-20 Thread Bryan Bende
Hello, Since the port is configurable and can easily be changed I don't think we would plan to change it. There are also lots of people who are not running NiFi Registry on the same server as Spark History Server, so I don't think changing it just for that makes sense. Thanks, Bryan On Wed, J

Re: Apache NiFi 1.7.0 RC1 Release Helper Guide

2018-06-20 Thread Otto Fowler
I am working on a script to automate a bunch of this. I just created a work in progress PR if you would like to check it out. https://github.com/apache/nifi/pull/2806 Checking the commit is next thing on my list to automate. On June 20, 2018 at 10:52:23, Bryan Bende (bbe...@gmail.com) wrote:

Default NiFi registry port

2018-06-20 Thread Jagrut Sharma
Hi - The default NiFi registry port is 18080, which is also the default port for Spark History Server UI. Due to this, the startup failed for the first time with 'Address already in use' exception. Changing it to 18081 resolve the issue. Just wanted to know if this is expected, or should the defaul

Re: Adding new data anonymization processor bundle

2018-06-20 Thread Mike Thomsen
Andy, You raise a great point about considering the provenance. Unless there's a way to exclude attributes from provenance tracking, I think we'd need to force the issue by not allowing attributes to be an input source for expression language. That's the only way to kinda force people to think "he

Re: [VOTE] Release Apache NiFi 1.7.0

2018-06-20 Thread Matt Burgess
+1 (binding) Verified all artifacts, full build with contrib-check, verified the Hive 3 NAR is not in the assembly unless the include-hive3 profile is activated, also ran through various flows to exercise Hive 3 and PutORC functionality (and their associated Record Readers, Writers, and intermedia

Re: Adding new data anonymization processor bundle

2018-06-20 Thread Andy LoPresto
Sivaprasanna, Thanks for joining this effort. I don’t recall what’s on the existing Jira, but please be very aware of the challenges in data anonymization and the various threat models — de-anonymizing data can lead to the leak of PII, EPHI, PCI data, etc. In some cases, it can even lead to phy

Re: Adding new data anonymization processor bundle

2018-06-20 Thread Sivaprasanna
Wow.. I dint realize there was a JIRA already. I'm interested and would be happy to contribute my time & efforts on this. On Wed, Jun 20, 2018 at 10:34 PM, Matt Burgess wrote: > I think is a great idea, I filed a Jira [1] a while ago in case > someone wanted to start working on it (or in case I

Re: Adding new data anonymization processor bundle

2018-06-20 Thread Matt Burgess
I think is a great idea, I filed a Jira [1] a while ago in case someone wanted to start working on it (or in case I got a chance). It mentions ARX but any Apache-friendly implementation is of course welcome. I think it should be in its own bundle as it is functionality separate from all our other b

Re: Adding new data anonymization processor bundle

2018-06-20 Thread Mike Thomsen
There's a framework called ARX that could very useful for this. The only question you have is how compliant it would be with different sets of distinct legal requirements for privacy handling. In the absence of strong legal guidance, I'd say err on the side of complying with health care regulations

Adding new data anonymization processor bundle

2018-06-20 Thread Sivaprasanna
With data becoming more critical and substantial to business development, new stringent regulations & law are getting introduced (GDPR being a recent example), I've been spending some time lately doing research on data anonymization and after some hefty thinking, I finally decided to go ahead with

Re: [VOTE] Release Apache NiFi 1.7.0

2018-06-20 Thread Mike Thomsen
+1 binding. Everything seemed to match when I checked the sums, looked at the legal documentation and I tried a deliberately cumbersome Mongo-based flow and it worked just fine for me. Didn't try enabling security. On Wed, Jun 20, 2018 at 3:16 AM Andy LoPresto wrote: > Hello, > > I am pleased to

Re: Upgrading NiFi Registry

2018-06-20 Thread Bryan Bende
Ah good point Mark... yes the old db properties are only need this first time so that it can auto-migrate the old DB to the new one, after that you don't need the old properties anymore. Thanks Kevin! On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran wrote: > Thanks Mark and Bryan. I will add a NiFi

Re: Upgrading NiFi Registry

2018-06-20 Thread Kevin Doran
Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2 migration guide to include these steps as part of updating the site with news of the new release. Thanks, Kevin From: Mark Bean Sent: Wednesday, June 20, 2018 8:53:39 AM To: dev@nifi.apache.org Subject

Re: Upgrading NiFi Registry

2018-06-20 Thread Mark Bean
Thanks Bryan. There is actually another step not explicitly mentioned. At least for 0.1.0 -> 0.2.0, I needed to modify the nifi-registry.properties file as well. The 0.2.0 version has new properties/values not in the 0.1.0. And, I had to set the following for the database (using values from 0.1.0).

Re: Upgrading NiFi Registry

2018-06-20 Thread Bryan Bende
Mark, The database directory and flow storage directory are where all the data are. By default these are created in the root of NiFi Registry, so depending how you want to set it up you could move those directories to the new install, or you could set them up to be external locations so you don't

Upgrading NiFi Registry

2018-06-20 Thread Mark Bean
How does one upgrade the NiFi Registry? After unpacking the .tar.gz file, how does one get all the flows registered in a previous version of NiFi Registry into the newly installed version? And, how does one ensure all the policies transfer as well? Thanks, Mark

Re: Apache NiFi 1.7.0 RC1 Release Helper Guide

2018-06-20 Thread Bryan Bende
Correct that should all be fine, mainly there shouldn't be any differences in any module/src path. On Wed, Jun 20, 2018 at 10:48 AM, Mike Thomsen wrote: > I took your suggestion and got this: > > Only in nifi: .git > > Only in nifi: .gitignore > > Only in temp/nifi-1.7.0/: DEPENDENCIES > > Only i

Re: Apache NiFi 1.7.0 RC1 Release Helper Guide

2018-06-20 Thread Mike Thomsen
I took your suggestion and got this: Only in nifi: .git Only in nifi: .gitignore Only in temp/nifi-1.7.0/: DEPENDENCIES Only in nifi/nifi-nar-bundles/nifi-standard-services/nifi-kerberos-credentials-service-api: .gitignore Only in nifi/nifi-nar-bundles/nifi-standard-services/nifi-kerberos-cred

Re: Apache NiFi 1.7.0 RC1 Release Helper Guide

2018-06-20 Thread Bryan Bende
Others may know a better way to do this, but the only way I know to truly verify the commit id is something like the following: git clone https://git-wip-us.apache.org/repos/asf/nifi.git git -C nifi checkout diff --brief -r For verifying the RC was branched off the correct git commit id, you l

Re: Securing Nifi registry with nginx Error

2018-06-20 Thread Mike Thomsen
> I am working on an ubuntu server. I do not have the possibility to generate the keychain and to access the graphical interface of nifi Where did you get the certificates if you are not able to generate the keychain yourself? It looks like whatever server cert you use for nginx and for the regist

Re: Apache NiFi 1.7.0 RC1 Release Helper Guide

2018-06-20 Thread Kevin Doran
Hi Mike, These values are in the VOTE email:https://lists.apache.org/thread.html/d8bfef873317c5f681a5deb226d9dd9483aec56a7abc9a72090cb570@ Cheers,Kevin On Wed, Jun 20, 2018 at 6:55 AM -0700, "Mike Thomsen" wrot

Re: Apache NiFi 1.7.0 RC1 Release Helper Guide

2018-06-20 Thread Mike Thomsen
Do we store these values somewhere in the zip? # Verify the git commit ID is correct # Verify the RC was branched off the correct git commit ID On Wed, Jun 20, 2018 at 3:16 AM Andy LoPresto wrote: > Hello Apache NiFi community, > > Please find the associated guidance to help those interested i

Re: Setting values as System properties

2018-06-20 Thread James Wing
I agree it is not what we would hope for. But I have not found any information to contradict you. The proxy setting we use now for Google Cloud Storage apparently does not apply to Google Cloud PubSub. What user experience would you propose? It seems reasonable that a user might be able to conf

Securing Nifi registry with nginx Error

2018-06-20 Thread amiraprd
I followed this tutorial to set up a secure version of Nifi registry: https://community.hortonworks.com/content/kbentry/170966/setting-up-a-secure-apache-nifi-registry.html I am working on an ubuntu server. I do not have the possibility to generate the keychain and to access the graphical interf

[VOTE] Release Apache NiFi 1.7.0

2018-06-20 Thread Andy LoPresto
Hello, I am pleased to be calling this vote for the source release of Apache NiFi nifi-1.7.0. The source zip, including signatures, digests, etc. can be found at: https://repository.apache.org/content/repositories/orgapachenifi-1127 and https://dist.apache.org/repos/dist/dev/nifi/nifi-1.7.0 T

Apache NiFi 1.7.0 RC1 Release Helper Guide

2018-06-20 Thread Andy LoPresto
Hello Apache NiFi community, Please find the associated guidance to help those interested in validating/verifying the release so they can vote. # Download latest KEYS file: https://dist.apache.org/repos/dist/dev/nifi/KEYS # Import keys file: gpg --import KEYS # [optional] Clear out local maven