Re: Data anonymization in Nifi

2017-10-17 Thread Andy LoPresto
Vyshali, You may be interested in format preserving encryption (FPE) [1] if you need to maintain format while performing data masking. There are also methods to derive a cryptographically secure hash function from encryption [2] so that you can have “one way” data transformation and maintain a

Re: Data anonymization in Nifi

2017-10-17 Thread Mike Thomsen
Not if you use hashing. You'll get a field value like this (sha1 algorithm): c3499c2729730a7f807efb8676a92dcb6f8a3f8f For getting closer to the original data in the sort of values present, you'll need to try something like ARX. On Tue, Oct 17, 2017 at 11:53 AM, Vyshali wrote: > Hi Chris, > > Ha

Change of persistent repo in MiNiFi C++

2017-10-17 Thread Marc
Hello, In preparation of solidifying some of our persistent storage code we've moved from LevelDB to RocksDB as the default persistent storage mechanism. There should be little change for you other than the installation of RocksDB. The net benefit will be improved reliability and maintainability o

Re: Data anonymization in Nifi

2017-10-17 Thread Vyshali
Hi Chris, Hashing using executescript processor means that I should write some coding logic to do that.If so,will the format of the field will remain the same ? Please explain me with examples. Regards, Vyshali -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Data anonymization in Nifi

2017-10-17 Thread Matt Burgess
Vyshali, Building on Chris's suggestion of using ExecuteScript, you could also include the ARX JAR(s) in your Module Directory property, and then leverage all the ARX goodness [1]. In general this does seem like a good idea for a processor, I have written NIFI-4492 [2] to add an AnonymizeRecord p

Re: PutSQL Processor

2017-10-17 Thread Matt Burgess
Shankha, PutSQL works by creating batches using a PreparedStatement, then executing those batches using the incoming flow files (up to the value of the Batch Size parameter). In order to support that, auto-commit has to be false or else each statement would have to be executed individually, causin

Re: Data anonymization in Nifi

2017-10-17 Thread Chris Herssens
You can use the ExecuteScript processor for hashing some fields is for instance CSV data Regards, Chris On Tue, Oct 17, 2017 at 8:41 AM, Vyshali wrote: > Hi, > > Please suggest possible ways to do data anonymization in Nifi such that PII > data is not exposed. > Suggest suitable processors for

PutSQL Processor

2017-10-17 Thread shankhamajumdar
Hi, I am trying to insert some data in oracle table using PutSQL Processor. But getting the below error. 2017-10-17 14:53:58,409 ERROR [Timer-Driven Process Thread-1] o.apache.nifi.processors.standard.PutSQL PutSQL[id=24b8613d-015f-1000-2f70-88d0c843300b] org.apache.nifi.processors.standard.PutSQ

Data anonymization in Nifi

2017-10-17 Thread Vyshali
Hi, Please suggest possible ways to do data anonymization in Nifi such that PII data is not exposed. Suggest suitable processors for the same. Thanks in advance. Regards, Vyshali -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/