Re: FuzzyHashContent/CompareFuzzyHash processor

[Andy LoPresto]

Hi Shankha,

The fuzzy hash processors operate on the content of the flowfile. You would 
first use a processor to ingest the “data file” content. This could be 
something like GetFile, GetHDFS, GetSFTP, InvokeHTTP, etc. depending on the 
source of the file. Once that step is done, the flowfile content will contain 
the data file bytes. If you want to perform the fuzzy hash calculation on the 
entire data file content, you can connect the success relationship from the 
ingest processor directly to FuzzyHashContent, and the resulting flowfile will 
contain an attribute with the calculated hash value. If you want to perform the 
calculation over only specific parts of the flowfile, you can use a processor 
to manipulate the content, for example EvaluateJsonPath, EvaluateXPath, 
ReplaceText, etc.

You can see an example flow which uses these processors in slide 21 of a 
presentation [1] André Fucs de Miranda and I gave recently, and André has 
published the flow XML here [2].

[1] 
https://github.com/alopresto/slides/blob/master/dws_sydney_2017/the_power_of_intelligent_flows.pdf
 

[2] https://github.com/fluenda/dataworks_summit_iot_botnet

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 6, 2017, at 4:27 AM, shankhamajumdar  
> wrote:
> 
> Hi,
> 
> I want to implement fuzzy logic on some fields in a data file using NiFi. I
> am trying to use  FuzzyHashContent/CompareFuzzyHash processor but not sure
> how to implement the flow. Can you please provide me an example?
> 
> Regards,
> Shankha
> 
> 
> 
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Nifi:How can i remove “.” from “.filename” in nifi custom processor

[sally]

actually  it never  fiishes  writing process  i mean  exceptions  like this 
can't rename  file ".conf.xml"



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Can't use renameTo in nifi custom processor code

[sally]

One  of the  most  important  part  of my code is  that  it  should put  file
in folder.   i use  this  code  for  renaming ".conf.xml"   to  "conf.xml" 
but it  doesn't  work  properly:
 final Path rootDirPath =
get("C://Users//user//Desktop//try2//nifi-1.3.0//1");
final Path tempCopyFile =
rootDirPath.resolve("."+flowFile.getAttribute(CoreAttributes.FILENAME.key()));
final Path dotCopyFile =tempCopyFile;
Path tempDotCopyFile = null;
tempDotCopyFile = dotCopyFile;

final Path
copyFile=rootDirPath.resolve(flowFile.getAttribute(CoreAttributes.FILENAME.key()));
Path finalCopyFile = copyFile;
if (!Files.exists(rootDirPath)) {
if (true) {
Files.createDirectories(rootDirPath);
}
}


final Path finalCopyFileDir = finalCopyFile.getParent();
if (Files.exists(finalCopyFileDir)) { // check if too many
files already
final int numFiles =
finalCopyFileDir.toFile().list().length;

if (numFiles >= 10) {
flowFile = session.penalize(flowFile);
logger.warn("Penalizing {} and routing to 'failure'
because the output directory {} has {} files, which exceeds the "
+ "configured maximum number of files", new
Object[]{flowFile, finalCopyFileDir, numFiles});
session.transfer(flowFile, REL_FAILURE);
return;
}
}
session.exportTo(flowFile,dotCopyFile,false);

//dotCopyFile.toFile().renameTo(finalCopyFile.toFile());

logger.info("added {} to flow", new
Object[]{flowFile,flowFile1});
boolean renamed = false;
for (int i = 0; i < 10; i++) { // try rename up to 10 times.
if
(dotCopyFile.toFile().renameTo(finalCopyFile.toFile())) {
renamed = true;
break;// rename was successful
}
Thread.sleep(100L);// try waiting a few ms to let
whatever might cause rename failure to resolve
}

if (!renamed) {
if (Files.exists(dotCopyFile) &&
dotCopyFile.toFile().delete()) {
logger.debug("Deleted dot copy file {}", new
Object[]{dotCopyFile});
}
throw new ProcessException("Could not rename: " +
dotCopyFile);
} else {
logger.info("Produced copy of {} at location {}", new
Object[]{flowFile, finalCopyFile});
}



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi:How can i remove “.” from “.filename” in nifi custom processor

[sally]

how can i see  that  flowfile  write is  already  finished  i have  renameTo 
code  like  this  :
 final Path rootDirPath =
get("C://Users//user//Desktop//try2//nifi-1.3.0//1");
final Path tempCopyFile =
rootDirPath.resolve("."+flowFile.getAttribute(CoreAttributes.FILENAME.key()));
final Path dotCopyFile =tempCopyFile;
Path tempDotCopyFile = null;
tempDotCopyFile = dotCopyFile;

final Path
copyFile=rootDirPath.resolve(flowFile.getAttribute(CoreAttributes.FILENAME.key()));
Path finalCopyFile = copyFile;
if (!Files.exists(rootDirPath)) {
if (true) {
Files.createDirectories(rootDirPath);
}
}


final Path finalCopyFileDir = finalCopyFile.getParent();
if (Files.exists(finalCopyFileDir)) { // check if too many
files already
final int numFiles =
finalCopyFileDir.toFile().list().length;

if (numFiles >= 10) {
flowFile = session.penalize(flowFile);
logger.warn("Penalizing {} and routing to 'failure'
because the output directory {} has {} files, which exceeds the "
+ "configured maximum number of files", new
Object[]{flowFile, finalCopyFileDir, numFiles});
session.transfer(flowFile, REL_FAILURE);
return;
}
}
session.exportTo(flowFile,dotCopyFile,false);

//dotCopyFile.toFile().renameTo(finalCopyFile.toFile());

logger.info("added {} to flow", new
Object[]{flowFile,flowFile1});
boolean renamed = false;
for (int i = 0; i < 10; i++) { // try rename up to 10 times.
if
(dotCopyFile.toFile().renameTo(finalCopyFile.toFile())) {
renamed = true;
break;// rename was successful
}
Thread.sleep(100L);// try waiting a few ms to let
whatever might cause rename failure to resolve
}

if (!renamed) {
if (Files.exists(dotCopyFile) &&
dotCopyFile.toFile().delete()) {
logger.debug("Deleted dot copy file {}", new
Object[]{dotCopyFile});
}
throw new ProcessException("Could not rename: " +
dotCopyFile);
} else {
logger.info("Produced copy of {} at location {}", new
Object[]{flowFile, finalCopyFile});
}



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Can't rename ".file.xml" to "file.xml" in nifi custom code

[sally]

I  have  this code  inside my  onTrigger   method in  order  to rename  file
and   then  put it in  folder  but  for unknown  reasons  i can't  do it  in 
log  file  i can't see  any  failure, error or  warn.  Mainly  i want  to
make  custom processor  which  get  xml  file from folder and  then update 
its  data, after all   sends  this data  to flowfile attributes and 
transfers  this newly created  flowfile.  then  i wnat to  update  renew
data in config.xml  file and send  it  back to  folder  but i cant  do this
, i can  send ".conf.xml" in  folder  but i can't rename it to 
"conf.xml".what should i change to make  this code  work?

 final Path rootDirPath =
get("C://Users//user//Desktop//try2//nifi-1.3.0//1");
final Path tempCopyFile =
rootDirPath.resolve("."+flowFile.getAttribute(CoreAttributes.FILENAME.key()));
final Path dotCopyFile =tempCopyFile;
Path tempDotCopyFile = null;
tempDotCopyFile = dotCopyFile;

final Path
copyFile=rootDirPath.resolve(flowFile.getAttribute(CoreAttributes.FILENAME.key()));
Path finalCopyFile = copyFile;
if (!Files.exists(rootDirPath)) {
if (true) {
Files.createDirectories(rootDirPath);
}
}


final Path finalCopyFileDir = finalCopyFile.getParent();
if (Files.exists(finalCopyFileDir)) { // check if too many
files already
final int numFiles =
finalCopyFileDir.toFile().list().length;

if (numFiles >= 10) {
flowFile = session.penalize(flowFile);
logger.warn("Penalizing {} and routing to 'failure'
because the output directory {} has {} files, which exceeds the "
+ "configured maximum number of files", new
Object[]{flowFile, finalCopyFileDir, numFiles});
session.transfer(flowFile, REL_FAILURE);
return;
}
}
session.exportTo(flowFile,dotCopyFile,false);

//dotCopyFile.toFile().renameTo(finalCopyFile.toFile());

logger.info("added {} to flow", new
Object[]{flowFile,flowFile1});
boolean renamed = false;
for (int i = 0; i < 10; i++) { // try rename up to 10 times.
if
(dotCopyFile.toFile().renameTo(finalCopyFile.toFile())) {
renamed = true;
break;// rename was successful
}
Thread.sleep(100L);// try waiting a few ms to let
whatever might cause rename failure to resolve
}

if (!renamed) {
if (Files.exists(dotCopyFile) &&
dotCopyFile.toFile().delete()) {
logger.debug("Deleted dot copy file {}", new
Object[]{dotCopyFile});
}
throw new ProcessException("Could not rename: " +
dotCopyFile);
} else {
logger.info("Produced copy of {} at location {}", new
Object[]{flowFile, finalCopyFile});
}



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

FuzzyHashContent/CompareFuzzyHash processor

[shankhamajumdar]

Hi,

I want to implement fuzzy logic on some fields in a data file using NiFi. I
am trying to use  FuzzyHashContent/CompareFuzzyHash processor but not sure
how to implement the flow. Can you please provide me an example?

Regards,
Shankha



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/