save variables to template when creating a template
Hi guys, I've been using the new variable registry feature in 1.4.0 and I like it very much. But when I define a variable in a processor group and try to create a template out out it, I noticed that the variable I defined is not included in the template after I import the template to another NIFI instance. Does anyone know if it's possible to keep the variable with the template? Thanks. /Ben
Re: Request: Add my account to the JIRA contributors list
Hi Yuri, I've added you to JIRA contributor list, you should be able to assign yourself now. Thanks for your contributions to enhance NiFi UX! Koji On Wed, Oct 11, 2017 at 3:07 AM, Yuri <1969yuri1...@gmail.com> wrote: > Hello, > I'd like to be able to assign JIRA issues to myself. > > My JIRA account uses this particular email address. > > Thanks in advance.
Re: SSLContextService Issue on Minifi
Hi Ryan, I believe this is the same core issue as described in MINIFI-403 [1]. This was addressed via PR #91 [2] where another user reported similar issues. The root cause of this was duplicate libraries that were treated in the system scope as a bundle and precluded the bundled versions from being used. Those incorrect and duplicate items were removed. If you are in a position to do so and interested, I would encourage you to perform a build and verify that it resolves your issue. If not, this fix will be in the next release which we are like due for in the near future. Sorry for the hiccup and let us know if you decide to attempt a build. --aldrin [1] https://issues.apache.org/jira/browse/MINIFI-403 [2] https://github.com/apache/nifi-minifi/pull/91 On Tue, Oct 10, 2017 at 4:52 PM, rpersaud wrote: > I have encountered a similar issue with ListenTCP, NiFi 1.3 and Minifi > 0.2.0: > > 2017-10-10 20:24:55,184 ERROR [main] o.apache.nifi.controller. > FlowController > Unable to start ListenTCP[id=f3feecbe-699d-34c5--] due to > java.lang.IllegalStateException: Processor ListenTCP is not in a valid > state > due to ['SSL Context Service' validated against > '71b53c95-b604-38c7--' is invalid because > StandardSSLContextService - 0.2.0 from org.apache.nifi.minifi - > minifi-ssl-context-service-nar is not compatible with SSLContextService - > unversioned from default - system] > > Looking at StandardSSLContextService in Github, both 1.3 and 1.2.0 (what > Minifi 0.2.0 is using) are the same version. Here is my configuration with > the sensitive bits removed: > > MiNiFi Config Version: 3 > Flow Controller: > name: LoadS3 > comment: '' > Core Properties: > flow controller graceful shutdown period: 10 sec > flow service write delay interval: 500 ms > administrative yield duration: 30 sec > bored yield duration: 10 millis > max concurrent threads: 1 > variable registry properties: '' > FlowFile Repository: > partitions: 256 > checkpoint interval: 2 mins > always sync: false > Swap: > threshold: 2 > in period: 5 sec > in threads: 1 > out period: 5 sec > out threads: 4 > Content Repository: > content claim max appendable size: 10 MB > content claim max flow files: 100 > always sync: false > Provenance Repository: > provenance rollover time: 1 min > Component Status Repository: > buffer size: 1440 > snapshot frequency: 1 min > Security Properties: > keystore: '' > keystore type: '' > keystore password: '' > key password: '' > truststore: '' > truststore type: '' > truststore password: '' > ssl protocol: '' > Sensitive Props: > key: > algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL > provider: BC > Processors: > - id: ec6fd775-8998-36f3-- > name: CompressContent > class: org.apache.nifi.processors.standard.CompressContent > max concurrent tasks: 1 > scheduling strategy: TIMER_DRIVEN > scheduling period: 0 sec > penalization period: 30 sec > yield period: 1 sec > run duration nanos: 0 > auto-terminated relationships list: > - failure > Properties: > Compression Format: gzip > Compression Level: '9' > Mode: compress > Update Filename: 'true' > - id: f3feecbe-699d-34c5-- > name: ListenTCP > class: org.apache.nifi.processors.standard.ListenTCP > max concurrent tasks: 1 > scheduling strategy: TIMER_DRIVEN > scheduling period: 0 sec > penalization period: 30 sec > yield period: 1 sec > run duration nanos: 0 > auto-terminated relationships list: [] > Properties: > Character Set: UTF-8 > Client Auth: NONE > Local Network Interface: > Max Batch Size: '1' > Max Number of TCP Connections: '2' > Max Size of Message Queue: '1' > Max Size of Socket Buffer: 1 MB > Message Delimiter: \n > Port: '1515' > Receive Buffer Size: 65507 B > SSL Context Service: 71b53c95-b604-38c7-- > - id: 96b4f9d1-5d24-3386-- > name: MergeContent > class: org.apache.nifi.processors.standard.MergeContent > max concurrent tasks: 1 > scheduling strategy: TIMER_DRIVEN > scheduling period: 0 sec > penalization period: 30 sec > yield period: 1 sec > run duration nanos: 0 > auto-terminated relationships list: > - failure > - original > Properties: > Attribute Strategy: Keep All Unique Attributes > Compression Level: '1' > Correlation Attribute Name: > Delimiter Strategy: Text > Demarcator File: |2+ > > Footer File: > Header File: > Keep Path: 'false' > Max Bin Age: 60 sec > Maximum Group Size: > Maximum Number of Entries: '100' > Maximum number of Bins: '5' > Merge Format: Binary Concatenation > Merge Strategy: Bin-Packing Algorithm > Minimum Group Size: 0 B > Minimum Number of Entries: '100' > - id: d45e9378-054d-33fb-- > name: PutS3Object > class: org.apache.nifi.p
Request: Add my account to the JIRA contributors list
Hello, I'd like to be able to assign JIRA issues to myself. My JIRA account uses this particular email address. Thanks in advance.
Re: SSLContextService Issue on Minifi
I have encountered a similar issue with ListenTCP, NiFi 1.3 and Minifi 0.2.0:
2017-10-10 20:24:55,184 ERROR [main] o.apache.nifi.controller.FlowController
Unable to start ListenTCP[id=f3feecbe-699d-34c5--] due to
java.lang.IllegalStateException: Processor ListenTCP is not in a valid state
due to ['SSL Context Service' validated against
'71b53c95-b604-38c7--' is invalid because
StandardSSLContextService - 0.2.0 from org.apache.nifi.minifi -
minifi-ssl-context-service-nar is not compatible with SSLContextService -
unversioned from default - system]
Looking at StandardSSLContextService in Github, both 1.3 and 1.2.0 (what
Minifi 0.2.0 is using) are the same version. Here is my configuration with
the sensitive bits removed:
MiNiFi Config Version: 3
Flow Controller:
name: LoadS3
comment: ''
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
variable registry properties: ''
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 2
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Provenance Repository:
provenance rollover time: 1 min
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: ''
keystore type: ''
keystore password: ''
key password: ''
truststore: ''
truststore type: ''
truststore password: ''
ssl protocol: ''
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- id: ec6fd775-8998-36f3--
name: CompressContent
class: org.apache.nifi.processors.standard.CompressContent
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
- failure
Properties:
Compression Format: gzip
Compression Level: '9'
Mode: compress
Update Filename: 'true'
- id: f3feecbe-699d-34c5--
name: ListenTCP
class: org.apache.nifi.processors.standard.ListenTCP
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list: []
Properties:
Character Set: UTF-8
Client Auth: NONE
Local Network Interface:
Max Batch Size: '1'
Max Number of TCP Connections: '2'
Max Size of Message Queue: '1'
Max Size of Socket Buffer: 1 MB
Message Delimiter: \n
Port: '1515'
Receive Buffer Size: 65507 B
SSL Context Service: 71b53c95-b604-38c7--
- id: 96b4f9d1-5d24-3386--
name: MergeContent
class: org.apache.nifi.processors.standard.MergeContent
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
- failure
- original
Properties:
Attribute Strategy: Keep All Unique Attributes
Compression Level: '1'
Correlation Attribute Name:
Delimiter Strategy: Text
Demarcator File: |2+
Footer File:
Header File:
Keep Path: 'false'
Max Bin Age: 60 sec
Maximum Group Size:
Maximum Number of Entries: '100'
Maximum number of Bins: '5'
Merge Format: Binary Concatenation
Merge Strategy: Bin-Packing Algorithm
Minimum Group Size: 0 B
Minimum Number of Entries: '100'
- id: d45e9378-054d-33fb--
name: PutS3Object
class: org.apache.nifi.processors.aws.s3.PutS3Object
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list:
- failure
- success
Properties:
AWS Credentials Provider service:
Access Key: REMOVED
Bucket: REMOVED
Communications Timeout: 30 secs
Content Type:
Credentials File:
Endpoint Override URL:
Expiration Time Rule:
FullControl User List: ${s3.permissions.full.users}
Multipart Part Size: 5 GB
Multipart Threshold: 5 GB
Multipart Upload AgeOff Interval: 60 min
Multipart Upload Max Age Threshold: 7 days
Object Key:
${now():format('-MM-dd'):prepend('dt='):append('/'):append(${filename}):prepend('logs/')}
Owner: ${s3.owner}
Proxy Host:
Proxy Host Port:
Read ACL User List: ${s3.permissions.readacl.users}
Read Permission User List: ${s3.permissions.read.users}
Region: us-east-1
SSL Context Service:
Secret Key: REMOVED
Signer Override: Default Signature
Re: Troubleshooting Apache NiFi 1.4.0 Authorizations
Thanks for replying back. Can you ensure your browser cache is cleared? If
that's not it, it's possible your running an older version of Chrome that
is incompatible.
Matt
On Tue, Oct 10, 2017 at 4:10 PM, Brian Ghigiarelli
wrote:
> Thanks for the quick response, Matt! Unfortunately, it wasn't letting me
> update the Policies either, but your response did remind to use Firefox vs.
> Chrome. Turns out that when I was clicking on the hamburger menu >
> Policies, Chrome was throwing a JS error on nf-canvas-all.js?1.4.0:47:
> 21377
> with "undefined is not a function" Looks like it maps to some
> K.resource.startsWith("/policies") line. Works fine in Firefox, and I can
> add all the policies I want through there.
>
> Thanks again,
> Brian
>
> On Tue, Oct 10, 2017 at 3:58 PM Matt Gilman
> wrote:
>
> > Brian,
> >
> > NiFi will only grant permissions to the components in the canvas if there
> > is an existing flow.xml.gz in your conf directory. This is due to
> lifecycle
> > constraints when clustered. This is described in the link you provided if
> > you scroll down a little bit and look for the (i) info icon.
> >
> > Thanks
> >
> > Matt
> >
> > On Tue, Oct 10, 2017 at 3:47 PM, Brian Ghigiarelli
> > wrote:
> >
> > > With a clean install of Apache NiFi 1.4.0 using certificate-based
> > > authentication through file providers, I am able to login to the NiFi
> > > canvas, but it is not authorizing me (the Initial Admin Identity) to
> > > perform any tasks. I'm following the first example configuration at
> > > https://nifi.apache.org/docs/nifi-docs/html/administration-
> > > guide.html#initial-admin-identity.
> > > I
> > > can see that the authorizations.xml file is being generated with the
> same
> > > UUID as the user that is generated in users.xml, and that user has
> > entries
> > > in the following policies:
> > >
> > >- /flow (R)
> > >- /restricted-components (W)
> > >- /tenants (R)
> > >- /tenants (W)
> > >- /policies (R)
> > >- /policies (W)
> > >- /controller (R)
> > >- /controller (W)
> > >
> > > Based on the docs, a few pieces of the authorizers.xml file changed
> from
> > > 1.3.0 to 1.4.0, and I changed the nifi.properties
> > > nifi.security.user.authorizer to managed-authorizer. Any ideas what I'm
> > > missing here? Or where to begin debugging?
> > >
> > > Thanks,
> > > Brian
> > >
> >
>
NIFI: how to update config data from custpm processor
n my custom processor i need to update config file(is placed in one of nifi
folder not original config it is created by me :D ) based on data which i
get from upstream connection but i can't get nor error neither desired
result what should i do,
1.is there any way i can controll upstream connection flowfile destination i
mean when i make debugging i saw that fileQueue.drainTo(file, batchSize); in
this part file was null
2.here is one more thing i am interested in, on this line session.commit() i
always get errors and whole operation is failed ,should i clean fileQueue
and then make session.commit()?
FlowFile flowfile; @Override public void onTrigger(ProcessContext context,
ProcessSession session) throws ProcessException { final String
conflictResponse = context.getProperty(CONFLICT_RESOLUTION).getValue();
final ArrayList value = new ArrayList<>();
flowfile = session.get();
if (flowfile == null) {
return;
}
value.add(flowfile.getAttribute("filename"));
session.remove(flowfile);
final File directory = new
File(context.getProperty(DIRECTORY).evaluateAttributeExpressions().getValue());
final boolean keepingSourceFile =
context.getProperty(KEEP_SOURCE_FILE).asBoolean();
final ComponentLog logger = getLogger();
if (fileQueue.size() < 100) {
final long pollingMillis =
context.getProperty(POLLING_INTERVAL).asTimePeriod(TimeUnit.MILLISECONDS);
if ((queueLastUpdated.get() < System.currentTimeMillis() -
pollingMillis) && listingLock.tryLock()) {
try {
final Set listing = performListing(directory,
fileFilterRef.get(),
context.getProperty(RECURSE).asBoolean().booleanValue());
queueLock.lock();
try {
listing.removeAll(inProcess);
if (!keepingSourceFile) {
listing.removeAll(recentlyProcessed);
}
fileQueue.clear();
fileQueue.addAll(listing);
queueLastUpdated.set(System.currentTimeMillis());
recentlyProcessed.clear();
if (listing.isEmpty()) {
context.yield();
}
} finally {
queueLock.unlock();
}
} finally {
listingLock.unlock();
}
}
}
final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
final List file = new ArrayList<>(batchSize);
queueLock.lock();
try {
fileQueue.drainTo(file, batchSize);
if (file.isEmpty()) {
return;
} else {
inProcess.addAll(file);
}
} finally {
queueLock.unlock();
}
//make xml parsing
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
try {
dBuilder = dbFactory.newDocumentBuilder();
} catch (ParserConfigurationException e) {
e.printStackTrace();
}
try {
File f = file.get(0);
doc = dBuilder.parse(f);
} catch (IOException e) {
e.printStackTrace();
} catch (org.xml.sax.SAXException e) {
e.printStackTrace();
}
NodeList nList = doc.getElementsByTagName("localAttributes");
for (int temp = 0; temp < nList.getLength(); temp++) {
Node nNode = nList.item(temp);
if (nNode.getNodeType() == Node.ELEMENT_NODE) {
Element eElement = (Element) nNode;
start =
eElement.getElementsByTagName("start").item(0).getTextContent();
startDate =
eElement.getElementsByTagName("startDate").item(0).getTextContent();
endDate =
eElement.getElementsByTagName("endDate").item(0).getTextContent();
patch =
eElement.getElementsByTagName("patch").item(0).getTextContent();
runAs =
eElement.getElementsByTagName("runAs").item(0).getTextContent();
makeVersion =
eElement.getElementsByTagName("makeVersion").item(0).getTextContent();
///parameter =
eElement.getElementsByTagName("parameter").item(0).getTextContent();
}
}
final ListIterator itr = file.listIterator();
FlowFile flowFile1 = null;
try {
final Path directoryPath = directory.toPath();
while (itr.hasNext()) {
final File files = itr.next();
final Path filePath = files.toPath();
final Path relativePath =
directoryPath.relativize(filePath.getParent());
String relativePathString = relativePath.toString() + "/";
if (relativePathString.isEmpty()) {
relativePathString = "./";
}
final Path absPath = filePath.toAbsolutePath();
final String absPathString = absPath.getParent().toString() +
"/";
flowFile1 = session.create();
final long importStart = System.nanoTime();
flowFile1 = session.importFrom(filePath, k
Re: Troubleshooting Apache NiFi 1.4.0 Authorizations
Thanks for the quick response, Matt! Unfortunately, it wasn't letting me
update the Policies either, but your response did remind to use Firefox vs.
Chrome. Turns out that when I was clicking on the hamburger menu >
Policies, Chrome was throwing a JS error on nf-canvas-all.js?1.4.0:47:21377
with "undefined is not a function" Looks like it maps to some
K.resource.startsWith("/policies") line. Works fine in Firefox, and I can
add all the policies I want through there.
Thanks again,
Brian
On Tue, Oct 10, 2017 at 3:58 PM Matt Gilman wrote:
> Brian,
>
> NiFi will only grant permissions to the components in the canvas if there
> is an existing flow.xml.gz in your conf directory. This is due to lifecycle
> constraints when clustered. This is described in the link you provided if
> you scroll down a little bit and look for the (i) info icon.
>
> Thanks
>
> Matt
>
> On Tue, Oct 10, 2017 at 3:47 PM, Brian Ghigiarelli
> wrote:
>
> > With a clean install of Apache NiFi 1.4.0 using certificate-based
> > authentication through file providers, I am able to login to the NiFi
> > canvas, but it is not authorizing me (the Initial Admin Identity) to
> > perform any tasks. I'm following the first example configuration at
> > https://nifi.apache.org/docs/nifi-docs/html/administration-
> > guide.html#initial-admin-identity.
> > I
> > can see that the authorizations.xml file is being generated with the same
> > UUID as the user that is generated in users.xml, and that user has
> entries
> > in the following policies:
> >
> >- /flow (R)
> >- /restricted-components (W)
> >- /tenants (R)
> >- /tenants (W)
> >- /policies (R)
> >- /policies (W)
> >- /controller (R)
> >- /controller (W)
> >
> > Based on the docs, a few pieces of the authorizers.xml file changed from
> > 1.3.0 to 1.4.0, and I changed the nifi.properties
> > nifi.security.user.authorizer to managed-authorizer. Any ideas what I'm
> > missing here? Or where to begin debugging?
> >
> > Thanks,
> > Brian
> >
>
Re: Troubleshooting Apache NiFi 1.4.0 Authorizations
Brian, NiFi will only grant permissions to the components in the canvas if there is an existing flow.xml.gz in your conf directory. This is due to lifecycle constraints when clustered. This is described in the link you provided if you scroll down a little bit and look for the (i) info icon. Thanks Matt On Tue, Oct 10, 2017 at 3:47 PM, Brian Ghigiarelli wrote: > With a clean install of Apache NiFi 1.4.0 using certificate-based > authentication through file providers, I am able to login to the NiFi > canvas, but it is not authorizing me (the Initial Admin Identity) to > perform any tasks. I'm following the first example configuration at > https://nifi.apache.org/docs/nifi-docs/html/administration- > guide.html#initial-admin-identity. > I > can see that the authorizations.xml file is being generated with the same > UUID as the user that is generated in users.xml, and that user has entries > in the following policies: > >- /flow (R) >- /restricted-components (W) >- /tenants (R) >- /tenants (W) >- /policies (R) >- /policies (W) >- /controller (R) >- /controller (W) > > Based on the docs, a few pieces of the authorizers.xml file changed from > 1.3.0 to 1.4.0, and I changed the nifi.properties > nifi.security.user.authorizer to managed-authorizer. Any ideas what I'm > missing here? Or where to begin debugging? > > Thanks, > Brian >
Nifi:how to handle flowfile data updateing and reitreving?
0
down vote
favorite
I want to reitreve data from folder and then update it and put it back to
its' destination and i want to make this operation many times , here is my
code i try to getfile and update it , but it doesn't roll back the file and
can't reitreve data with the same filename(sometimes when i start this
processor first time it reireives data and then rolls back updated one but
then it remembers state or flowfile information i guess and doen't reitreive
same updated file) , can someone help me what should i change to make this
code work?
here is error i got :2017-10-08 21:40:55,959 ERROR [Timer-Driven Process
Thread-9] Reader.MyProcessor
MyProcessor[id=fcaf839f-015e-1000-da5d-a3256b960a67]
MyProcessor[id=fcaf839f-015e-1000-da5d-a3256b960a67] failed to process due
to java.lang.IllegalArgumentException: Cannot transfer FlowFiles that are
created in this Session back to self; rolling back session: {}
java.lang.IllegalArgumentException: Cannot transfer FlowFiles that are
created in this Session back to self at
org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1848)
at Reader.MyProcessor.onTrigger(MyProcessor.java:732)
public class MyProcessor extends AbstractProcessor {
public String start, startDate, endDate, makeVersion, runAs, patch;
@Override
public void onTrigger(final ProcessContext context, final ProcessSession
session) throws ProcessException {
final StopWatch stopWatch = new StopWatch(true);
final File directory = new
File(context.getProperty(DIRECTORY).evaluateAttributeExpressions().getValue());
final boolean keepingSourceFile =
context.getProperty(KEEP_SOURCE_FILE).asBoolean();
final String conflictResponse =
context.getProperty(CONFLICT_RESOLUTION).getValue();
final Integer maxDestinationFiles = 30;
final ComponentLog logger = getLogger();
if (fileQueue.size() < 100) {
final long pollingMillis =
context.getProperty(POLLING_INTERVAL).asTimePeriod(TimeUnit.MILLISECONDS);
if ((queueLastUpdated.get() < System.currentTimeMillis() -
pollingMillis) && listingLock.tryLock()) {
try {
final ArrayList listing =
performListing(directory, fileFilterRef.get(),
context.getProperty(RECURSE).asBoolean().booleanValue());
queueLock.lock();
try {
listing.removeAll(inProcess);
if (!keepingSourceFile) {
listing.removeAll(recentlyProcessed);
}
fileQueue.clear();
fileQueue.addAll(listing);
queueLastUpdated.set(System.currentTimeMillis());
recentlyProcessed.clear();
if (listing.isEmpty()) {
context.yield();
}
} finally {
queueLock.unlock();
}
} finally {
listingLock.unlock();
}
}
}
final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
final List files = new ArrayList<>(batchSize);
queueLock.lock();
try {
fileQueue.drainTo(files, batchSize);
if (files.isEmpty()) {
return;
} else {
inProcess.addAll(files);
}
} finally {
queueLock.unlock();
}
//make xml parsing
DocumentBuilderFactory dbFactory =
DocumentBuilderFactory.newInstance();
try {
dBuilder = dbFactory.newDocumentBuilder();
} catch (ParserConfigurationException e) {
e.printStackTrace();
}
try {
File f = files.get(0);
doc = dBuilder.parse(f);
} catch (IOException e) {
e.printStackTrace();
} catch (org.xml.sax.SAXException e) {
e.printStackTrace();
}
NodeList nList = doc.getElementsByTagName("localAttributes");
for (int temp = 0; temp < nList.getLength(); temp++) {
Node nNode = nList.item(temp);
if (nNode.getNodeType() == Node.ELEMENT_NODE) {
Element eElement = (Element) nNode;
start =
eElement.getElementsByTagName("start").item(0).getTextContent();
startDate =
eElement.getElementsByTagName("startDate").item(0).getTextContent();
endDate =
eElement.getElementsByTagName("endDate").item(0).getTextContent();
patch =
eElement.getElementsByTagName("patch").item(0).getTextContent();
runAs =
eElement.getElementsByTagName("runAs").item(0).getTextContent();
makeVersion =
eElement.getElementsByTagName("makeVersion").item(0).getTextContent();
}
}
final ListIterator it
Nifi:how to handle flowfile data updateing and reitreving?
I want to reitreve data from folder and then update it and put it back to
its' destination and i want to make this operation many times , here is my
code i try to getfile and update it , but it doesn't roll back the file and
can't reitreve data with the same filename(sometimes when i start this
processor first time it reireives data and then rolls back updated one but
then it remembers state or flowfile information i guess and doen't reitreive
same updated file) , can someone help me what should i change to make this
code work?
here is error i got :2017-10-08 21:40:55,959 ERROR [Timer-Driven Process
Thread-9] Reader.MyProcessor
MyProcessor[id=fcaf839f-015e-1000-da5d-a3256b960a67]
MyProcessor[id=fcaf839f-015e-1000-da5d-a3256b960a67] failed to process due
to java.lang.IllegalArgumentException: Cannot transfer FlowFiles that are
created in this Session back to self; rolling back session: {}
java.lang.IllegalArgumentException: Cannot transfer FlowFiles that are
created in this Session back to self at
org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1848)
at Reader.MyProcessor.onTrigger(MyProcessor.java:732)
public class MyProcessor extends AbstractProcessor {
public String start, startDate, endDate, makeVersion, runAs, patch;
@Override
public void onTrigger(final ProcessContext context, final ProcessSession
session) throws ProcessException {
final StopWatch stopWatch = new StopWatch(true);
final File directory = new
File(context.getProperty(DIRECTORY).evaluateAttributeExpressions().getValue());
final boolean keepingSourceFile =
context.getProperty(KEEP_SOURCE_FILE).asBoolean();
final String conflictResponse =
context.getProperty(CONFLICT_RESOLUTION).getValue();
final Integer maxDestinationFiles = 30;
final ComponentLog logger = getLogger();
if (fileQueue.size() < 100) {
final long pollingMillis =
context.getProperty(POLLING_INTERVAL).asTimePeriod(TimeUnit.MILLISECONDS);
if ((queueLastUpdated.get() < System.currentTimeMillis() -
pollingMillis) && listingLock.tryLock()) {
try {
final ArrayList listing =
performListing(directory, fileFilterRef.get(),
context.getProperty(RECURSE).asBoolean().booleanValue());
queueLock.lock();
try {
listing.removeAll(inProcess);
if (!keepingSourceFile) {
listing.removeAll(recentlyProcessed);
}
fileQueue.clear();
fileQueue.addAll(listing);
queueLastUpdated.set(System.currentTimeMillis());
recentlyProcessed.clear();
if (listing.isEmpty()) {
context.yield();
}
} finally {
queueLock.unlock();
}
} finally {
listingLock.unlock();
}
}
}
final int batchSize = context.getProperty(BATCH_SIZE).asInteger();
final List files = new ArrayList<>(batchSize);
queueLock.lock();
try {
fileQueue.drainTo(files, batchSize);
if (files.isEmpty()) {
return;
} else {
inProcess.addAll(files);
}
} finally {
queueLock.unlock();
}
//make xml parsing
DocumentBuilderFactory dbFactory =
DocumentBuilderFactory.newInstance();
try {
dBuilder = dbFactory.newDocumentBuilder();
} catch (ParserConfigurationException e) {
e.printStackTrace();
}
try {
File f = files.get(0);
doc = dBuilder.parse(f);
} catch (IOException e) {
e.printStackTrace();
} catch (org.xml.sax.SAXException e) {
e.printStackTrace();
}
NodeList nList = doc.getElementsByTagName("localAttributes");
for (int temp = 0; temp < nList.getLength(); temp++) {
Node nNode = nList.item(temp);
if (nNode.getNodeType() == Node.ELEMENT_NODE) {
Element eElement = (Element) nNode;
start =
eElement.getElementsByTagName("start").item(0).getTextContent();
startDate =
eElement.getElementsByTagName("startDate").item(0).getTextContent();
endDate =
eElement.getElementsByTagName("endDate").item(0).getTextContent();
patch =
eElement.getElementsByTagName("patch").item(0).getTextContent();
runAs =
eElement.getElementsByTagName("runAs").item(0).getTextContent();
makeVersion =
eElement.getElementsByTagName("makeVersion").item(0).getTextContent();
}
}
final ListIterator itr = files.listIterator()
Troubleshooting Apache NiFi 1.4.0 Authorizations
With a clean install of Apache NiFi 1.4.0 using certificate-based authentication through file providers, I am able to login to the NiFi canvas, but it is not authorizing me (the Initial Admin Identity) to perform any tasks. I'm following the first example configuration at https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#initial-admin-identity. I can see that the authorizations.xml file is being generated with the same UUID as the user that is generated in users.xml, and that user has entries in the following policies: - /flow (R) - /restricted-components (W) - /tenants (R) - /tenants (W) - /policies (R) - /policies (W) - /controller (R) - /controller (W) Based on the docs, a few pieces of the authorizers.xml file changed from 1.3.0 to 1.4.0, and I changed the nifi.properties nifi.security.user.authorizer to managed-authorizer. Any ideas what I'm missing here? Or where to begin debugging? Thanks, Brian
Re: FuzzyHashContent/CompareFuzzyHash processor
Hi Andy, I am having multiple lines in the file. For example Col1 Col2 Col3 Col4 Col5 Test1Test2 Test3 Test4 Test5 Data1 Data2Data3Data4Data5 .. .. .. In the output file I want to write all the fields where there will be a fuzzy matching between Col2 and Col3. Regards, Shankha -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
