Configuration of ConsumeTwitter

2024-07-31 Thread James McMahon
I have configured a ConsumeTwitter processor as Pierre Villard shows in this example: https://www.youtube.com/watch?v=QZimEK6IAFI There isn't much to it: take the bearer token associated with your twitter developer account, drop it into the processor, set the fields you want from twitter. But I

Re: ConsumeAMQP not delivering message from RMQ to Success

2024-07-08 Thread James McMahon
> messages arrive, they will be discarded due to the processor's stopped > state. > Have you tried it with starting the processor and giving some time to the > client to get initialized and receive the messages? > > Best regards, > Peter Turcsanyi > > > On Mon, Jul 8, 2024 at 12:34 P

Re: ConsumeAMQP not delivering message from RMQ to Success

2024-07-08 Thread James McMahon
, 2024 at 1:28 PM James McMahon wrote: > I have a queue in RabbitMQ that I try to ConsumeAMQP from, configured like > this: > > Queue > detection-responses > Auto-Acknowledge Messages > false > Batch Size > 1 > Prefetch Count > 0 > Header Output Format > Co

Containerizing, clustering, Registry - how to begin?

2024-07-06 Thread James McMahon
I would like to build a containerized, scalable, highly available NiFi architecture - likely docker, likely on EC2s. I intend to have dev, int, and prod containerized groups, version controlled through NiFi Registry. I am trying to understand how NiFi clusters, Registry, and containerization get

ConsumeAMQP not delivering message from RMQ to Success

2024-07-05 Thread James McMahon
I have a queue in RabbitMQ that I try to ConsumeAMQP from, configured like this: Queue detection-responses Auto-Acknowledge Messages false Batch Size 1 Prefetch Count 0 Header Output Format Comma-Separated String Header Separator , Remove Curly Braces False BrokersNo value setHost Name

Re: Accessing File Size in ExecuteGroovyScript script

2024-06-17 Thread James McMahon
fined or zero, which prevents division.") session.transfer(ff, REL_FAILURE) return } log.info("File size: ${file_size}") On Mon, Jun 17, 2024 at 8:31 AM James McMahon wrote: > I should mention that I have also tried to access fileSize like this > without success: >

Re: Accessing File Size in ExecuteGroovyScript script

2024-06-17 Thread James McMahon
FlowFile attributes include: entryDate lineageStartDate fileSize filename path uuid On Mon, Jun 17, 2024 at 7:50 AM James McMahon wrote: > I am trying to use the file size. On the DEtails tab for my flowfile in > queue, I see that my File Size is 8.01 GB. > > I log the fo

Accessing File Size in ExecuteGroovyScript script

2024-06-17 Thread James McMahon
I am trying to use the file size. On the DEtails tab for my flowfile in queue, I see that my File Size is 8.01 GB. I log the following from this section of a Groovy script, running in an ExecuteGroovyScript processor: def ff = session.get() if (!ff) return def jsonFactory = new JsonFactory()

Re: FlattenJSON fails on large json file

2024-06-14 Thread James McMahon
ot the limiting factor. > > -Eric > > On Fri, Jun 14, 2024, 10:59 AM James McMahon wrote: > >> I have a json file, incoming.json. It is 9 GB in size. >> >> I want to flatten the json so that I can tabulate the number of times >> each

FlattenJSON fails on large json file

2024-06-14 Thread James McMahon
I have a json file, incoming.json. It is 9 GB in size. I want to flatten the json so that I can tabulate the number of times each key appears. Am using a FlattenJson 2.0.0-M2 processor, with this configuration: Separator . Flatten Mode

Re: Insufficient permissions on initial start up (NiFi 2.0)

2024-04-25 Thread James McMahon
ps! > > Matt > > On Wed, Apr 24, 2024 at 8:41 PM James McMahon > wrote: > >> Looking at the nifi-user.log, I find I am getting a Conflict response, >> Access Token not found. >> >> more ./nifi-user.log >> 2024-04-25 00:23:49,329 INFO [main] o.a.n.a.

Re: Insufficient permissions on initial start up (NiFi 2.0)

2024-04-24 Thread James McMahon
d with. Can you compare > that with the user that you've configured the policies for. Hopefully, that > will help point to where the issue is. > > Matt > > On Wed, Apr 24, 2024 at 5:03 PM James McMahon > wrote: > >> I still cannot access my own NiFi 2.0

Re: Insufficient permissions on initial start up (NiFi 2.0)

2024-04-24 Thread James McMahon
is my users.xml (nifi creates at first startup): On Wed, Apr 24, 2024 at 8:21 AM James McMahon wrote: > I'll review this closely once again when I get back to this system tonight > - thanks very much fo

Re: Insufficient permissions on initial start up (NiFi 2.0)

2024-04-24 Thread James McMahon
pace matter. Since you > are getting insufficient permissions instead of unknown user, I don’t think > that’s your problem here. Still, it may be worth checking for a mismatch in > the initial admin identity vs initial user identity vs certificate. > > > > Regards, > > > &

Insufficient permissions on initial start up (NiFi 2.0)

2024-04-23 Thread James McMahon
I am trying to start my new NiFi 2.0 installation. I have a user admin2 that has a cert. The nifi server also has a cert. Both are signed by the same CA. At start up in my browser I am denied due to insufficient privileges: Unable to view the user interface. Contact the system administrator.

Re: How to upload flow definition to new 2.0 instance?

2024-04-19 Thread James McMahon
, drag and drop a process group on the > canvas, and click the upload icon on the right of the name input. You'll be > able to select your JSON file. > > HTH > > Le jeu. 11 avr. 2024 à 02:14, James McMahon a > écrit : > >> I had developed an extensive NiFi Flow in v1.1

Re: Unable to securely connect to NiFi 2.0 instance

2024-04-19 Thread James McMahon
gt; something like this: > > > > openssl s_client -connect : -debug -cert client.pem -key > clientkey.pem -CAfile rootcert.pem > > > > This should give you a lot of details, including information from the > server that specifies which CAs it will accept for client certs. > > >

Re: Unable to securely connect to NiFi 2.0 instance

2024-04-18 Thread James McMahon
thoughts - fixing, debugging, anything? On Wed, Apr 17, 2024 at 8:44 PM James McMahon wrote: > I have installed and configured NiFi 2.0 with TLS. My nifi 2.0 instance > appears to start without errors, judging by the contents of nifi-app.log. > > When I try to access my nifi instance

Unable to securely connect to NiFi 2.0 instance

2024-04-17 Thread James McMahon
I have installed and configured NiFi 2.0 with TLS. My nifi 2.0 instance appears to start without errors, judging by the contents of nifi-app.log. When I try to access my nifi instance through its https setting in nifi.properties, I get this error in my browser: This site can’t provide a secure

How to upload flow definition to new 2.0 instance?

2024-04-10 Thread James McMahon
I had developed an extensive NiFi Flow in v1.16. I have initialized an instance of NiFi 2.0. I downloaded my flow to file NiFi_Flow.json from my 1.16 instance. But I can find no way to import this or load this to my 2.0 instance. We currently have no Registry. How can I do this?

Re: JoltTransformJSON error from NiFi, but is valid in tester

2024-02-19 Thread James McMahon
t;, > "dates": "dates", > "triage": "triage", > "payload": "payload" > } > }, > { > "operation": "default", > "spec": { > "payload&

JoltTransformJSON error from NiFi, but is valid in tester

2024-02-18 Thread James McMahon
I have this JSON as flowfile content: { "dates" : { "date_file" : "20240115184407", "ingested" : "20240217175748", "latest_date" : "1980", "earliest_date" : "1980", "date_info" : "MMDD" }, "parents" : { "md5" : "86107362084b86ea64dc33dfde5e14ff",

Re: Can we access Queued Duration as an attribute?

2024-02-16 Thread James McMahon
ince epoch. >> >> So you might have a RouteOnAttribute that has a property named “old” with >> a value of: >> ${lastQueueDate:lt( ${now():minus(1)} )} >> >> So any FlowFile that has been queued for more than 10 seconds would be >> routed to “old”, anything else

Re: ExecuteStreamCommand failing to unzip incoming flowfiles

2024-02-15 Thread James McMahon
circle back and post it. On Fri, Feb 2, 2024 at 6:03 PM Michael Moser wrote: > Yes, that's exactly what those commands do. Your linux commands like > unzip and tar can probably read directly from /dev/stdin and write directly > to /dev/stdout if you want to. > > -- Mike > > &

Re: Can we access Queued Duration as an attribute?

2024-02-15 Thread James McMahon
ow()} or ${now():toNumber()} expression language function. > > https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#now > > Then later on in the flow you can compare current time to the saved error > time to see how much time has elapsed. > > — Jim > > &g

Re: Can we access Queued Duration as an attribute?

2024-02-15 Thread James McMahon
in the queue from the UI. One can make an assumption that all of these 5000 flowfiles that failed InvokeHttp share a similar range of lineageStartDate, but that will not necessarily be true depending on flow complexity. On Wed, Feb 14, 2024 at 9:49 AM James McMahon wrote: > What a great workaro

Re: Can we access Queued Duration as an attribute?

2024-02-14 Thread James McMahon
> ${entryDate} = 1707859943778 > ${lineageStartDate} = 1707859943778 > ${lineageStartDate:format("-MM-dd HH:mm:ss.SSS")} = 2024-02-13 > 21:32:23.778 > > -- Mike > > > On Mon, Feb 12, 2024 at 11:38 AM James McMahon > wrote: > >> When we examine the cont

Can we access Queued Duration as an attribute?

2024-02-12 Thread James McMahon
When we examine the contents of a queue through the UI and select a flowfile from the resulting list, we see FlowFile Details in the Details tab. Are those key/values accessible from nifi expression language? I would like to access Queued Duration. I have a queue that holds flowfiles with

Re: ExecuteStreamCommand failing to unzip incoming flowfiles

2024-02-02 Thread James McMahon
tar command to tar $tmpdir to $tmptarfile > cat $tmptarfile >> /dev/stdout > > #cleanup > rm -f $tmpzipfile > rm -f $tmptarfile > rm -rf $tmpdir > > > > On Wed, Jan 31, 2024 at 12:55 PM James McMahon > wrote: > >> If anyone can show me how to get

Re: ExecuteStreamCommand failing to unzip incoming flowfiles

2024-01-31 Thread James McMahon
If anyone can show me how to get my ExecuteStreamCommand configured properly as a workaround, I am still interested in that. Jim On Wed, Jan 31, 2024 at 12:39 PM James McMahon wrote: > I tried to find a Create option for tickets here, > https://issues.apache.org/jira/projects/NIFI/issue

Re: ExecuteStreamCommand failing to unzip incoming flowfiles

2024-01-31 Thread James McMahon
wrote: > I went ahead and wrote it up here > https://issues.apache.org/jira/browse/NIFI-12709 > > Thanks > > On Wed, Jan 31, 2024 at 10:30 AM James McMahon > wrote: > >> Happy to do that Joe. How do I create and submit a JIRA for >> consideration? I have no

Re: ExecuteStreamCommand failing to unzip incoming flowfiles

2024-01-31 Thread James McMahon
dard in the > case of a Zip but perhaps still capturable as extra fields. > > Thanks > > On Wed, Jan 31, 2024 at 10:01 AM James McMahon > wrote: > >> I tried to use UnpackContent to extract the files within a zip file named >> ABC DEF (1).zip. (the filename has spac

ExecuteStreamCommand failing to unzip incoming flowfiles

2024-01-31 Thread James McMahon
I tried to use UnpackContent to extract the files within a zip file named ABC DEF (1).zip. (the filename has spaces in its name). UnpackContent seemed to work, but it did not preserve file attributes from the files in the zip. For example, the lastModifiedTime is not available so downstream I

Re: Error on InvokeHTTP

2024-01-13 Thread James McMahon
: I GenerateFlowFile with a body that is english text. I IdentifyMimeType. I send that through InvokeHTTP. On Fri, Jan 12, 2024 at 5:04 PM Juan Pablo Gardella < gardellajuanpa...@gmail.com> wrote: > it seems charset issue. if it is a json add charset=utf-8 > > On Fri, Jan 12, 2024

Error on InvokeHTTP

2024-01-12 Thread James McMahon
I have a text flowfile that I am trying to send to a translation service on a remote EC2 instance from my nifi insurance on my EC2. I am failing with only this somewhat-cryptic error: InvokeHTTP[id=a72e1727-3da0-1d6c-164b-e43c1426fd97] Routing to Failure due to exception: Unexpected char 0x20 at

Re: Extract from jars and nars

2024-01-01 Thread James McMahon
po? If not you can use > UnpackContent with a Packaging Type of zip. I tried on both JARs and > NARs and it works. > > Regards, > Matt > > On Sun, Dec 31, 2023 at 12:37 PM James McMahon > wrote: > > > > I have a NiFi flow that handles many jar and nar archive files as &

Extract from jars and nars

2023-12-31 Thread James McMahon
I have a NiFi flow that handles many jar and nar archive files as incoming flowfiles. I am trying to figure out a way I can extract files from these archives - for example, in most cases one incoming jar has a number of files in its archive. So one flowfile should yield N output flowfiles if there

Re: Configuring ExecuteStreamCommand on jar flowfiles

2023-12-03 Thread James McMahon
jar file is mostly just a standard zip file, can you use a built > in processor instead? > > On Sun, 3 Dec 2023, 15:36 James McMahon, wrote: > >> I have a large volume of a wide variety of incoming data files. A subset >> of these are jar files. Can the ExecuteStreamC

Configuring ExecuteStreamCommand on jar flowfiles

2023-12-03 Thread James McMahon
I have a large volume of a wide variety of incoming data files. A subset of these are jar files. Can the ExecuteStreamCommand be configured to run the equivalent of jar -xf ${flowfile} and will that automatically direct each output file to a new flowfile, or does ESC need to be told to direct

Apache Tike compatible with NiFi 1.16.3

2023-11-12 Thread James McMahon
Where can I look to determine which Apache Tika version should be downloaded to add to my lib directory for my Apache NiFi 1.16.3 installation?

ExecuteStreamCommand fails to run file against flowfile

2023-11-04 Thread James McMahon
I am having some difficulty getting the file command to run from ExecuteStreamCommand, and am hoping someone can see my error. At my linux command line, file --brief myfile.mdb returns Microsoft Access Database as expected. I am reading files into a nifi flow, and trying to apply

Re: curl from ExecuteStreamCommand

2023-10-22 Thread James McMahon
s:amz:$REGION:es > > Kind Regards, > Lehel > > -- > *From:* James McMahon > *Sent:* Saturday, October 21, 2023 20:25 > *To:* users > *Subject:* curl from ExecuteStreamCommand > > I have tested this curl from my ec2 command line: > curl -XPUT -u 'myusernm:myuserpw

curl from ExecuteStreamCommand

2023-10-21 Thread James McMahon
I have tested this curl from my ec2 command line: curl -XPUT -u 'myusernm:myuserpw' ' https://vpc-rampart-test-opensearch-nrqyb7jjpvmji6cp2qcvmyhcgq.us-east-1.es.amazonaws.com/movies/_doc/1' -d '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi"], "year": 1996, "actor": ["Jack

How to configure ConvertExcelToCSVProcessor processor?

2023-09-28 Thread James McMahon
I have an incoming xlsx file, with many sheets. I am trying to use ConvertExcelToCSVProcessor processor to extract the sheets. It is currently erroring when it extracts the header. Here is what the header is in one of the sheets (commas added by me for clarity):

Re: How can I View my flowfile records?

2023-09-25 Thread James McMahon
10:30 PM Chris Sampson < >>> chris.samp...@naimuri.com> wrote: >>> >>>> An UpdateAttribute could also be used to update the mime.type, e.g. to >>>> text/csv. >>>> >>>> I'd think the csv record writer should probably do thi

Re: How can I View my flowfile records?

2023-09-24 Thread James McMahon
ough IdentifyMimeType then you can > likely view it just fine. > > Thanks > > On Sun, Sep 24, 2023 at 3:21 PM James McMahon > wrote: > >> I sure can Joe. Here they are: >> >> RouteOnAttribute.Route >> isExcel >> execution.command >> /u

Re: How can I View my flowfile records?

2023-09-24 Thread James McMahon
not be able to render it. Can you show what flowfile attributes are > present at the point you attempt to view it? > > Thanks > > On Sun, Sep 24, 2023 at 3:03 PM James McMahon > wrote: > >> Hello. I have converted incoming Excel files to csv. I'd like to look at >> th

How can I View my flowfile records?

2023-09-24 Thread James McMahon
Hello. I have converted incoming Excel files to csv. I'd like to look at the result, but when I select my flowfiles from the output queue, I can only select "View as hex" - but I cannot get the display to show me the records in the form I expect. Viewing them using the hex display is not helpful.

SQL query to count fields with values.

2023-09-07 Thread James McMahon
I have converted incoming Excel files to csv files. My incoming files vary in structure, and so do have different headers and also different field names. I have the header field names in an attribute. How can I employ a QueryRecord or an ExecuteSQL processor to return a count for each field of

Re: BufferedReader best option to search through large flowfiles?

2023-06-05 Thread James McMahon
> > Thanks > -Mark > > > > On Jun 5, 2023, at 8:09 AM, James McMahon wrote: > > > > Hello. I have a requirement to scan for multiple regex patterns in very > large flowfiles. Given that my flowfiles can be very large, I think my best > approach is to employ an ExecuteGr

Re: Adjusting FlattenJson output

2023-06-01 Thread James McMahon
those fields by name and set them to > null. > > Regards, > Matt > > On Mon, May 29, 2023 at 10:38 AM James McMahon > wrote: > > > > I have incoming JSON data that begins like this, and that I am trying to > flatten with FlattenJSON v1.16.3: > > > &

Adjusting FlattenJson output

2023-05-29 Thread James McMahon
I have incoming JSON data that begins like this, and that I am trying to flatten with FlattenJSON v1.16.3: { "meta" : { "view" : { "id" : "kku6-nxdu", "name" : "Demographic Statistics By Zip Code", "assetType" : "dataset", "attribution" : "Department of Youth and

How to determine groovy version

2023-05-13 Thread James McMahon
I'm using a series of Groovy scripts running from NiFi v1.16.3 ExecuteScript. How can I determine which version of Groovy is baked into NiFi v1.16.3?

Re: Generalizing QueryRecord to changing inferred CSV headers

2023-04-23 Thread James McMahon
: 12, "First-Citizens Bank & Trust Company": 11, "Ameris Bank": 10, "U.S. Bank N.A.": 9, "Community & Southern Bank": 8, "Centennial Bank": 7, "Stearns Bank, N.A.": 7, "Bank of the Ozarks": 7, "Republic Bank of Chicago&

Re: Generalizing QueryRecord to changing inferred CSV headers

2023-04-18 Thread James McMahon
except change X to Y". Does it need to be SQL executed against the > individual fields? If not, take a look at ScriptedTransformRecord doc > (and its Additional Details page). IIRC you're a Groovy guy now ;) so > you should be able to alter the fields as you see fit using Groovy > r

Generalizing QueryRecord to changing inferred CSV headers

2023-04-18 Thread James McMahon
Hello. I recently asked the community a question about processing CSV files. I received some helpful advice about using processors such as ConvertRecord and QueryRecord, and was encouraged to employ Readers and RecordSetWriters. I've done that, and thank all who replied. My incoming CSV files

Re: Handling CSVs dynamically with NiFi

2023-04-12 Thread James McMahon
our N outputs, you may be able to do this in one go or chain > a few Processors (say “state”, then “city” then “zipcode”). > > > > Regards, > > > > Isha > > > > *Van:* James McMahon > *Verzonden:* woensdag 12 april 2023 14:56 > *Aan:* users@nifi.ap

Re: Handling CSVs dynamically with NiFi

2023-04-12 Thread James McMahon
f different > json files with a handful of records each, then splitting per row might be > quicker than copying the entire json file that many times. > > > > Regards, > > > > Isha > > > > *Van:* James McMahon > *Verzonden:* vrijdag 7 april 2023 17:14 &

Re: Handling CSVs dynamically with NiFi

2023-04-07 Thread James McMahon
record per flow file, this is an anti-pattern that leads to poor > performance in the flow. > > Thanks, > > Bryan > > On Fri, Apr 7, 2023 at 9:41 AM James McMahon wrote: > > > > Very interesting, very helpful insights. Thank you again, Mike. > > Late last night I

Re: Handling CSVs dynamically with NiFi

2023-04-07 Thread James McMahon
data they are traversing. > > > > Mike Sofen > > > > *From:* James McMahon > *Sent:* Thursday, April 06, 2023 2:03 PM > *To:* users@nifi.apache.org > *Subject:* Re: Handling CSVs dynamically with NiFi > > > > Can I ask you one follow-up? I've gotten my

Re: Handling CSVs dynamically with NiFi

2023-04-06 Thread James McMahon
json keys definition changes depending on the lfowfile, what should JsonPathExpression be set to in the SplitJson configuration? On Thu, Apr 6, 2023 at 9:59 AM Mike Sofen wrote: > Jim – that’s exactly what I did on that “pre” step – generate a schema > from the CSVReader and use that to dynamica

Re: Handling CSVs dynamically with NiFi

2023-04-06 Thread James McMahon
le when processing the actual files > prior to storing into the destination table. > > > > Nifi was VERY fast and efficient in this, as was Postgres. > > > > Mike Sofen > > > > *From:* James McMahon > *Sent:* Thursday, April 06, 2023 4:35 AM > *To

Handling CSVs dynamically with NiFi

2023-04-06 Thread James McMahon
We have a task requiring that we transform incoming CSV files to JSON. The CSVs vary in schema. There are a number of interesting flow examples out there illustrating how one can set up a flow to handle the case where the CSV schema is well known and fixed, but none for the generalized case. The

IAM for authentication and authorization to NiFi?

2023-03-17 Thread James McMahon
Hello. We run nifi on an AWS EC2 instance. I currently employ certs for nifi user authentication. The CA is in our nifi truststore. Users install certs issued by the CA in their browsers. I've set up a parsing pattern in nifi.properties to extract user identities from the CN of the cert, and I

Re: FlattenJSON is failing

2023-02-15 Thread James McMahon
SHA256":"e3daeb8cfd6db4aad20bb42900bc5fa4815eba7e55d97cb01a1a9674668f20b2", > "sourceMD5":"a18eed985ddb04cbe13b487062628585" >}, >"triage":{ > "datatype":"mdb", > "mdb":{ >

FlattenJSON is failing

2023-02-14 Thread James McMahon
I have used AttributeToJSON to generate this JSON:

Re: How to cherry pick a specific line from a flowfile?

2023-02-11 Thread James McMahon
header for table in mdb file', e) session.transfer(ff, REL_FAILURE) } Jim On Fri, Feb 10, 2023 at 7:19 AM James McMahon wrote: > Ah - of course. I went overboard here. Just because I don't use the > OutputStream for this purpose doesn't mean I can assume the method > signature for

Re: How to cherry pick a specific line from a flowfile?

2023-02-10 Thread James McMahon
owFile input) > > Sent from my iPhone > > On Feb 9, 2023, at 9:34 PM, James McMahon wrote: > >  > Mark, your RouteText blog worked perfectly. Thank you very much. > > Matt, I still want to get the BufferedReader working. I'm close. Here is > my code, with the error that re

Re: How to cherry pick a specific line from a flowfile?

2023-02-09 Thread James McMahon
ifi-flow-routetext-5068a3b4efb3 > > Thanks > Mark > > Sent from my iPhone > > On Feb 9, 2023, at 8:06 PM, James McMahon wrote: > >  > My version of nifi does not have Range Sampling unfortunately. > If I get the flowfile through a session as done in the Cook

Re: How to cherry pick a specific line from a flowfile?

2023-02-09 Thread James McMahon
. I think some/most/all record processors write this attribute, > and they work record by record so they don’t load the whole thing into > memory. Even SampleRecord adds a record.count attribute but if you specify > one line the value will be 1 :) > > Regards, > Matt > > > On Feb 9,

How to cherry pick a specific line from a flowfile?

2023-02-09 Thread James McMahon
Hello. I am trying to identify a header line and a data line count from a flowfile that is in csv format. Most of us are familiar with Matt B's outstanding Cookbook series, and I am trying to use that as my starting point. Here is my Groovy code: import org.apache.commons.io.IOUtils import

Detecting if a flowfile is a packed file

2023-01-13 Thread James McMahon
Hello. Does anyone have an approach they would recommend for reliably detecting if a flowfile is a packed file that needs to be routed through UnpackContent? I have a RouteOnAttribute within which I check for filenames that end in tar or zip, but that seems unsophisticated and overly reliant on

Re: Unable to view files in queue

2023-01-12 Thread James McMahon
tion instead of view. > > On Tue, Jan 10, 2023 at 9:06 PM James McMahon > wrote: > >> I have a group of csv files in a queue that I want to inspect. I List >> Queue, select one that appears to be 110MB in size, go to my Details tab, >> then click View. The UI spi

Unable to view files in queue

2023-01-10 Thread James McMahon
I have a group of csv files in a queue that I want to inspect. I List Queue, select one that appears to be 110MB in size, go to my Details tab, then click View. The UI spins and spins, displays a header as I would expect, but never returns data rows. I'm not sure how to debug this. The

Re: NiFi failing to start

2023-01-02 Thread James McMahon
on startup, so you need to retain and configure the same > CA for toolkit of you plan to use it to issue new certs in future. > > On Wed, 28 Dec 2022, 17:32 James McMahon, wrote: > >> I continue to experience errors when I try to start my nifi 1.16.3 >> instance. I h

Getting the proper CN in self-signed certs from tls-toolkit?

2022-12-29 Thread James McMahon
Not sure whether this question belongs in the users or developers domain. Am asking in both hoping to get assistance. I am trying to use tls-toolkit to create a CA and self-signed certs. I notice that my CN in my pem file is not what I request on the command line. How can I successfully force

NiFi failing to start

2022-12-28 Thread James McMahon
I continue to experience errors when I try to start my nifi 1.16.3 instance. I have followed this guide in an effort to use the toolkit to generate self-0signed certs for user admin, signed by a nifi truststore: Apache NiFi Walkthroughs

Re: Failing to start - keystore properties invalid

2022-12-28 Thread James McMahon
eption on object creation; nested exception is java.lang.IllegalStateException: Flow controller TLS configuration is invalid Bryan, if you see this can you please comment? On Tue, Dec 27, 2022 at 4:13 PM James McMahon wrote: > Hello. I am trying to start a secure instance of nifi version 1.1

Failing to start - keystore properties invalid

2022-12-27 Thread James McMahon
Hello. I am trying to start a secure instance of nifi version 1.16.3. I am getting this error on start attempt: 2022-12-27 20:44:21,765 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 90 properties. Including precedence overrides effective accessible registry key size is 90

Re: authorizers.xml for simple single user configuration?

2022-12-13 Thread James McMahon
gt; Also if you are trying to remove authentication/authorization and run > over http, then why declare the SingleUserAuthorizer at all? > > On Tue, Dec 13, 2022 at 1:43 PM James McMahon > wrote: > > > > Hello. I am having difficulty getting nifi to start for a simple s

authorizers.xml for simple single user configuration?

2022-12-13 Thread James McMahon
Hello. I am having difficulty getting nifi to start for a simple single node configuration without user authentication. My goal is to get a nifi instance running over http. I understood that there would be no user authentication in such a case. Why then is my nifi instance failing to start with

Re: Error on nifi start

2022-12-13 Thread James McMahon
access> ) Would I set this in bootstrap.conf? How? On Tue, Dec 13, 2022 at 7:45 AM James McMahon wrote: > I am using an Ansible role from Ansible GALAXY that has been tested and > validated up through Apache NiFi v1.14.0. I download and install 1.14.0.bin > from the Apache NiFi archi

Error on nifi start

2022-12-13 Thread James McMahon
I am using an Ansible role from Ansible GALAXY that has been tested and validated up through Apache NiFi v1.14.0. I download and install 1.14.0.bin from the Apache NiFi archives fir this reason. I am using ansible to install on and AWS EC2 instance. My java version on this instance is: openjdk

Re: Unable to start nifi service

2022-12-11 Thread James McMahon
; Instead of the bootstrap log check the nifi-app.log. It might be out of > memory or have the port it needs already in use. > > Regards, > > Jeremy > > > On 11 Dec 2022, at 11:01, James McMahon wrote: > >  > I am trying to start nifi on an AWS EC2 instance. My bootst

Unable to start nifi service

2022-12-10 Thread James McMahon
I am trying to start nifi on an AWS EC2 instance. My bootstrap.conf says the service does not start, but I see no indication why. I am trying to start the nifi service using sudo ../bin/nifi.sh start How can I debug this? Here is the nifi-bootstrap log: 2022-12-11 02:54:02,932 INFO [main]

Customizing NiFi in a Docker Container on EC2

2022-11-11 Thread James McMahon
The NiFi System Administration Guide makes many recommendations for configuration changes to optimize nifi performance. These "best practices", for example: * https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices Placement of repos on separate disk

Re: NiFi on AWS EC2

2022-11-10 Thread James McMahon
. Jim On Tue, Nov 8, 2022 at 10:28 PM David Handermann < exceptionfact...@apache.org> wrote: > Jim, > > You're welcome! Thanks for following up and confirming the solution, great > collaborative effort! > > Regard, > David Handermann > > > > > On Tue,

Re: NiFi on AWS EC2

2022-11-08 Thread James McMahon
gt;> Make sure you use your full domain name >> ec2-3-238-27-220.compute-1.amazonaws.com >> David shorten it in his code >> >> On November 8, 2022 5:57:26 p.m. James McMahon >> wrote: >> >>> Thank you, David. I’ve made that change, adding the pr

Re: NiFi on AWS EC2

2022-11-08 Thread James McMahon
Yes sir, I did. I used the full public domain name. On Tue, Nov 8, 2022 at 8:08 PM Dmitry Stepanov wrote: > Make sure you use your full domain name > ec2-3-238-27-220.compute-1.amazonaws.com > David shorten it in his code > > On November 8, 2022 5:57:26 p.m. James McMahon >

Re: NiFi on AWS EC2

2022-11-08 Thread James McMahon
gt; > Regards, > David Handermann > > On Tue, Nov 8, 2022 at 6:06 PM James McMahon wrote: > >> Hi David. This is very helpful, thank you. I feel like I am close, but I >> get an error. My Inbound Rules for my security group now include: >> 8443 TCP (MyIP)/32 >> 44

Re: NiFi on AWS EC2

2022-11-08 Thread James McMahon
gt; docker logs nifi | grep Generated > > Regards, > David Handermann > > [1] https://hub.docker.com/r/apache/nifi > > On Tue, Nov 8, 2022 at 4:00 PM James McMahon wrote: > >> Hi and thank you, David and Dmitry. In my case I was following this >> example, >

Re: NiFi on AWS EC2

2022-11-08 Thread James McMahon
n > > On Tue, Nov 8, 2022 at 1:54 PM James McMahon wrote: > >> Has anyone successfully configured NiFi on AWS, and accessed it from a >> browser on a Windows desktop? I’ve tried following a few links to do this. >> I’ve verified that my instance security grou

NiFi on AWS EC2

2022-11-08 Thread James McMahon
Has anyone successfully configured NiFi on AWS, and accessed it from a browser on a Windows desktop? I’ve tried following a few links to do this. I’ve verified that my instance security group allows access to 8080 via its inbound rules. I’ve putty’ed into the instance via ssh port 22 to verify

system-diagnostics for cluster or node?

2022-10-20 Thread James McMahon
When I am on a node that is part of a nifi cluster configuration and I issue this REST API call from a browser... https://1.2.3.4:8443/nifi-api/system-diagnostics ...is there a parameter that can be applied on that call to tell nifi I want aggregated statistics for the cluster? And is that the

Change replication in the cluster

2022-10-05 Thread James McMahon
We have been experiencing occasional failures to restart nifi services on a cluster node because the flow.xml.gz falls out of synch with the other nodes of the cluster. While researching this, found an article that discusses possible causes. That article mentions this: "A change replication

Re: Expression language to handle a capture group

2022-10-04 Thread James McMahon
Number()}'} Unfortunately it continues to choke. I don't think it likes the single quotes in the second expression of the replaceFirst('', '') being followed immediately by the single quote at the front of $1. Jim On Tue, Oct 4, 2022 at 11:19 AM James McMahon wrote: > I have an

Expression language to handle a capture group

2022-10-04 Thread James McMahon
I have an incoming filename that includes the pattern prefix20221004035958postfix. I need to convert that yMdHms to seconds since the epic within that filename. This is the expression I attempt to use, but it seems to choke on the capture group reference: filename

Is result milliseconds since epoch?

2022-10-03 Thread James McMahon
I have a string representation of a datetime parsed from a lengthy filename, attribute myValue with value of 20221003055959. If I convert to a true datetime value and then apply toNumber() to that, does toNumber() return milliseconds since the epoch? The expression language guide doesn’t go into

Re: Can ExecuteStreamCommand do this?

2022-09-30 Thread James McMahon
we avoid anything beyond gzip > because S3 is so cheap. > > On Thu, Sep 29, 2022 at 10:51 AM James McMahon > wrote: > > > > Thank you Mark. Had no idea there was this file-based dependency to 7z > files. Since my workaround appears to be working I think I may just move > f

Re: Can ExecuteStreamCommand do this?

2022-09-29 Thread James McMahon
ere is > discussion about this down at sourceforge but the detail is blocked by my > employer’s firewall. > > p7zip / Discussion / Help: E_NOTIMPL for stdin / stdout pipe > <https://sourceforge.net/p/p7zip/discussion/383044/thread/8066736d/> > > https://sourceforge.net/p/p

  1   2   3   4   5   >