Hi Peter,
Thanks for letting us know you found a solution and for the additional context.
Provenance performance is a key area of focus in the next couple releases, so
hopefully we will have that fixed soon.
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3
Prabhu,
Would ExtractEmailHeaders[1] and ExtractEmailAttachments[2] cover your use
case?
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.email.ExtractEmailHeaders/index.html
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.email.ExtractEma
Hi,
I am new to the NIFI. I have just use Consume IMAP Processor to retrieve
attachement from mail Server.
If i use it then i can able to download attachement but that document
having MIME type information with addition of EMail Data like below
screenshot.
I need to extract the exact data only
Hello -
I was wondering if NiFi can support multiple queries in the same PutSQL
processor. For example, if an attribute is set to 'update' - will PutSQL
run the defined update query and next time when it is an 'insert' - it runs
the insert query. Or should we go ahead and add two separate processo
Andy/Bryan,
Thanks for all of the detail, it’s been helpful.
I actually did an experiment this morning where I modified the processor to
force it to keep calling `get` until it had all 1 million FlowFiles. Since I
was calling it sequentially it was able to move files out of swap and into
activ
Matt,
Thank you for looking at this. I was finding it particularly weird I
couldn't find a way of downloading the content. :-)
Cheers
On Wed, Sep 21, 2016 at 2:24 AM, Matt Gilman
wrote:
> I think I see the issue and someone else just submitted a similar JIRA [1]
> which is caused by the same b
Andy,
That was my thinking. An easy test might be to bump the threshold up to
100k (increase heap if needed) and see if it starts grabbing 100k every
time.
If it does then I would think it is swapping related, then need to figure
out if you really want to get all 1 million in a single batch, and
Bryan,
That’s a good point. Would running with a larger Java heap and higher swap
threshold allow Peter to get larger batches out?
Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Sep 20, 2016, at 1:41 PM, Bry
Peter,
Does 10k happen to be your swap threshold in nifi.properties by any chance
(it defaults to 20k I believe)?
I suspect the behavior you are seeing could be due to the way swapping
works, but Mark or others could probably confirm.
I found this thread where Mark explained how swapping works w
I have 500+ HTTP request and that will return files which has various size
that will be stored into s3..
For each http (oai-pmh) request we will get file to put into s3.
So content repository keep on increasing for the file size. One sudden
point it reaches 4.6 GB and that's the avaible disk spa
Let's fade the connection slowly to an inverted if backpressure engages?
On Tue, Sep 20, 2016, 1:17 PM Rob Moran wrote:
> Agreed – thanks for calling that out, Andy.
>
> Rob
>
> On Tue, Sep 20, 2016 at 1:13 PM, Andy LoPresto
> wrote:
>
>> In this and other UI discussions going on, I would reque
Agreed – thanks for calling that out, Andy.
Rob
On Tue, Sep 20, 2016 at 1:13 PM, Andy LoPresto wrote:
> In this and other UI discussions going on, I would request that everyone
> keep in mind the usability of the software by people with visual and other
> impairments. The US Federal Government
In this and other UI discussions going on, I would request that everyone keep
in mind the usability of the software by people with visual and other
impairments. The US Federal Government has guidelines referred to as “Section
508” [1] which cover the design and usability of softwares specificall
No need to go wild, changing processor colors should be enough, IMO. PG and
RPG are possible candidates, but they are different enough already, I guess.
What I heard quite often was to differentiate between regular processors,
incoming sources of data and out only (data producers?). Maybe even wit
Good points. I was thinking a label would be tied to the group of
components to which it was applied, but that could also introduce problems
as things move and are added to a flow.
So would you all expect to be able to change the color of every component
type, or just processors?
Andrew - your co
I think I see the issue and someone else just submitted a similar JIRA [1]
which is caused by the same bug. When using an authentication which will
use the API tokens, download requests are processed using a one-time
password token (since they become part of the URL). These are only honored
for cer
Hi Selvam,
As mentioned, please keep messages to the one list. Moving dev to bcc
again.
Archiving is only applicable for that content which has exited the flow and
is not referenced by any FlowFiles currently in your processing graph,
similar to garbage collection in Java. For this particular in
In my case it is going out of disk space.
i set nifi.content.repository.archive.enabled=false. (when i changed this
have restarted nifi cluster )
But still i can see the processor keep on writing here on the disk.
On Tue, Sep 20, 2016 at 4:34 PM, Joe Witt wrote:
> Hello
>
> Please only post to
Andre/Matt,
Sorry, my memory was wrong. My experience matches Andre’s, it only errors when
I click Download; View is fine.
We are running a customized build of 1.0 and I made the assumption that this
was an issue caused by a bad merge on our part and wasn’t paying it much
attention. I have not
Downloading and viewing should be the same permissions. If you're seeing
otherwise please file a JIRA with the details. Is the instance clustered,
what permissions to you have set on the source component, etc?
Andre,
The 'view the data' is the correct policy that you need to configure. Is
your in
Peter,
Quite curious as I am able to view the flowfile but unable to download it.
Seems something we should either document (how to setup properly) or to fix
in the next release.
Have you already raised a JIRA?
On Wed, Sep 21, 2016 at 12:30 AM, Peter Wicks (pwicks)
wrote:
> No help here, exc
Hello
Please only post to one list. I have moved 'dev@nifi' to bcc.
In the docs for this processor [1] you'll find reference to "Multipart
Part Size". Set that to a smaller value appropriate for your JVM
memory settings. For instance, if you have a default JVM heap size of
512MB you'll want so
This is the exact error.
On Tue, Sep 20, 2016 at 4:30 PM, Selvam Raman wrote:
> HI,
>
> I am pushing data to s3 using puts3object. I have setup nifi 1.0 zero
> master cluster.
>
> Ec2 instance having only 8GB of hard disk. Content repository writing till
> 4.6 gb of data then it throws jvm o
HI,
I am pushing data to s3 using puts3object. I have setup nifi 1.0 zero
master cluster.
Ec2 instance having only 8GB of hard disk. Content repository writing till
4.6 gb of data then it throws jvm out of memory error.
I changed nifi.properties for nifi.content.archive to false. but still it
i
I like the tooltip addition of yours.
For more interactive feedback on the canvas I can immediately think of 2
items.
1. Indicator for when backpressure was configured on a connection (although
it's now always added by default, maybe less useful).
2. Changing the color of a connection when backp
Hi,
We are making OAI-PMH requests invoked within Http.
On Tue, Sep 20, 2016 at 9:31 AM, Pierre Villard wrote:
> Hi Selvam,
>
> Supposing that your source if a SQL-like source, you should have a look at
> QueryDatabaseTable [1] processor. It proposes a 'Maximum-value Columns'
> that gives you t
I agree. Labels are great for grouping, beyond PGs. Processor colors
individually add value. E.g. flow terminator colored in red was a very
common pattern I used. Besides, labels are not grouped with components, so
moving things and re-arranging is a pain.
Andrew
On Tue, Sep 20, 2016, 11:21 AM Jo
Rob,
The labelling functionality you described sounds very useful in general.
But, I miss the processor color too.
I think labels are really useful for identifying groups of components and
areas in the flow, but I worry that needing to use them in volume for
processor coloring will increase the A
What if we promote the use of Labels as a way to highlight things. We could
add functionality to expand their usefulness as a way to highlight things
on the canvas. I believe that is their intended use.
Today you can create a label and change its color to highlight single or
multiple components. E
No help here, except to share that I’ve also seen this error. I’ve been
working around it by downloading the FlowFile instead of viewing it.
From: Andre [mailto:andre-li...@fucs.org]
Sent: Monday, September 19, 2016 11:18 PM
To: users@nifi.apache.org
Subject: Download item from queue - what perm
I'm using JSONToSQL, followed by PutSQL. I'm using Teradata, which supports a
special JDBC mode called FastLoad, designed for a minimum of 100,000 rows of
data per batch.
What I'm finding is that when PutSQL requests a new batch of FlowFiles from the
queue, which has over 1 million rows in it,
Andrew,
Thanks for the feedback on the status bar. Separation between each item
helps but realize after your comments how it can not feel like a single,
cohesive group of items. We could probably tighten things up a bit.
I think another part of this that could help would be to address some of
the
Selvam,
The specific endpoint is
http://{host}:{port}/nifi-api/flow/cluster/summary
This will return a ClusterSummaryDTO [1] (this is incorrect in the
documentation). I'm having trouble accessing JIRA right now but we'll get
that fixed in the next release.
Matt
[1]
https://github.com/apache/ni
Hi,
Have a look at the REST API :
https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
Pierre
2016-09-20 11:32 GMT+02:00 Selvam Raman :
> Hi,
>
> How to check nifi running mode (cluster, standalone).
> is there any command to check.
>
> Thanks,
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் ந
Hi,
How to check nifi running mode (cluster, standalone).
is there any command to check.
Thanks,
Selvam Raman
"லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
--
Selvam Raman
"லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
Hi Selvam,
Supposing that your source if a SQL-like source, you should have a look at
QueryDatabaseTable [1] processor. It proposes a 'Maximum-value Columns'
that gives you the possibility to specify the column containing an ID and
or timestamp. The processor will keep track of the maximum value f
36 matches
Mail list logo
