Hi Bryan, thanks for the help. Now that I've used nifi a little longer I'm
getting the hang of it. I was trying to do basic stuff like formatting
monetary values and masking text but I ended doing it in another way.
Thanks for the help anyway!
2017-10-21 17:05 GMT-02:00 Bryan Bende :
> Eric,
>
>
Thanks Pierre, that's what I was looking for. By the way congrats on your
blog. I've being an avid reader of you and Matt.
Regards!
2017-10-23 10:14 GMT-02:00 Pierre Villard :
> Hi Eric,
>
> You can add a bending point by double clicking on the black line of the
> relationship (and remove a bend
Hi Mark, thanks the clarification!
Cheers!
2017-10-23 11:04 GMT-02:00 Mark Payne :
> Hi Eric,
>
> Just to elaborate - once you double-click on the connection, it will
> create a new bend point as Pierre noted. this allows you to
> then bend the line however you'd like. In addition to this, thoug
Thanks all, as a workaround I disabled local state storage which
resolved the issue.
Pierre: the use case was using NiFi FlowFiles to trigger
metadata-driven jobs (as part of this software, which uses NiFi
https://kylo.io/)
Joe: Yes, that seemed to be the case.
On 20 October 2017 at 22:52, Joe Wi
Austin,
Once you convert from Avro to CSV you can probably use ReplaceText to
do what you want with the column names. For now, ExecuteSQL only
outputs Avro so it has to normalize the names, which includes
replacing the spaces. I have written NIFI-4517 [1] to track a new
processor (or changed featu
I figured it out. My avro schema was rewriting the column names after I was
aliasing haha. Related question. Is there a way to make it so that I can
use spaces instead of underscores (it says that spaces are an illegal
character) or is it possible for me to erase the first row and replace it
with t
Austin,
What version of NiFi are you using? There was an issue with aliases
(at least for MySQL) before NiFi 1.1.0, fixed by NIFI-3064 [1]. Also
what database and driver version are you using? Since NiFi 1.1.0, we
are following the JDBC 4 spec which says the drivers (when
getColumnLabel is calle
I am using an executesql processor to perform a select query. I want to
select data from a table and have the column names displayed as something
else (alias). Example would be select answer_1 as "how_did_it_go" from
schema.table. When I run a flowfile through it it queries the data but the
alias i
Perhaps a quick(ish) win would be to implement a
DeadlockDetectionReportingTask, where you could specify processor IDs
(or names but that can get dicey) and it would monitor those
processors for incoming connections that all have backpressure
applied, and that the processor has not run for X amount
I see your point, and once we start sending updates throughout the cluster
we probably need to start worrying about race conditions where data may
arrive before a schema at a node or something similar.
I'll drop the idea and look at using an existing schema registry if it
become relevant for me at
Pape
Grab the data
Use 'MergeContent' on zip archive mode
Send to HDFS.
https://nifi.apache.org/docs.html Take a look at MergeContent.
Thanks
Joe
On Mon, Oct 23, 2017 at 11:08 AM, Papa Samba DIOP <
papasamba.d...@orange-sonatel.com> wrote:
>
>
> Sonatel:Scan antiv
Sonke
The built-in one was basically meant as a read-only access for schema
lookups by name. The idea you bring up definitely makes sense but to
Andrew's point once we're talking about supporting an automated
complete lifecycle it probably makes sense for the schema registry
employed to be its ow
Hi Andrew,
My question was generally about the back pressure deadlock problem. The
specific PublishJMS scenario from our test environment was just meant as an
example.
Having that said, ANY resource can get full or become unavailable. JMS queues
can have maximum queue depths configured, which
Hey Andrew,
that helps, yes. Thank you very much. I'll probably stick with putting the
schema in the flowfile for now then and revisit this issue once the service
has been up and running for a while. I don't envision the schema to change
often enough that running a schema registry alongside Nifi i
Sonatel:Scan antiviral effectue (1)
Hi all,
How could write a file from many flowfiles. i.e Get 10 files from HDFS, zip
them in a file, and finally store to HDFS back.
Regards,
Pape
I wonder which jms broker you are using. The situation where a jms
destination is full is absurd, the whole point was to decouple publishers
and consumers. I would additionally look into what jms broker settings are
available to address the situation.
Andrew
On Mon, Oct 23, 2017, 10:32 AM Arne De
Hi,
Using an external schema registry is the way to go. The embedded one is
meant for ease of use, but once you grow beyond an initial phase, the best
practice is to have a full service and potentially use standard InvomeHTTP
to perform operations beyond just lookups.
Does it help?
Andrew
On Mon
Hi Mark,
Don’t get me wrong, NiFi is great! Much appreciated that it is constantly being
improved. Would be great if better support for looping connections would be one
of those improvements in the future :-) In the meantime, we can live with one
of the solutions you suggested. Thanks for descr
Arne,
Fair enough. NiFi could perhaps be smarter about looping connections instead of
stopping at self-loops.
Another approach to this situation, which I have used, though, would be rather
than having a flow that loops like you laid out
with PublishJMS -> LogAttribute -> Back to PublishJMS,
you
Hi Mark,
Thanks for clarifying that self-looping connections will still be processed in
back pressure situations.
For this specific case, we can probably live without the additional routing to
the logging component and back.
I think, however, that there are cases when such ping-pong routing in
Hi Eric,
Just to elaborate - once you double-click on the connection, it will create a
new bend point as Pierre noted. this allows you to
then bend the line however you'd like. In addition to this, though, once a bend
point has been created, you can click on a
connection's label and drag it to a
Hi Arne,
Generally, the approach that is used in such a situation would be to route
failure back to the PublishJMS processor
itself (without diverting first to a LogAttribute processor). The PublishJMS
processors itself should be logging an error
with the FlowFile's identity. Then, troubleshooti
Hi Eric,
You can add a bending point by double clicking on the black line of the
relationship (and remove a bending point by double clicking on it).
Pierre
2017-10-23 11:38 GMT+02:00 Eric Chaves :
> Hi, my flow has a lot processors coming out from a single router processor
> which makes the dia
Hi,
We came across a situation when we experience a kind of “back pressure dead
lock”.
In our setup, this occurs around PublishJMS when the target JMS queue is full.
Please find attached a screenshot of the relevant flow.
The failure relation we route to a logging component, and then back t
Hi everybody,
I am developing a custom ingest processor that writes data out as a binary
avro stream. Currently I simply store the schema in an attribute of the
flowfile and that works quite nicely with an AvroRecordReader deserializing
it.
To make things "nicer" I thought I'd have a look at the
Hi, my flow has a lot processors coming out from a single router processor
which makes the diagram a mess with queue's boxes overlapping each other.
sometimes when I connect two processors the queue box connecting them has a
small yellow square that allows moving the box around producing angled
li
26 matches
Mail list logo
