Ravi
Correct. We've created integrations that we know work with Hive 1.1, Hive
1.2, and Hive 3.x. There has not been work toward Hive 2.x.
The above correlate to CDH, HDP, and HDP respectively.
You can see how there have been Hive bundles for different versions and
could model off that. In
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.9.1.
Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data. Apache NiFi was made for dataflow. It supports highly configurable
directed graphs
of data routing, transformation,
:
> Congratulations everyone!
>
>
> On March 18, 2019 at 09:07:25, Joe Witt (joew...@apache.org) wrote:
>
> Hello
>
> The Apache NiFi team would like to announce the release of Apache NiFi
> 1.9.1.
>
> Apache NiFi is an easy to use, powerful, and reliable system t
Docker build is fixed and out too. Thanks Aldrin
On Mon, Mar 18, 2019 at 11:48 AM Joe Witt wrote:
> Thanks Otto.
>
> To others: Please avoid replying all (which includes announce). we have
> to reject that.
>
> But I hear you on the docker thing...I screwed up that part o
Denes
You might want to ensure FetchFile is able to understand such flowfiles as
well and to skip them so it doesn't try to Fetch a directory and/or it
handles it gracefully.
Thanks
On Mon, Mar 11, 2019 at 12:51 PM Denes Arvay wrote:
> Hi Jim,
>
> I suppose you want to monitor the newly
Hello
The variables of a pg are not, at this time, for sensitive values. You can
set the sens values programatically to ensure they are never shown.
We will likely add support for secrets (ie sensitive variables) but eta
there depends on progress in the community.
thanks
On Thu, Mar 21, 2019,
Hello
If this is a secured instance please ensure your user has the proper
permissions to access prov.
thanks
On Thu, Feb 7, 2019, 8:31 PM Kon Soulianidis Hi,
>
> After upgrading from 1.6 to 1.8 recently, I’ve noticed Provenance events
> aren’t being generated (or if they are they aren’t
dan
we did rc1 this week and will have rc2 up today or tomorrow ideally.
thanks
On Sat, Feb 16, 2019, 10:42 AM dan young Heya folks,
>
> Any insight on 1.9 release date? Looks like a lot of goodies and fixes
> included...
>
> Regards,
>
> Dano
>
and
> set them to be indexed for the provenance, the mentioned rate should be
> alright?
>
> Cheers,
> Ali
>
> On Sat, Feb 16, 2019 at 2:56 PM Joe Witt wrote:
>
>> Ali
>>
>> You certainly can and at the rates you mention you should be able to keep
>> it for
Rich
I haven't experimented with NFS in a long time but early results with NFS
were very uneven. Performance in general was fine but stability as it
related to locking and other behaviors was less desirable. NiFi is no
longer as aggressive with file locks as it used to be so it is possible
Chad,
In your conf/nifi.properties please see what the implementation is for your
provenance repository. This specied on
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
Is that what you have?
The above error I believe could occur if the
...interesting. I dont have an answer but will initiate some research.
Hopefully someone else replies if they know off-hand.
Thanks
On Thu, Feb 14, 2019 at 11:43 AM Pat White
wrote:
> Hi Folks,
>
> Could someone point me at the correct way to modify Nifi's embedded jetty
> configuration
Ali
You certainly can and at the rates you mention you should be able to keep
it for a good while.
Just set the properties you need for your system and measure the rate at
which prov storage fills.
Thanks
On Fri, Feb 15, 2019 at 10:29 PM Ali Nazemian wrote:
> I didn't mean to use Nifi
I agree that there is value in having EL enabled properties for some of the
SSLContext properties. I dont understand the security concern raised but
am open to what I might be missing. It would need variable and env var
access.
Thanks
Joe
On Tue, Feb 19, 2019 at 9:16 PM Beutel, Maximilian <
Ryan,
That block of text that shows up in the log could arguably said "WARN"
because the flow will continue to function as it did before.
However, the reason it is an error is that you really should follow its
advice and specifically follow the secure nifi configuration guidance.
By not
>
> On Thu, Jan 24, 2019 at 10:39 AM Joe Witt wrote:
>
>> Ryan,
>>
>> That block of text that shows up in the log could arguably said "WARN"
>> because the flow will continue to function as it did before.
>>
>> However, the reason it is an error is
hey josef. yeah we need to add a min file age property to ListSftp.
please file a jira.
thanks
On Thu, Jan 24, 2019, 11:13 AM Hi guys
>
>
>
> We need your advice,… we use the ListSFTP processor to read files on a
> remote folder. The files gets written like that:
>
>
>
>- File1
>-
gt; org.apache.nifi.StdErr Failed to start web server: Unable to start Flow
> Controller.
> 2019-01-24 17:33:08,479 ERROR [NiFi logging handler]
> org.apache.nifi.StdErr Shutting down...
>
>
> -Ryan H
>
> On Thu, Jan 24, 2019 at 11:48 AM Joe Witt wrote:
>
>>
Hello
The Flow Registry is designed to deal with challenge #1 as far as versioned
flows go. For the extensions themselves you'll really want to take
advantage of the nar versioning capability since versioned flows can use
specific versioned extensions. Once we make an extension registry happen
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.9.2.
Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data. Apache NiFi was made for dataflow. It supports highly configurable
directed graphs
of data routing, transformation,
mike
lsof -p
with the pid of the actual nifi process is probably better to look at for
nifi resource handling observation. what is that count. yes the jars and
such will all be loaded. you can expect a few thousand off that. then
there are sockets and content and prov and flowfilewhich
Got to about 6500-6800 before hitting
>>> the ceiling.
>>>
>>> On Fri, Apr 12, 2019 at 7:30 AM Joe Witt wrote:
>>>
>>>> mike
>>>>
>>>> lsof -p
>>>>
>>>> with the pid of the actual nifi process is p
Hello
I am not very familiar with beats or this processor specifically but it
does appear to have some beats specific framing that it does by looking at
the processor code. Json based framing appears supported as well but it
isn't clear that it would be or is intended to be valid JSON in a
Dave
Using Wait/Notify would ensure you only have one message in flight at a
time (or it can/should). But the message will be ack'd before processed.
For Kafka and some of these message queue mechanisms if we want to offer a
'do not ack until the whole flow is done' behavior we should update
...this feels like a bug to me. I think Erik-Jan's expectation that
nothing would have begun for ListenUDP given primary node only config is
fair. I also think our current position of 'just not calling onTrigger' is
fair too but less intuitive for users.
What do ya'll think?
On Wed, Jun 5,
You can also identify where top performance hitters are and ensure that a
ControlRate or otherwise throttled amount of data and/or threads are
leveraged at once. This allows you to effectively control how much effort
to put on any single point of the flow at once. This is necessary when you
want
Shanker
It sounds like you've gone through some changes in general and have worked
through those. Now you have a flow running with a high volume of data
(history load) and want to know which parts of the flow are most
expensive/consuming the CPU.
You should be able to look at the statistics
Joe
When you view top or other tools what is dominating the cpu?
thanks
joe
On Sun, Jun 9, 2019, 5:35 PM Joe Gresock wrote:
> I posted about this a while back on 1.6.0, but as far as I can tell it has
> only gotten worse in 1.9.2.
>
> I have a cluster of 7 nifi nodes running on CentOS 6 VMs.
the pipeline is *just
> too less* to throttle my CPU ideally.
>
> The machine config and NiFi config remains untouched - this has left me
> confused where the problem might be. Something which had been running
> smoothly since months, has become a challenge now.
>
> On Fri
des and throttled at
>1600%.
>
>
> Meanwhile, I am trying to clear up all FlowFiles from disk and start the
> flows afresh.
>
>
> On Mon, Jun 10, 2019 at 5:42 PM Joe Witt wrote:
>
>> Sneh
>>
>> It was stable for months but now is high...
>>
>
ons in version 1.8 to resolve these? If not then I
> will plan to upgrade soon. If there are any quicker solutions which can be
> applied in 1.8 without any data loss then I ll upgrade it later.
>
> On Thursday, May 9, 2019, Joe Witt wrote:
>
>> Suman
>>
>> Yea
Suman
Yeah it looks related to the queue/load balance fixes. Latest release
should be much better for you.
Thanks
On Thu, May 9, 2019, 5:30 AM Suman B N wrote:
> Team,
> We are running a 3 node nifi cluster in docker. Version is 1.8.
> Everything has been running smoothly from the last 2-3
Jon
Just want to make sureare you sure each relationship is unique (1..7)
and not copied? Could you share a screenshot perhaps?
Thanks
On Thu, May 23, 2019 at 11:48 AM Jon Belanger <
jon.belan...@fidelissecurity.com> wrote:
> I’ve got a Distribute Load processor with a single incoming
Jianan
We have not done the work to fully isolate the concept of a NiFi Archive
(Nar) such that it could be used outside of NiFi as a general classloader
isolation pattern.
There was one other person interested in helping make this happen in the
past but I'm not sure where it has gone.
With
James
For distributing work across the cluster the load balanced connection
capability in NiFi 1.8 and beyond is the right answer - purpose built for
the job. I'd strongly recommend upgrading to avoid use of s2s for this
scenario and instead use load balanced connections. When using load
planation correctly: behavior exhibited through the first 4000 flowfiles
> as past performance may not represent future results. It will do what it
> does, and I may find that node1 does get loaded as I work through flowfiles
> in steady state.
> Again, thanks.
>
> On Wed, Jul 1
Hello
I suspect you have to add the new policy. Please see in the migration
guide from old version you had until now.
Thanks
On Thu, Jul 11, 2019, 2:56 PM Mikhail Rolshud (BLOOMBERG/ 120 PARK) <
mrols...@bloomberg.net> wrote:
> Hi,
>
> We noticed that after some time 1.9.2 stops showing
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.9.2.
Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data. Apache NiFi was made for dataflow. It supports highly configurable
directed graphs
of data routing, transformation,
Hello
You could see hung threads like this because the processor is simply taking
a long time to do its task (possible in certain listing cases but probably
not ListS3), or because it is truly stuck such as a live-lock or timeout
condition it has hit. These are almost always bugs and avoidable.
Dave,
Generally such a case where a processor combined with a flowfile can result
in an error of some kind should have a failure relationship (or similarly
named) and the flowfile should go there. However, some processors in
certain cases will just rollback/fail and the data will sit in the
Noe
Just activate compression on the s2s port and the client will honor it if
able. I dont believe the protocol has changed in quite a while so you
should be fine with the versions noted.
Thanks
Joe
On Mon, Jul 15, 2019 at 9:08 AM Noe Detore wrote:
> Hello,
>
> What is the best way to
Jason
The work dir gets created at startup and possible as new nars are loaded.
I think you'd be ok to scan this.
The flowfile and content repository and provenance directories as
configured should be skipped. The logs dir should be skipped. The state
directory should be skipped. All else I
Peter
All the details you can share on this would be good. First, we should be
resilient to any sort of repo corruption in the event of heap issues.
While obviously the flow isn't in a good state at that point the saved
state should be reliable/recoverable. Second, how the repo/journals got
Muazma,
It is strongly recommended to have a single shared registry across the
environments if your policies allow. This will give the best (by design)
experience of porting flows from one environment to another. The remaining
challenges you see with this is that you will have to enter things
<
giulia.scalabe...@genomedics.it> wrote:
> Hi all,
>
> As requested by Joe Witt, here I attach the dump log after the lock.
>
> My Nifi after nearly 30 days of scheduled activity has locked as usual…
>
>
>
> BG,
>
> Giulia
>
>
>
> *Da:* Joe Witt [mailt
adam
can you confirm which list and ask them to forward. A couple of us
moderate but nothing has lingered that i know of.
if you know time and subject that helps too.
thanks
On Wed, Sep 4, 2019 at 6:56 PM Adam Taft wrote:
> I am trying to help a colleague get a message through to the user
Just to follow up Adam sent me some more detail offline. No sign of the
emails so recommend subscribing and trying again.
Thanks
On Wed, Sep 4, 2019 at 8:06 PM Joe Witt wrote:
> adam
>
> can you confirm which list and ask them to forward. A couple of us
> moderate but nothing
quick reply: There is a zookeeper state migrator utility in the toolkit I
believe. That should be quite helpful.
http://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#zookeeper_migrator
Thanks
On Tue, Sep 17, 2019 at 11:35 AM Noe Detore wrote:
> Hello,
>
> I am currently using a
Hello
The 100 threads for the controller overall is the maximum number of threads
that could run concurrently. On a 16 core system and a flow which is very
I/O bound this is definitely achievable. Generally you want to look at
some multiple of the number of physical cores such as 2,4,8, etc..
Giulia,
When you're experiencing this condition can you capture a thread dump and
share the logs (bootstrap/app)? To create this you can run
/bin/nifi.sh dump threaddump-locked-prov.log
Thanks
On Thu, Aug 1, 2019 at 3:57 AM Giulia Scalaberni <
giulia.scalabe...@genomedics.it> wrote:
> Hi,
>
>
thanks for reporting. please file a jira showing your steps and example
to reproduce.
thanks
On Wed, Jul 17, 2019 at 8:43 AM Dnyaneshwar Pawar <
dnyaneshwar_pa...@persistent.com> wrote:
> Hi,
>
> We are using QueryRecord processor to read and parse the the CSV files
> using CSV Reader as
Joseph
You are absolutely right that it would be terrible to have to edit the
truststore on the nifi server(s) each time you wanted to add a client
cert. You're also right that there is a way to never do this. I'll poke
around for some links to help send you in the right direction.
Thanks
On
at 11:58 AM Joe Witt wrote:
> Joseph
>
> You are absolutely right that it would be terrible to have to edit the
> truststore on the nifi server(s) each time you wanted to add a client
> cert. You're also right that there is a way to never do this. I'll poke
> around for some link
Yeah to double down on Bryan's comments I know this is frequently done.
For instance we often would pull data from a website that we'd use for
lookups/enrichments in the flow. We'd use GetHTTP or related processors to
grab the contents of the given URL constantly and honor things like
Craig
I plan to RM the release and am awaiting a set of jiras tagged as 1.10 and
with a lot of review traction to merge. I also think it makes sense to
attempt to scan and find lingering prs as there is a lot of good work there
that needs review attention.
But I am hopeful 1.10 rc processes
...hmmm good point. We need to clarify this in the migration guide and
possibly elsewhere!
Thanks
On Fri, Nov 8, 2019 at 1:18 PM Jeff Zemerick wrote:
> Pierre, thanks for confirming what I was seeing.
>
> Jeff
>
> On Fri, Nov 8, 2019 at 12:08 PM Pierre Villard <
> pierre.villard...@gmail.com>
Peter
The most common challenge is if two isolated instances both want to use a
native lib. No two native libs with the same name can be in the same jvm.
We need to solve that for sure.
Thanks
On Mon, Nov 11, 2019 at 9:53 AM Peter Turcsanyi
wrote:
> Hi Hai Luo,
>
>
bcc: dev/users@nifi.a.o
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.10.0.
Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data. Apache NiFi was made for dataflow. It supports highly configurable
directed graphs
of data
I read/replied too fast - if you mean that you want them together but
sorted by date then it makes sense we'd offer a SortRecord processor. If
you wanted to simply group them by month then PartitionRecord should do the
trick.
On Tue, Dec 3, 2019 at 8:10 AM Joe Witt wrote:
> Sounds l
Sounds like a perfect use of PartitionRecord. And if you wanted larger
bundles of such things you could then follow it with MergeRecord correlated
on that same partitioned value.
Thanks
On Tue, Dec 3, 2019 at 8:09 AM Mike Thomsen wrote:
> We need to be able to split a record set by examining
Sanjeet
Your e-mail was sent 10 hours ago and is during the weekend. Please be
patient.
I would do two nars. The controller service nar and the processor nar
which depends on it. This way you can have other processor nars that also
depend on that controller service nar if necessary.
Thanks
Lei
Please do not post to both users and dev list. Dropping dev.
Yes these scenarios are based on limitations of a given processor
implementation or the nature of a given protocol/mechanism. NiFi enforces
back pressure by slowing/stopping scheduling a component and making fact of
back pressure
Lei
The order won't necessarily be the same. You'd want EnforceOrder to follow
the load balanced connection most likely.
It is important to keep in mind the queues are basically insertion order
and the system is inherently multi-threaded so the data can be shuffled in
such cases. You can
Hello
You should only have 1 or a few tasks at most for this processor.
Scheduling can be frequent but choosing different options and seeing for
your case is best.
This processor is relatively difficult to configure correctly as it is a
complex case and has powerful options. What you will need
is a daily effort at this point. i am close to pushing first rc. have
been watching for stability on bug fixes.
On Fri, Oct 18, 2019 at 1:10 PM Juan Pablo Gardella <
gardellajuanpa...@gmail.com> wrote:
> Any ETA for Nifi 1.10 release?
>
> On Fri, 18 Oct 2019 at 13:39, Mike Thomsen wrote:
>
>>
Hello
Is the idea to have a place to store Jolt specifications that you could
then access in various components?
If so a simple ControllerService such as 'JoltSpecControllerService' which
has a list of keys (names of specs) and values (the spec) would probably do
the trick.
Thanks
On Wed, Nov
figured
> through the UI but there could be other future implementations that are
> based on fetching from remote services, etc.
>
> Thanks
> -Mark
>
> Sent from my iPhone
>
> On Nov 20, 2019, at 10:28 AM, Joe Witt wrote:
>
>
> Yeah filing a JIRA would be good. Cont
It is likely the default settings in nifi.properties should be changed for
provenance. Have they?
Thanks
On Fri, Nov 29, 2019 at 1:39 PM Luis Carmona
wrote:
>
> Hi Dieter,
>
> I got that problem once, with version 1.10, and was finally solved once I
> corrected the configuration of Max Open
You can use SASL/Plain today with the kafka 2 procs in NiFi but as noted it
is unpleasant to configure. A much easier/clear configuration is being
worked on right now. Not positive what the JIRA for it is though. Should
be available quite soon.
Thanks
On Thu, Oct 10, 2019 at 9:31 AM
Tomas
It just need someone to implement various standards. Right now I believe
it is purely TLS one way or mutual auth and also supports basic and digest.
Thanks
On Thu, Oct 10, 2019 at 11:00 AM Tomas Hudik wrote:
> Hi Erik
> thank you very much for the mail.
>
> My try is not about coding
Jean
Id recommend switching to the new provenance repo called
WriteAheadProvenaceRepository. Look at a new nifi downloads
nifi.properties as it has been the default for a while. This will help the
prov stuff. You may also want to stop using g1gc if on java 8.
I cant explain the status history
Chandra
Something is making them compile scope likely. Use maven tools like
help:effective-pom to see what the actual pom is when building the nar.
You can then zero in by doing the same on the processor bundle.
thanks
On Mon, Oct 14, 2019 at 7:51 AM Chandrashekhar Kotekar <
Dan,
Not sure what others see but for me your email cuts off in the middle of a
line.
You might want to file a JIRA with your observation/logs.
Thanks
On Wed, Feb 12, 2020 at 11:46 AM dan young wrote:
> Hello,
>
> We're seeing the following messages in nifi logs on our cluster nodes. Using
oduction,
> but we want to validate the correctness first.
>
> -Eric
>
> On Mon, Feb 24, 2020 at 10:45 PM Joe Witt wrote:
>
>> Eric
>>
>> You wrote a great email and all your points make a lot of sense. I'll be
>> blunt and state - we didn't design fo
Dano
You can obtain Apache NiFi 1.11.2 from the archives if you need it.
http://archive.apache.org/dist/nifi/1.11.2/
We released Apache NiFi 1.11.3 last night and I'll send an email on that
later today (waiting for mirrors to catch up).
The primary push for 1.11.3 was to resolve a resource
Josef
I dont know what role, if any, nifi can play in keeping open files on a
remote system. If the system on which nifi runs has open resources then we
can take action.
It is possible this could occur if nifi is being configured to pull data
while a writing process is not yet done with sending
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.11.3.
Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data. Apache NiFi was made for dataflow. It supports highly configurable
directed graphs
of data routing, transformation,
Eric
You wrote a great email and all your points make a lot of sense. I'll be
blunt and state - we didn't design for that and we should have.
When we launched the NiFi Registry we were seeing a lot of large scale
monolith NIFi clusters being deployed where each cluster ran the flows of
many
bcc: dev/users@nifi.a.o
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.11.0.
Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data. Apache NiFi was made for dataflow. It supports highly configurable
directed graphs
of data
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.11.1.
Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data. Apache NiFi was made for dataflow. It supports highly configurable
directed graphs
of data routing, transformation,
Hello
The Apache NiFi team would like to announce the release of Apache NiFi
1.11.2.
Apache NiFi is an easy to use, powerful, and reliable system to process and
distribute
data. Apache NiFi was made for dataflow. It supports highly configurable
directed graphs
of data routing, transformation,
Either way the answer is to use load balanced connections to single node
once data is on the desired cluster. You still cannot send to a ‘specific
node’ but you can to a single node.
Consider the case where data was split up for various reasons but needs to
be all back together. Load balancing
Hello
I cannot quite gather what you're observing from your description or the
image. Can you send an image that is a bit zoomed out to show the full
picture of the flow that is impacted?
Thanks
On Mon, Feb 10, 2020 at 5:48 AM Etienne Jouvin
wrote:
> Hello All.
>
> Here is an extract of my
it is under vote now. so within a day or so possibly
thanks
On Tue, Feb 18, 2020 at 9:26 AM dan young wrote:
> Howdy folks,
>
> Just curious what the word on the street for releasing 1.11.2
>
>
> Regards,
>
> Dano
>
>
Hello
This is likely a corrected issue with a certain usage pattern. Try a newer
version and see if less data is retained.
That said in correct operation you can tune retention behavior with repo
settings.
Thanks
On Fri, Jan 17, 2020 at 10:54 PM Jean-Sebastien Vachon <
jsvac...@brizodata.com>
because splitting them just to achieve this means creating potentially
thousands of flowfiles rather than playing the data as it lies and in its
most efficient form.
The idea to enable certain automatically managed things which one could
inject into their records like record number (relative to
> On Fri, Jan 3, 2020 at 1:32 PM James McMahon wrote:
>
>> It would be very useful indeed. Thanks very much for your comments.
>>
>> On Fri, Jan 3, 2020 at 1:29 PM Joe Witt wrote:
>>
>>> because splitting them just to achieve this means creating potentiall
Eric,
It is probably easier to use blogspot or something like that. But if you
want to offer a guest submission I'm sure we can figure it out for the
Apache blog too. I'm just not sure on the steps.
Thanks
On Thu, Mar 5, 2020 at 11:27 AM Eric Ladner wrote:
> how would I submit something to
The provenance repo is in large scale use by many many users so
fundamentally it does work. There are conditions that apparently need
improving. In the past couple days these items have been flagged by folks
on this list, JIRAs and PRs raised and merged, all good. If you can help by
creating a
Eric
Thanks for reporting. If you dont mind/can do please file a JIRA with this
description in it and version of nifi and registry you are using.
Thanks
On Tue, Apr 14, 2020 at 7:41 PM Eric Secules wrote:
> Hello everyone,
>
> I am noticing that when I delete a parameter from a parameter
James
Using the provenance events from this processor is the best way. Grab all
receive events for the time period of interest.
You can do this in a few ways but one that works well is to send prov
events via reporting task, filter events for that component, write those
out to a file or set of
ll search for
> the example.
>
> On Tue, Apr 14, 2020 at 8:08 AM Joe Witt wrote:
>
>> James
>>
>> Using the provenance events from this processor is the best way. Grab
>> all receive events for the time period of interest.
>>
>> You can do this in a few w
If the zk-migrator can be configured to write to stdout instead of a file
then yes.
On Fri, Apr 10, 2020 at 3:52 PM sanjeet rath wrote:
> Hi,
>
> Thanks for your quick reply.Yeah i am using executestreamcommand to
> execute bellow script
>
> zk-migrator.sh -s -z
>
PG1 should load data into Target1 then send the data to PG1-out.
PG1-out should then be sent to PG2/3/4-In at the same time.
PG2/3/4 is triggered by arrival of this data and does its full DB
lookup/etc.. to do its thing.
It sounds like the vital point here is you want to ensure each time
Take a look at DistributeLoad.
thanks
On Fri, Apr 24, 2020 at 7:05 AM Darren Govoni wrote:
> Hi
>
> Let's say I have a splitjson processor. I want to connect 10 processors to
> it such that it will send one output to one processor in an evenly
> distributed manner.
>
> Can Nifi do this?
>
>
gt; processor would distribute flowfiles.
>
> Is there a special use of this processor to have to evenly distribute the
> received flowfiles across outbound queues?
>
> thanks!!
> --
> *From:* Joe Witt
> *Sent:* Friday, April 24, 2020 8:08 AM
> *T
Robert,
Can you please detail the record readers and writers involved and how
schemas are accessed? There can be very important performance related
changes in the parsers/serializers of the given formats. And we've added a
lot to make schema caching really capable but you have to opt into it.
Mike
With that setting you're telling NiFi to run that processor no more than 8
times per second per thread. If the processor involved is written to do so
in batches then this can be ok. But often you just want to let it rip and
set it to 0 as the scheduling which means give it a thread to run
Yeah what a cool note and thanks for taking the time to share.
On Tue, May 12, 2020 at 3:49 PM Andy LoPresto wrote:
> Thanks Boris, this is really interesting to read and I appreciate that
> you’re sharing it with the community. We’re glad NiFi can help with these
> important use cases.
>
>
and I had forgotten to release them :/. Just did that.
On Mon, Mar 23, 2020 at 5:58 AM Mike Thomsen wrote:
> As a rule of thumb, you need to wait until there's a formal announcement
> email to be able to safely get all of the artifacts.
>
> On Mon, Mar 23, 2020 at 5:56 AM wrote:
>
>>
601 - 700 of 927 matches
Mail list logo
