Hello,
I would like to use the existing ScrollElasticsearchHttp to perform a search
that returns a potentially large number of hits. The parameters of the search
need to reference one or more flow file attributes.
Looking at the source code for this processor it appears that the QUERY
ks/destinations available. I'll follow those
> up with a PrometheusRecordSink and perhaps some KafkaRecordSink
> implementations as well.
>
> Happy to discuss this more, and please feel free to look at the PR and
> comment on it as you like, I'm always looking for feedback!
>
> Reg
Thanks Mike,
I understand what you are saying but I am really trying to avoid having a
secondary flow in NiFi if I can avoid it.
It seems like NiFi was designed to allow this kind of custom reporting task and
that I should ideally be able to do this without using the NiFi-provided S2S
I would like to implement some custom monitoring logic that captures certain
information from the provenance repository. It would be similar in some ways to
the existing SiteToSiteProvenanceReporting task, but it will not be sending
information to another NiFi node but instead sending things to
I am trying to understand the use of the NiFi registry’s bundles API endpoints
in version 0.4. We have numerous custom processors and controller services and
I am wondering if these new APIs will help manage versions for these custom
components.
Are these APIs ready for production use? Are
the packaging work that the Maven
> NAR plugin does? If not, that's probably a lot of the problem right there.
>
>> On Tue, May 21, 2019 at 10:19 AM Tim Dean wrote:
>> Hello,
>>
>> I have a custom NiFi processor and controller service that makes use of the
>&g
Hello,
I have a custom NiFi processor and controller service that makes use of the
JSON streaming API, as defined by JSR-374. This API is defined by an API jar
file and requires a separate jar file with an implementation of the API. I have
chosen to use the Glassfish reference implementation.
I am trying to design a flow where I search against an ElasticSearch index, do
some processing on each document in the result set, and then merge the
resulting processed documents into a single JSON document. I want to make sure
that the merged document contains ALL of the results that were
n the
> queue, but in v2 A is no longer connected to B, you won't be able to
> upgrade until you clear the queue which can be done by stopping A
> first and letting it bleed out.
>
> All other changes besides this can be applied in place and everything
> will continue as normal.
>
>
I’m trying to understand what can and cannot happen to in-process flow files
during an upgrade for a versioned process group. For example, what if version 1
of a PG has several flow files queued up waiting to be processed when I upgrade
to version 2? Will NiFi prevent me from upgrading until
be
> valid so it would enabled, and third time through there are no more
> services to enable.
>
> If you are working Python then I believe NiPyApi also has a similar command.
>
> Thanks,
>
> Bryan
>
> On Thu, Feb 21, 2019 at 5:14 PM Tim Dean wrote:
>&
with the services inside a nested versioned process group and it
> is definitely a bug, although I don't know where yet.
>
> I created this JIRA - https://issues.apache.org/jira/browse/NIFI-6001
>
> Thanks,
>
> Bryan
>
> On Wed, Feb 6, 2019 at 2:04 AM Tim Dean wr
S2.
>
> That doesn't mean there isn't an issue though. Let us know if you
> figure out a specific scenario that creates a problem.
>
> On Tue, Feb 5, 2019 at 11:57 AM Tim Dean wrote:
>>
>> Thanks again Bryan
>>
>> For my current use case I think I can ge
inally meant to be done through the UI
> which is why it is a bit tricky to script, but I would really like to
> get all of this stuff built into the CLI.
>
> -Bryan
>
>> On Tue, Feb 5, 2019 at 12:49 AM Tim Dean wrote:
>>
>> Bryan,
>>
>> I’ve been wo
eral you'd have to start
> at the lowest level, and work your way up the levels, applying the
> correct ids from the level below.
>
> -Bryan
>
>
> On Fri, Feb 1, 2019 at 11:44 AM Tim Dean wrote:
>>
>> Thanks Bryan -
>>
>> If I use a nested versioned process
a cohesive
> unit.
>
> -Bryan
>
> On Thu, Jan 31, 2019 at 7:00 PM Tim Dean <mailto:tim.d...@gmail.com>> wrote:
> I am trying to automate deployment of a NiFi flow with several versioned
> process groups using the NiFi APIs. The basic setup I have is this:
> I h
I am trying to automate deployment of a NiFi flow with several versioned
process groups using the NiFi APIs. The basic setup I have is this:
I have a dozen or so process groups, each of which has been versioned within a
NiFi registry
My root process group contains each of those process groups,
he minimums are not reached and
> it will get kicked out based on the 5 min timer.
>
> Lastly, consider switching to using MergeRecord and having JSON
> readers/writers in it. It will take care of the framing you're trying
> to do with these demarcators.
>
> Thanks
> Joe
as 'good enough'. Consider making
> this larger than 10 but if you know there aren't more than 10 needed
> then you're good. You also dont want to go wild with this value
> either as it can result in more memory usage than necessary.
>
> Thanks
>
>
> On Tue, Au
uan Sequeiros wrote:
>
> Hi,
>
> For:
>
> Correlation Attribute Name: ${myFlowfileAttributeName}
> Should be set to: myFlowfileAttributeName
>
> NOT ${myFlowfileAttibuteName)
>
> Hope that helps.
>
> On Tue, Aug 28, 2018 at 12:07 PM Tim Dean <mailt
I have a flow that sends a large number of JSON files into a MergeContent
processor. The job of that processor is to combine all the incoming flow files
with a particular flow file attribute into a single flow file, creating a JSON
array containing each of the input flow files’ JSON.
I have
ing them, each implementation only depends on the API and
> doesn't need to worry about each other's specific implementation
> dependencies.
>
> Thanks,
>
> Bryan
>
>
> On Mon, Jul 30, 2018 at 2:35 PM, Tim Dean wrote:
>> We’ve got a handful of custom NiFi controlle
We’ve got a handful of custom NiFi controller services. The documentation
describes how to properly separate the API of a custom service from its
implementation using different NAR files. When we don’t do that, we see a
message that says the following:
org.apache.nifi.nar.ExtensionManager
Mike -
The standard build of NiFi includes the bin/nifi.sh script, and one of the
options for that script is the install command. When executed, that command
writes a file called /etc/init.d/nifi and also manipulates a few links. I
believe that something within that file and link creation causes
Jeff,
We ran into the exact same problem after trying to install as a service on
Ubuntu. Once the service is installed, all subsequent apt-based installs
fail with the message you are seeing.
We worked around it by writing our own script to install the service in the
correct way for Ubuntu. I've
fi.pid | xargs kill -3"
>
>
>
>> On May 23, 2018, at 11:13 AM, Tim Dean <tim.d...@gmail.com
>> <mailto:tim.d...@gmail.com>> wrote:
>>
>> Joe -
>>
>> I am currently in this state where new provenance events are not showing up
>>
nt :)
>
> It is possible indexing was stuck with the older implementation. Can
> you run 'bin/nifi.sh dump' and share the logs/nifi-bootstrap.log
> file if it is in that state/behavior again?
>
> Thanks
>
> On Tue, May 22, 2018 at 1:33 PM, Tim Dean <tim.d...@gmai
sharding/etc.. that help with performance but the above are
> good to do now regardless of performance.
>
> Thanks
>
> On Tue, May 22, 2018 at 10:50 AM, Tim Dean <tim.d...@gmail.com> wrote:
>> Thanks Joe:
>>
>> I have not yet made any changes to the configuration
>
> Tim,
>
> The default configuration for provenance event retention is
> potentially a factor.
>
> Did you make any changes to those? Can you share relevant segments
> from the nifi.properties file?
>
> Thanks
>
> On Mon, May 21, 2018 at 8:32 PM, Tim Dean <
make it easier to experiment with the search
> results. The provenance search will show the oldest event date in the top
> right, and with no search filter you should see the most recent events.
>
> Does that help?
>
> Thanks,
>
> James
>
>
> On Mon, May 2
Hello,
I am having a hard time troubleshooting a NiFi flow to see where things are
failing. I am trying to look at the provenance repository for a variety of
processors, but for some reason nothing more recent seems to be appearing
there. For example:
At approximately 10:30 this morning I
I am creating a handful of custom NiFi processors, and a couple of controller
services that are to be used by those processors. For a variety of reasons,
those services are currently being developed independently. In other words,
each service will be in its own nar file.
Most of my processors
services.
>
> I don't see any issues with memory as long as your cache doesn't grow
> indefinitely.
>
> -Bryan
>
>
> On Tue, May 1, 2018 at 6:47 AM, Otto Fowler <ottobackwa...@gmail.com> wrote:
>> https://hc.apache.org/httpcomponents-client-ga/tutorial/h
to the web service.
-Tim
> On May 1, 2018, at 5:47 AM, Otto Fowler <ottobackwa...@gmail.com> wrote:
>
> https://hc.apache.org/httpcomponents-client-ga/tutorial/html/caching.html
> <https://hc.apache.org/httpcomponents-client-ga/tutorial/html/caching.html> ?
>
>
>
Hello,
I have a custom NiFi controller service that retrieves data from an external
web service via HTTP requests. The results from these HTTP requests will be
needed at various points throughout my process flow. In some situations, I
could end up needing to access the HTTP response dozens or
> .description("LookupService test processor")
> .identifiesControllerService(LookupService.class)
> .required(true)
> .build());
> return properties;
> }
>
> }
>
>
> On April 27, 2018 at 17:25:26, Otto
The NiFi developer documentation does a good job of describing how to unit test
custom processors, including how to configure a controller service that the
processor depends on. However, it doesn’t provide information on how to best
unit test a custom controller service.
The biggest challenge
Hello,
I have a custom NiFi processor that invokes an external HTTP endpoint. That
endpoint will be hosted by services running at customer sites, and those
customer sites require the ability to define when the service can be called by
my processor. Their goal is to prevent calls from coming in
Hello,
I am working with NiFi 1.5.0 and I have created a customer service controller
for my project. I am able to add my controller service to my NiFi flow
configuration, but I am unable to enable that controller service without first
encountering error messages that I need to dismiss. After
39 matches
Mail list logo
