Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
I have rebased my branch and created new pull request here:
https://github.com/apache/nifi/pull/2553. Please ignore this pull request as
the new one supersedes it.
---
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
Sure, I will start re-basing now.
---
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2493
Pulsar looks interesting, so if you can do the rebase I'll try to help out
with the review.
---
Github user MikeThomsen commented on the issue:
https://github.com/apache/nifi/pull/2493
@david-streamlio Looks like you pulled in about 30 some commits from other
folks into this. Can you rebase the branch so that only your commits are part
of the PR?
---
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
At the risk of sounding pushy.Is there anything else I need to do on
this PR, such as add the record-based processors?
---
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2493
on the consume side you would generate a flowfile. But you'd then write to
it via the streaming API. #neverbytearray is a movement we should start.
---
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
I completely agree on your observation about having to land the entire,
unbroken file into NiFi's repo rather than streaming through memory is
inefficient. I will talk to the Pulsar team
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2493
yes correct it would be awesome if the client/broker interface could
support a genuinely streaming interface rather than a byte[] as this helps
frameworks that can operate on the streams rather than
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
The Pulsar client API, https://pulsar.apache.org/api/client/, currently
only supports byte[] payloads. By true streaming, I am assuming you mean an API
that return an IO stream object of some
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2493
@david-streamlio i'm supportive of doing non record ones for the cases you
mention. Does Pulsar by any chance offer a true streaming API instead of
byte[] then so handling the types of objects you
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/2493
@david-streamlio HBase is an example of what @markap14 suggested.
https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-standard-services
and there is a new PR (#2498) currently
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
@markap14. I completely agree that is the best approach for handling
text-based payloads into and out of Pulsar, and am starting to work on the
record based processors now. However, I do see
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/2493
@david-streamlio the idea behind the record-oriented processors is that
they make building your flow in NiFi much easier. Without that, you end up
having to split your data up into tons of FlowFiles
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
I created a separate JIRA for record-based
processors.https://issues.apache.org/jira/browse/NIFI-4914, but would like
to get these processors into the release as well. Primarily because
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2493
@david-streamlio @markap14 i'd recommend we dont even bother with non
record based producers/consumers with Pulsar.
---
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
Thanks @markap14. This has been a great learning experience for me, and I
am happy to contribute back to such a great project. I am still new to
developing processors, so I really appreciate
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/2493
Hey @david-streamlio this is very cool! I've been thinking about writing
processors for interacting with Pulsar myself but haven't had a chance yet.
Just a few things that we should think through a
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
When I started writing these processors, I used the Kafka ones as a model.
However, I felt that it would be easier to configure the Pulsar client once,
such as SSLContextService integration,
Github user tasanuma commented on the issue:
https://github.com/apache/nifi/pull/2493
Hi @david-streamlio. Thanks for the work. This would really helpful for
Pulsar users.
I have one question. Are there any reasons that this processor uses
Controller Service? I think this is
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
I have updated all the nifi component version references to 1.6.0-SNAPSHOT,
and successfully rebuilt the project using mvn -Pcontrib-check clean install.
---
Github user joewitt commented on the issue:
https://github.com/apache/nifi/pull/2493
please update all the nifi component version references to 1.6.0-SNAPSHOT
---
Github user david-streamlio commented on the issue:
https://github.com/apache/nifi/pull/2493
NIFI-4908
---
22 matches
Mail list logo
