[jira] [Commented] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[ https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680675#comment-16680675 ] ASF GitHub Bot commented on NIFI-5794: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3159 > ConsumeKafka and PublishKafka should allow empty string demarcator > -- > > Key: NIFI-5794 > URL: https://issues.apache.org/jira/browse/NIFI-5794 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > ConsumeKafka processors should allow "empty string" as a message demarcator. > This would allow consuming Avro data without the > serialization/de-serialization cost while still allowing the use of Record > processors once the data is in NiFi. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[
https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680672#comment-16680672
]
ASF GitHub Bot commented on NIFI-5794:
--
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/3159
LGTM +1. I was able to confirm the expected benefit of using an empty
demarcator string to combine multiple Avro formatted Kafka messages into a
single FlowFile. Thanks @pvillard31!
For those who want to use this improvement with older NiFi versions, using
EL `${empty}` can mimic the non-empty validation and result the same effect .
This assumes there's no variable named `empty`, the EL evaluation result will
be an empty string while this is a valid configuration since it's not an empty
property value.
> ConsumeKafka and PublishKafka should allow empty string demarcator
> --
>
> Key: NIFI-5794
> URL: https://issues.apache.org/jira/browse/NIFI-5794
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> ConsumeKafka processors should allow "empty string" as a message demarcator.
> This would allow consuming Avro data without the
> serialization/de-serialization cost while still allowing the use of Record
> processors once the data is in NiFi.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[ https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680673#comment-16680673 ] ASF subversion and git services commented on NIFI-5794: --- Commit d5bce91970b287009dc09d0ce197a537dc0be207 in nifi's branch refs/heads/master from [~pvillard] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=d5bce91 ] NIFI-5794 Allow empty string demarcator in ConsumeKafka processors This closes #3159. Signed-off-by: Koji Kawamura > ConsumeKafka and PublishKafka should allow empty string demarcator > -- > > Key: NIFI-5794 > URL: https://issues.apache.org/jira/browse/NIFI-5794 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > ConsumeKafka processors should allow "empty string" as a message demarcator. > This would allow consuming Avro data without the > serialization/de-serialization cost while still allowing the use of Record > processors once the data is in NiFi. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[ https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679642#comment-16679642 ] ASF GitHub Bot commented on NIFI-5794: -- GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3159 NIFI-5794 Allow empty string demarcator in Consume/PublishKafka proce… …ssors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5794 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3159.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3159 commit 88fbcffdc3fa6aadac14c23af9a5012edaa21e0f Author: Pierre Villard Date: 2018-11-07T09:55:10Z NIFI-5794 Allow empty string demarcator in Consume/PublishKafka processors > ConsumeKafka and PublishKafka should allow empty string demarcator > -- > > Key: NIFI-5794 > URL: https://issues.apache.org/jira/browse/NIFI-5794 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > ConsumeKafka(_*) and PublishKafka(_*) processors should allow "empty string" > as a message demarcator. This would allow consuming Avro data without the > serialization/de-serialization cost while still allowing the use of Record > processors once the data is in NiFi. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[ https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679648#comment-16679648 ] ASF GitHub Bot commented on NIFI-5794: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3159 In the end, I only changed the ConsumeKafka processors since it does not make sense in PublishKafka (one would just have to not set any demarcator to get the same result: all messages in the flow files are sent as one single message in Kafka). The main interest of this PR is to increase efficiency when we consume Avro data from Kafka: there is no need to set a demarcator and we do want to have multiples messages in the same flow file. I've used this [template](https://gist.github.com/pvillard31/57e627d206e7965cfb87cc27c09caf44) to test the change and to confirm the behavior: when pulling Avro data from Kafka with "empty string" as demarcator to have multiple messages in the same flow file, we are able to process the records in the next processor using a ConvertRecord for instance. > ConsumeKafka and PublishKafka should allow empty string demarcator > -- > > Key: NIFI-5794 > URL: https://issues.apache.org/jira/browse/NIFI-5794 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > ConsumeKafka(_*) and PublishKafka(_*) processors should allow "empty string" > as a message demarcator. This would allow consuming Avro data without the > serialization/de-serialization cost while still allowing the use of Record > processors once the data is in NiFi. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
