[jira] [Commented] (NIFI-2826) GetEventHubProcessor Support Enqueue Time
[ https://issues.apache.org/jira/browse/NIFI-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542565#comment-15542565 ] Joseph Percivall commented on NIFI-2826: Hey [~josephxsxn] you had a couple check style issues (revealed by running "mvn clean install -Pcontrib-check") but I fixed them and amended your commit. +1 Visually verified code and did a contrib check build (passed after my fix was added). In a standealone instance, tested using Azure Event hub and IoT hub. Properly saw data come in as it was produced and also saw historical data come in when the enqueue time was set. Thanks for the contribution Joe! I look forward to reviewing more of your commits in the future. I will merge it in. > GetEventHubProcessor Support Enqueue Time > - > > Key: NIFI-2826 > URL: https://issues.apache.org/jira/browse/NIFI-2826 > Project: Apache NiFi > Issue Type: Improvement > Environment: centos6, jdk1.8, nifi 1.0.0 >Reporter: Joseph Niemiec >Assignee: Joseph Niemiec >Priority: Minor > Fix For: 1.1.0 > > Attachments: > 0001-NIFI-2826-Adding-enqueue-time-to-GetAzureEventHub-pr.patch, > 0002-NIFI-2826-converted-from-epoch-to-iso8061-enqueue-in.patch, > 0003-NIFI-2826-Adding-enqueue-time-to-GetAzureEventHub-pr.patch, patch.1 > > > The current GetEventHubProcessor should support allowing an enqueue time like > eventhubs to read messages from the past. This would also support reading > messages missed during downtime. Today the behaviour is for the processor to > start at the enqueue of when it was started by the user missing any past > messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2826) GetEventHubProcessor Support Enqueue Time
[ https://issues.apache.org/jira/browse/NIFI-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15542568#comment-15542568 ] ASF subversion and git services commented on NIFI-2826: --- Commit 34e5a5321ab5e5d02e22932f115be982b3b5b304 in nifi's branch refs/heads/master from [~josephxsxn] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=34e5a53 ] NIFI-2826 Adding enqueue time to GetAzureEventHub processor Signed-off-by: jpercivall > GetEventHubProcessor Support Enqueue Time > - > > Key: NIFI-2826 > URL: https://issues.apache.org/jira/browse/NIFI-2826 > Project: Apache NiFi > Issue Type: Improvement > Environment: centos6, jdk1.8, nifi 1.0.0 >Reporter: Joseph Niemiec >Assignee: Joseph Niemiec >Priority: Minor > Fix For: 1.1.0 > > Attachments: > 0001-NIFI-2826-Adding-enqueue-time-to-GetAzureEventHub-pr.patch, > 0002-NIFI-2826-converted-from-epoch-to-iso8061-enqueue-in.patch, > 0003-NIFI-2826-Adding-enqueue-time-to-GetAzureEventHub-pr.patch, patch.1 > > > The current GetEventHubProcessor should support allowing an enqueue time like > eventhubs to read messages from the past. This would also support reading > messages missed during downtime. Today the behaviour is for the processor to > start at the enqueue of when it was started by the user missing any past > messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2826) GetEventHubProcessor Support Enqueue Time
[ https://issues.apache.org/jira/browse/NIFI-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530237#comment-15530237 ] Joseph Percivall commented on NIFI-2826: Ticket for better handling restarts: https://issues.apache.org/jira/browse/NIFI-2835 > GetEventHubProcessor Support Enqueue Time > - > > Key: NIFI-2826 > URL: https://issues.apache.org/jira/browse/NIFI-2826 > Project: Apache NiFi > Issue Type: Improvement > Environment: centos6, jdk1.8, nifi 1.0.0 >Reporter: Joseph Niemiec >Assignee: Joseph Niemiec >Priority: Minor > Fix For: 1.1.0 > > Attachments: > 0001-NIFI-2826-Adding-enqueue-time-to-GetAzureEventHub-pr.patch, patch.1 > > > The current GetEventHubProcessor should support allowing an enqueue time like > eventhubs to read messages from the past. This would also support reading > messages missed during downtime. Today the behaviour is for the processor to > start at the enqueue of when it was started by the user missing any past > messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2826) GetEventHubProcessor Support Enqueue Time
[ https://issues.apache.org/jira/browse/NIFI-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530209#comment-15530209 ] Joseph Percivall commented on NIFI-2826: Ticket for the configurable internal batching: https://issues.apache.org/jira/browse/NIFI-2834 > GetEventHubProcessor Support Enqueue Time > - > > Key: NIFI-2826 > URL: https://issues.apache.org/jira/browse/NIFI-2826 > Project: Apache NiFi > Issue Type: Improvement > Environment: centos6, jdk1.8, nifi 1.0.0 >Reporter: Joseph Niemiec >Assignee: Joseph Niemiec >Priority: Minor > Fix For: 1.1.0 > > Attachments: > 0001-NIFI-2826-Adding-enqueue-time-to-GetAzureEventHub-pr.patch, patch.1 > > > The current GetEventHubProcessor should support allowing an enqueue time like > eventhubs to read messages from the past. This would also support reading > messages missed during downtime. Today the behaviour is for the processor to > start at the enqueue of when it was started by the user missing any past > messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2826) GetEventHubProcessor Support Enqueue Time
[ https://issues.apache.org/jira/browse/NIFI-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530059#comment-15530059 ] Joseph Percivall commented on NIFI-2826: Hey [~josephxsxn] a couple thoughts: 1: The input type for enqueue time should probably be changed from epoch time to a standard date format. We try to cater to the non-technical users of NiFi when it comes to processor configuration and I fear most non-technical users would have no idea what epoch time is. Also it would just be annoying to convert to. 2: The default value should be removed and instead add a check to the "getReceiver" method to see if ENQUEUE_TIME is set. We try our best to not change any default behavior in non-major releases and there would be a difference between "when the property gets populated with the default value" and "when the receiver gets instantiated". So just a check on line 197 where if ENQUEUE_TIME is set use it and else use "Instant.now()". 3: Looking at this, the processor could use some improvements in regards to durable restart. Currently on every start/stop the partition receiver list is cleared. It would be nice to keep those until the configuration changes so that the processor could pick back up right where it left off so no data is lost. That's outside the scope of this ticket but just putting down some thoughts (I'll create another ticket). Also another ticket to be created is to make the "Max number of events received" configurable (line 224). Let me know your thoughts and we can iterate on the patch. Thanks! > GetEventHubProcessor Support Enqueue Time > - > > Key: NIFI-2826 > URL: https://issues.apache.org/jira/browse/NIFI-2826 > Project: Apache NiFi > Issue Type: Improvement > Environment: centos6, jdk1.8, nifi 1.0.0 >Reporter: Joseph Niemiec >Assignee: Joseph Niemiec >Priority: Minor > Fix For: 1.1.0 > > Attachments: > 0001-NIFI-2826-Adding-enqueue-time-to-GetAzureEventHub-pr.patch, patch.1 > > > The current GetEventHubProcessor should support allowing an enqueue time like > eventhubs to read messages from the past. This would also support reading > messages missed during downtime. Today the behaviour is for the processor to > start at the enqueue of when it was started by the user missing any past > messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2826) GetEventHubProcessor Support Enqueue Time
[
https://issues.apache.org/jira/browse/NIFI-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526324#comment-15526324
]
Joseph Percivall commented on NIFI-2826:
Hey [~josephxsxn], I just tried to "git am" your patch and it appears it
doesn't have any author or commit information, just the diff of the file. In
order to create a proper patch file (so that you'll get credit for the commit
in the NiFi repo!) you can follow these steps:
1: checkout master ("git checkout master") and pull down the latest ("git pull")
2: create a new branch to do the work on ("git checkout -B NIFI-2826")
3: now check the status of the repo using "git status". You'll should now be on
branch NIFI-2826, potentially with any changes you made.
4: If the only uncommitted changes are the ones you want to make the patch for
you can do "git commit -am 'NIFI-2826 Adding enqueue time to GetAzureEventHub
processor" (see below if you want to clear out your working repo)
5: you should now have one commit different than the master branch, so you can
do "git format-patch master". This will create a patch file called
NIFI-2826_Adding_enqueue_time_to_GetAzureEventHub_processor.patch. It will have
the diff of all the files as well as the author/commit information.
6: you can then attach that patch file here.
If you have items in your repo that are uncommitted that you don't care about
you can do "git reset HEAD --hard" (this will remove any changes to any files
that git is currently tracking). If there are files in the repo that git isn't
tracking that you want to remove you can do "git clean -i -d" (this will
interactively let you chose which files to remove). After you get to a clean
repo, you can do "git apply patch.1" in order have the diff you attached here
back and commit it.
This is a lot of information so please comment if anything was unclear. Also if
you want to learn more about git, I'd suggest checking out this D3 animation:
https://onlywei.github.io/explain-git-with-d3/
> GetEventHubProcessor Support Enqueue Time
> -
>
> Key: NIFI-2826
> URL: https://issues.apache.org/jira/browse/NIFI-2826
> Project: Apache NiFi
> Issue Type: Improvement
> Environment: centos6, jdk1.8, nifi 1.0.0
>Reporter: Joseph Niemiec
>Assignee: Joseph Niemiec
>Priority: Minor
> Fix For: 1.1.0
>
> Attachments: patch.1
>
>
> The current GetEventHubProcessor should support allowing an enqueue time like
> eventhubs to read messages from the past. This would also support reading
> messages missed during downtime. Today the behaviour is for the processor to
> start at the enqueue of when it was started by the user missing any past
> messages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (NIFI-2826) GetEventHubProcessor Support Enqueue Time
[ https://issues.apache.org/jira/browse/NIFI-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526230#comment-15526230 ] Joseph Percivall commented on NIFI-2826: Hello [~josephxsxn], thank you for opening this ticket and doing the work on the patch. I'll start a review and comment with any findings. > GetEventHubProcessor Support Enqueue Time > - > > Key: NIFI-2826 > URL: https://issues.apache.org/jira/browse/NIFI-2826 > Project: Apache NiFi > Issue Type: Improvement > Environment: centos6, jdk1.8, nifi 1.0.0 >Reporter: Joseph Niemiec >Assignee: Joseph Niemiec >Priority: Minor > Fix For: 1.1.0 > > Attachments: patch.1 > > > The current GetEventHubProcessor should support allowing an enqueue time like > eventhubs to read messages from the past. This would also support reading > messages missed during downtime. Today the behaviour is for the processor to > start at the enqueue of when it was started by the user missing any past > messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-2826) GetEventHubProcessor Support Enqueue Time
[ https://issues.apache.org/jira/browse/NIFI-2826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15524828#comment-15524828 ] Joseph Niemiec commented on NIFI-2826: -- I have an initial working version, just refining it and making tests before attaching a patch. > GetEventHubProcessor Support Enqueue Time > - > > Key: NIFI-2826 > URL: https://issues.apache.org/jira/browse/NIFI-2826 > Project: Apache NiFi > Issue Type: Bug > Environment: centos6, jdk1.8, nifi 1.0.0 >Reporter: Joseph Niemiec > > The current GetEventHubProcessor should support allowing an enqueue time like > eventhubs to read messages from the past. This would also support reading > messages missed during downtime. Today the behaviour is for the processor to > start at the enqueue of when it was started by the user missing any past > messages. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
