[jira] [Commented] (NIFI-11985) Implement a processor to consume documents from Elasticsearch indices
[
https://issues.apache.org/jira/browse/NIFI-11985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763452#comment-17763452
]
Chris Sampson commented on NIFI-11985:
--
[^NIFI-11985_Flow.json] provides a Flow Definition exhibiting the
{{ConsumeElasticsearch}} processor (along with all of the other Elasticsearch
processors and the changes added for NIFI-11016)
> Implement a processor to consume documents from Elasticsearch indices
> -
>
> Key: NIFI-11985
> URL: https://issues.apache.org/jira/browse/NIFI-11985
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
> Attachments: NIFI-11985_Flow.json
>
>
> It is possible to use Elasticsearch to store series data, i.e. data is
> continually added to an Elasticsearch index over time, with a {{date}} or a
> 1-up numeric {{long}} field.
> This is more likely with the advent of [Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html]
> or the recent [Time Series Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html],
> both of which use a {{@timestamp}} field to indicate when a document was
> added to the stream.
> There are use cases where NiFi users may want to consume new data from the
> Elasticsearch index/data stream after it's arrived, then pass it to another
> service.
> NiFi would need to:
> * know which field to use as the "series field" (e.g. {{@timestamp}})
> * track the last read "series field" value via State so that the same
> documents are not retrieved from Elasticsearch multiple times
> * allow for the optional specification of the "last read" field value, e.g.
> if a user wants to offset the start of the documents to be read (this value
> should only be used if a value doesn't also exist within the processor's
> State)
> * allow for the fact that the "last read" vlaue will be blank when the
> processor is first run (and the value is not otherwise specified), meaning we
> want to retrieve all existing data
> * allow for users to specify an optional Query Filter to apply to the search
> within Elasticsearch when finding documents to retrieve
> Possible implementations should consider using the {{SearchElasticsearch}}
> processor as a basis, which already uses State tracking between processor
> executions and allows for the retrieval of Elasticsearch documents in a
> paginated manner (thus avoiding pulling too much data in a single request).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-11985) Implement a processor to consume documents from Elasticsearch indices
[
https://issues.apache.org/jira/browse/NIFI-11985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Sampson updated NIFI-11985:
-
Attachment: NIFI-11985_Flow.json
> Implement a processor to consume documents from Elasticsearch indices
> -
>
> Key: NIFI-11985
> URL: https://issues.apache.org/jira/browse/NIFI-11985
> Project: Apache NiFi
> Issue Type: New Feature
>Reporter: Chris Sampson
>Assignee: Chris Sampson
>Priority: Minor
> Fix For: 1.latest, 2.latest
>
> Attachments: NIFI-11985_Flow.json
>
>
> It is possible to use Elasticsearch to store series data, i.e. data is
> continually added to an Elasticsearch index over time, with a {{date}} or a
> 1-up numeric {{long}} field.
> This is more likely with the advent of [Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html]
> or the recent [Time Series Data
> Streams|https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html],
> both of which use a {{@timestamp}} field to indicate when a document was
> added to the stream.
> There are use cases where NiFi users may want to consume new data from the
> Elasticsearch index/data stream after it's arrived, then pass it to another
> service.
> NiFi would need to:
> * know which field to use as the "series field" (e.g. {{@timestamp}})
> * track the last read "series field" value via State so that the same
> documents are not retrieved from Elasticsearch multiple times
> * allow for the optional specification of the "last read" field value, e.g.
> if a user wants to offset the start of the documents to be read (this value
> should only be used if a value doesn't also exist within the processor's
> State)
> * allow for the fact that the "last read" vlaue will be blank when the
> processor is first run (and the value is not otherwise specified), meaning we
> want to retrieve all existing data
> * allow for users to specify an optional Query Filter to apply to the search
> within Elasticsearch when finding documents to retrieve
> Possible implementations should consider using the {{SearchElasticsearch}}
> processor as a basis, which already uses State tracking between processor
> executions and allows for the retrieval of Elasticsearch documents in a
> paginated manner (thus avoiding pulling too much data in a single request).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (NIFI-12028) Add s3.region attribute to ListS3 generated flow files
[ https://issues.apache.org/jira/browse/NIFI-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763446#comment-17763446 ] ASF subversion and git services commented on NIFI-12028: Commit 5be76cdddef928744ee2c41f90955d4c372961f6 in nifi's branch refs/heads/support/nifi-1.x from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5be76cddde ] NIFI-12028 - Add s3.region attribute to ListS3 generated flow files This closes #7672 Signed-off-by: Mike Thomsen > Add s3.region attribute to ListS3 generated flow files > -- > > Key: NIFI-12028 > URL: https://issues.apache.org/jira/browse/NIFI-12028 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add s3.region attribute to ListS3 generated flow files so that it can be > referenced in FetchS3 processor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12028) Add s3.region attribute to ListS3 generated flow files
[ https://issues.apache.org/jira/browse/NIFI-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen updated NIFI-12028: Resolution: Fixed Status: Resolved (was: Patch Available) > Add s3.region attribute to ListS3 generated flow files > -- > > Key: NIFI-12028 > URL: https://issues.apache.org/jira/browse/NIFI-12028 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add s3.region attribute to ListS3 generated flow files so that it can be > referenced in FetchS3 processor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12028) Add s3.region attribute to ListS3 generated flow files
[ https://issues.apache.org/jira/browse/NIFI-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763445#comment-17763445 ] ASF subversion and git services commented on NIFI-12028: Commit 47f4c8ce2e0b1ec5fa176cb44a94e64dce354fa0 in nifi's branch refs/heads/main from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=47f4c8ce2e ] NIFI-12028 - Add s3.region attribute to ListS3 generated flow files This closes #7672 Signed-off-by: Mike Thomsen > Add s3.region attribute to ListS3 generated flow files > -- > > Key: NIFI-12028 > URL: https://issues.apache.org/jira/browse/NIFI-12028 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > Fix For: 1.latest, 2.latest > > Time Spent: 20m > Remaining Estimate: 0h > > Add s3.region attribute to ListS3 generated flow files so that it can be > referenced in FetchS3 processor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [nifi] asfgit closed pull request #7672: NIFI-12028 - Add s3.region attribute to ListS3 generated flow files
asfgit closed pull request #7672: NIFI-12028 - Add s3.region attribute to ListS3 generated flow files URL: https://github.com/apache/nifi/pull/7672 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (MINIFICPP-1755) Use std::span instead of gsl::span
[ https://issues.apache.org/jira/browse/MINIFICPP-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-1755. - Resolution: Fixed > Use std::span instead of gsl::span > -- > > Key: MINIFICPP-1755 > URL: https://issues.apache.org/jira/browse/MINIFICPP-1755 > Project: Apache NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Marton Szasz >Assignee: Adam Debreceni >Priority: Major > Labels: MiNiFi-CPP-Hygiene > Time Spent: 2h 10m > Remaining Estimate: 0h > > The move is not trivial, because there is no .as_span member of std::span. > There is `as_bytes()`, but we do (rarely) use `as_span` with types other > than `std::byte`, too. We could write a free-standing utility function for > the non-byte case of `as_span` (and maybe some of those cases could use > `std::byte` instead). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (MINIFICPP-2211) update windows sqliteodbc hash to v0.9999
[ https://issues.apache.org/jira/browse/MINIFICPP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz updated MINIFICPP-2211: Fix Version/s: 0.16.0 > update windows sqliteodbc hash to v0. > - > > Key: MINIFICPP-2211 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2211 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marton Szasz >Assignee: Marton Szasz >Priority: Major > Fix For: 0.16.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (MINIFICPP-2211) update windows sqliteodbc hash to v0.9999
[ https://issues.apache.org/jira/browse/MINIFICPP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz resolved MINIFICPP-2211. - Resolution: Fixed > update windows sqliteodbc hash to v0. > - > > Key: MINIFICPP-2211 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2211 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marton Szasz >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (MINIFICPP-2211) update windows sqliteodbc hash to v0.9999
[ https://issues.apache.org/jira/browse/MINIFICPP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Szasz reassigned MINIFICPP-2211: --- Assignee: Marton Szasz > update windows sqliteodbc hash to v0. > - > > Key: MINIFICPP-2211 > URL: https://issues.apache.org/jira/browse/MINIFICPP-2211 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Marton Szasz >Assignee: Marton Szasz >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12034) Update commons-compress to 1.24.0
Mike R created NIFI-12034: - Summary: Update commons-compress to 1.24.0 Key: NIFI-12034 URL: https://issues.apache.org/jira/browse/NIFI-12034 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.23.2 Reporter: Mike R Update commons-compress to 1.24.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
