[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464423#comment-16464423
 ] 

ASF subversion and git services commented on NIFI-5130:
---

Commit ed30bb9b7872840b3b5348766c0e68b69617abd8 in nifi's branch 
refs/heads/master from [~michal_misiewicz]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=ed30bb9 ]

NIFI-5130 ExecuteInfluxDBQuery processor chunking support

This closes #2666

Signed-off-by: Mike Thomsen 


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464424#comment-16464424
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2666


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464417#comment-16464417
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2666#discussion_r186226336
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/ExecuteInfluxDBQuery.java
 ---
@@ -86,6 +93,18 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor INFLUX_DB_QUERY_CHUNK_SIZE = 
new PropertyDescriptor.Builder()
+.name("influxdb-query-chunk-size")
+.displayName("Results chunk size")
+.description("Chunking can be used to return results in a 
stream of smaller batches "
++ "(each has a partial results up to a chunk size) rather 
than as a single response. "
++ "Chunking queries can return an unlimited number of 
rows. Note: Chunking is enable when result chunk size is greater than 0")
+
.defaultValue(String.valueOf(DEFAULT_INFLUX_RESPONSE_CHUNK_SIZE))
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.createLongValidator(0, 
Integer.MAX_VALUE, true))
--- End diff --

@michalmisiewicz I'll just go ahead and do the fix and merge. Thanks for 
the patch.


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464365#comment-16464365
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2666#discussion_r186213122
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/ExecuteInfluxDBQuery.java
 ---
@@ -86,6 +93,18 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor INFLUX_DB_QUERY_CHUNK_SIZE = 
new PropertyDescriptor.Builder()
+.name("influxdb-query-chunk-size")
+.displayName("Results chunk size")
+.description("Chunking can be used to return results in a 
stream of smaller batches "
++ "(each has a partial results up to a chunk size) rather 
than as a single response. "
++ "Chunking queries can return an unlimited number of 
rows. Note: Chunking is enable when result chunk size is greater than 0")
+
.defaultValue(String.valueOf(DEFAULT_INFLUX_RESPONSE_CHUNK_SIZE))
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.createLongValidator(0, 
Integer.MAX_VALUE, true))
--- End diff --

Ok, go with the default @michalmisiewicz 


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464341#comment-16464341
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user timhallinflux commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2666#discussion_r186204325
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/ExecuteInfluxDBQuery.java
 ---
@@ -86,6 +93,18 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor INFLUX_DB_QUERY_CHUNK_SIZE = 
new PropertyDescriptor.Builder()
+.name("influxdb-query-chunk-size")
+.displayName("Results chunk size")
+.description("Chunking can be used to return results in a 
stream of smaller batches "
++ "(each has a partial results up to a chunk size) rather 
than as a single response. "
++ "Chunking queries can return an unlimited number of 
rows. Note: Chunking is enable when result chunk size is greater than 0")
+
.defaultValue(String.valueOf(DEFAULT_INFLUX_RESPONSE_CHUNK_SIZE))
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.createLongValidator(0, 
Integer.MAX_VALUE, true))
--- End diff --

Aligning with the default value seems very rational.


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464333#comment-16464333
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2666#discussion_r186203118
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/ExecuteInfluxDBQuery.java
 ---
@@ -86,6 +93,18 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor INFLUX_DB_QUERY_CHUNK_SIZE = 
new PropertyDescriptor.Builder()
+.name("influxdb-query-chunk-size")
+.displayName("Results chunk size")
+.description("Chunking can be used to return results in a 
stream of smaller batches "
++ "(each has a partial results up to a chunk size) rather 
than as a single response. "
++ "Chunking queries can return an unlimited number of 
rows. Note: Chunking is enable when result chunk size is greater than 0")
+
.defaultValue(String.valueOf(DEFAULT_INFLUX_RESPONSE_CHUNK_SIZE))
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.createLongValidator(0, 
Integer.MAX_VALUE, true))
--- End diff --

@timhallinflux Can you jump in here and give us your opinion?


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464331#comment-16464331
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2666#discussion_r186202668
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/ExecuteInfluxDBQuery.java
 ---
@@ -86,6 +93,18 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor INFLUX_DB_QUERY_CHUNK_SIZE = 
new PropertyDescriptor.Builder()
+.name("influxdb-query-chunk-size")
+.displayName("Results chunk size")
+.description("Chunking can be used to return results in a 
stream of smaller batches "
++ "(each has a partial results up to a chunk size) rather 
than as a single response. "
++ "Chunking queries can return an unlimited number of 
rows. Note: Chunking is enable when result chunk size is greater than 0")
+
.defaultValue(String.valueOf(DEFAULT_INFLUX_RESPONSE_CHUNK_SIZE))
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.createLongValidator(0, 
Integer.MAX_VALUE, true))
--- End diff --

If the default limit is 10k for the server, I think that should be taken as 
a warning sign that you really need to know what you're doing if you mess 
around beyond that. So let's just go with 10k for now.


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464255#comment-16464255
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user michalmisiewicz commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@MikeThomsen I've updated the code.  
Please let me know if I should change maximum chunk size.


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464240#comment-16464240
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user michalmisiewicz commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2666#discussion_r186177020
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/ExecuteInfluxDBQuery.java
 ---
@@ -86,6 +93,18 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor INFLUX_DB_QUERY_CHUNK_SIZE = 
new PropertyDescriptor.Builder()
+.name("influxdb-query-chunk-size")
+.displayName("Results chunk size")
+.description("Chunking can be used to return results in a 
stream of smaller batches "
++ "(each has a partial results up to a chunk size) rather 
than as a single response. "
++ "Chunking queries can return an unlimited number of 
rows. Note: Chunking is enable when result chunk size is greater than 0")
+
.defaultValue(String.valueOf(DEFAULT_INFLUX_RESPONSE_CHUNK_SIZE))
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.createLongValidator(0, 
Integer.MAX_VALUE, true))
--- End diff --

According to InfluxDB 
[documentation](https://docs.influxdata.com/influxdb/v1.5/guides/querying_data/).
  
> The maximum row limit only applies to non-chunked queries. Chunked 
queries can return an unlimited number of points.  

How big limit do you suggest ?


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464057#comment-16464057
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2666#discussion_r186126226
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/ExecuteInfluxDBQuery.java
 ---
@@ -189,9 +209,10 @@ public void onTrigger(final ProcessContext context, 
final ProcessSession session
 
 try {
 long startTimeMillis = System.currentTimeMillis();
-QueryResult result = executeQuery(context, database, query, 
queryResultTimeunit);
+int chunkSize = 
Integer.valueOf(context.getProperty(INFLUX_DB_QUERY_CHUNK_SIZE).evaluateAttributeExpressions(outgoingFlowFile).getValue());
--- End diff --

You can use `asInteger()` or `asLong()` here instead of using 
`Integer.valueOf` and `getValue()`


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464058#comment-16464058
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2666#discussion_r186124698
  
--- Diff: 
nifi-nar-bundles/nifi-influxdb-bundle/nifi-influxdb-processors/src/main/java/org/apache/nifi/processors/influxdb/ExecuteInfluxDBQuery.java
 ---
@@ -86,6 +93,18 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor INFLUX_DB_QUERY_CHUNK_SIZE = 
new PropertyDescriptor.Builder()
+.name("influxdb-query-chunk-size")
+.displayName("Results chunk size")
+.description("Chunking can be used to return results in a 
stream of smaller batches "
++ "(each has a partial results up to a chunk size) rather 
than as a single response. "
++ "Chunking queries can return an unlimited number of 
rows. Note: Chunking is enable when result chunk size is greater than 0")
+
.defaultValue(String.valueOf(DEFAULT_INFLUX_RESPONSE_CHUNK_SIZE))
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
+.addValidator(StandardValidators.createLongValidator(0, 
Integer.MAX_VALUE, true))
--- End diff --

If the default query result limit is 10k, what is the likelihood that there 
is a good reason for this to be above 10k? Even if we go beyond that for 
chunks, Integer.MAX_VALUE is way past a sane limit.


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462861#comment-16462861
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user michalmisiewicz commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@MikeThomsen it supposed to fetch the entire result set of that select 
statement each run


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462836#comment-16462836
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@michaelandrepearce so I pushed about 30k events into InfluxDB using some 
artifacts from @mans2singh's PRs and ran `select * from /*/` on the database. 
Let it run twice and the result sets were different in size. Is each iteration 
of the processor supposed to be a new chunk or is it supposed to chunk the 
entire result set of that select statement each run?


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462818#comment-16462818
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@mans2singh do you have free time to help review this?


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462541#comment-16462541
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user michalmisiewicz commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@MikeThomsen yes. Chunking is the only way to fetch unlimited number of 
rows in a single query from InfluxDB that has set `max-row-limit` property.

> Many production InfluxDB installation has limited number of rows returned 
in a single query (by default 10k). Chunking query can return an unlimited 
number of rows.


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462415#comment-16462415
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2666
  
@michalmisiewicz so if I understand the Jira ticket correctly, the purpose 
of this ticket is to make it possible to stream out everything that matches the 
query, not just one large result payload. Is that right?


> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5130) ExecuteInfluxDBQuery processor chunking support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457946#comment-16457946
 ] 

ASF GitHub Bot commented on NIFI-5130:
--

GitHub user michalmisiewicz opened a pull request:

https://github.com/apache/nifi/pull/2666

NIFI-5130 ExecuteInfluxDBQuery processor chunking support

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [x] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/michalmisiewicz/nifi 
NIFI-5130-ExecuteInfluxDBQuery-chunking-support

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2666.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2666






> ExecuteInfluxDBQuery processor chunking support
> ---
>
> Key: NIFI-5130
> URL: https://issues.apache.org/jira/browse/NIFI-5130
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Michał Misiewicz
>Priority: Minor
>
> Many production InfluxDB installation has limited number of rows returned in 
> a single query (by default 10k). In case of huge collections, 10k rows can 
> correspond to less than 1 minute of events, which make usage of 
> ExecuteInfluxDBQuery processor inconvenient. I suggest adding support for 
> chunking queries. Chunking can be used to return results in a stream of 
> smaller batches (each has a partial results up to a chunk size) rather than 
> as a single response. Chunking query can return an unlimited number of rows.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)