[
https://issues.apache.org/jira/browse/NIFI-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15950834#comment-15950834
]
ASF GitHub Bot commented on NIFI-3268:
--------------------------------------
Github user ijokarumawak commented on the issue:
https://github.com/apache/nifi/pull/1376
@qfdk Thank you very much for recording and sharing the video. I can see
how it improves execution time of generated SQL. This enhancement is
significant for the first execution of GenerateTableFetch processor if there
are many records to fetch. The performance difference is huge!
For the 2nd time or later, GenerateTableFetch uses 'where' clause using
'Max value columns' so it will not be a problem after 1st run if the max value
columns are properly indexed. But if there are so many records inserted or
updated, between GenerateTableFetch runs, it will have the same issue if we use
`limit offset`.
I understand how it should work now. Please let me know when you finish
updating PR. I will review it as soon as possible. Thanks!
> Add AUTO_INCREMENT column in GenerateTableFetch to benefit index
> ----------------------------------------------------------------
>
> Key: NIFI-3268
> URL: https://issues.apache.org/jira/browse/NIFI-3268
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Affects Versions: 1.1.1
> Environment: - ubuntu 16.04
> - java version "1.8.0_111"
> - Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> - Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
> Reporter: qfdk
> Labels: easyfix
>
> I added AUTO_INCREMENT column in GenerateTableFetch to benefit index column
> By default this processor uses OFFSET, i have problems with large data.
> somme column has index so we could use index to speed up query time.
> I posted question here :
> https://community.hortonworks.com/questions/72586/how-can-i-use-an-array-with-putelasticsearch.html
> If you indexed un column (id), you could use this sql
> ```
> select xxx
> From xxxxx
> where 200000=>id
> order by id
> limit 200000
> ```
> “OFFSET is bad for skipping previous rows.” [Online]. Available:
> http://Use-The-Index-Luke.com/sql/partial-results/fetch-next-page. [Accessed:
> 27-Dec-2016].
> Thank you in advance
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
