[ https://issues.apache.org/jira/browse/NIFI-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195076#comment-16195076 ]
ASF GitHub Bot commented on NIFI-3248: -------------------------------------- GitHub user JohannesDaniel opened a pull request: https://github.com/apache/nifi/pull/2199 NIFI-3248: Improvement of GetSolr Processor Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JohannesDaniel/nifi NIFI-3248 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/2199.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2199 ---- commit 8a5f7e54edc5640655edd19f15d22fada6ca9900 Author: JohannesDaniel <johannesdaniel.pe...@gmx.de> Date: 2017-10-05T20:57:53Z NIFI-3248: Improvement of GetSolr Processor ---- > GetSolr can miss recently updated documents > ------------------------------------------- > > Key: NIFI-3248 > URL: https://issues.apache.org/jira/browse/NIFI-3248 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Affects Versions: 1.0.0, 0.5.0, 0.6.0, 0.5.1, 0.7.0, 0.6.1, 1.1.0, 0.7.1, > 1.0.1 > Reporter: Koji Kawamura > Assignee: Johannes Peter > Attachments: nifi-flow.png, query-result-with-curly-bracket.png, > query-result-with-square-bracket.png > > > GetSolr holds the last query timestamp so that it only fetches documents > those have been added or updated since the last query. > However, GetSolr misses some of those updated documents, and once the > documents date field value becomes older than last query timestamp, the > document won't be able to be queried by GetSolr any more. > This JIRA is for tracking the process of investigating this behavior, and > discussion on them. > Here are things that can be a cause of this behavior: > |#|Short description|Should we address it?| > |1|Timestamp range filter, curly or square bracket?|No| > |2|Timezone difference between update and query|Additional docs might be > helpful| > |3|Lag comes from NearRealTIme nature of Solr|Should be documented at least, > add 'commit lag-time'?| > h2. 1. Timestamp range filter, curly or square bracket? > At the first glance, using curly and square bracket in mix looked strange > ([source > code|https://github.com/apache/nifi/blob/support/nifi-0.5.x/nifi-nar-bundles/nifi-solr-bundle/nifi-solr-processors/src/main/java/org/apache/nifi/processors/solr/GetSolr.java#L202]). > But these difference has a meaning. > The square bracket on the range query is inclusive and the curly bracket is > exclusive. If we use inclusive on both sides and a document has a time stamp > exactly on the boundary then it could be returned in two consecutive > executions, and we only want it in one. > This is intentional, and it should be as it is. > h2. 2. Timezone difference between update and query > Solr treats date fields as [UTC > representation|https://cwiki.apache.org/confluence/display/solr/Working+with+Dates|]. > If date field String value of an updated document represents time without > timezone, and NiFi is running on an environment using timezone other than > UTC, GetSolr can't perform date range query as users expect. > Let's say NiFi is running with JST(UTC+9). A process added a document to Solr > at 15:00 JST. But the date field doesn't have timezone. So, Solr indexed it > as 15:00 UTC. Then GetSolr performs range query at 15:10 JST, targeting any > documents updated from 15:00 to 15:10 JST. GetSolr formatted dates using UTC, > i.e. 6:00 to 6:10 UTC. The updated document won't be matched with the date > range filter. > To avoid this, updated documents must have proper timezone in date field > string representation. > If one uses NiFi expression language to set current timestamp to that date > field, following NiFi expression can be used: > {code} > ${now():format("yyyy-MM-dd'T'HH:mm:ss.SSSZ")} > {code} > It will produce a result like: > {code} > 2016-12-27T15:30:04.895+0900 > {code} > Then it will be indexed in Solr with UTC and will be queried by GetSolr as > expected. > h2. 3. Lag comes from NearRealTIme nature of Solr > Solr provides Near Real Time search capability, that means, the recently > updated documents can be queried in Near Real Time, but it's not real time. > This latency can be controlled by either on client side which requests the > update operation by specifying "commitWithin" parameter, or on the Solr > server side, "autoCommit" and "autoSoftCommit" in > [solrconfig.xml|https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig#UpdateHandlersinSolrConfig-Commits]. > Since commit and updating index can be costly, it's recommended to set this > interval long enough up to the maximum tolerable latency. > However, this can be problematic with GetSolr. For instance, as shown in the > simple NiFi flow below, GetSolr can miss updated documents: > {code} > t1: GetSolr queried > t2: GenerateFlowFile set date = t2 > t3: PutSolrContentStream stored new doc > t4: GetSolr queried again, from t1 to t4, but the new doc hasn't been indexed > t5: Solr completed index > t6: GetSolr queried again, from t4 to t6, the doc didn't match query > {code} > This behavior should be at least documented. > Plus, it would be helpful to add a new configuration property to GetSolr, to > specify commit lag-time so that GetSolr aims older timestamp range to query > documents. > {code} > // with commit lag-time > t1: GetSolr queried > t2: GenerateFlowFile set date = t2 > t3: PutSolrContentStream stored new doc > t4: GetSolr queried again, from (t1 - lag) to (t4 - lag), but the new doc > hasn't been indexed > t5: Solr completed index > t6: GetSolr queried again, from (t4 - lag) to (t6 - lag), the doc can match > query > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)