mikemccand commented on PR #12750:
URL: https://github.com/apache/lucene/pull/12750#issuecomment-1820713053
This looks like the root cause?:
```
java.lang.AssertionError: inconsistent endOffset 1 pos=0 posLen=1
token=/a/b expected:<2> but was:<4>
```
Indeed I think the issue is a problem with `PathHierarchyTokenizer`: it
produces tokens all on top of one another (instead of in sequence at
incrementing positions) yet the tokens claim different start/end offsets, and
`BaseTokenStreamTestCase` detects that as a corrupt token graph. I think this
tokenizer should be setting the `PositionLengthAttribute` as well, to indicate
that each token reaches to a further position ... this should make
`BaseTokenStreamTestCase` happy.
See [this blog
post](https://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html)
for more details about how `TokenStream`s are actually graphs in Lucene.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
