[jira] Updated: (LUCENE-1391) Token type and flags values get lost when using ShingleMatrixFilter
[ https://issues.apache.org/jira/browse/LUCENE-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1391: -- Affects Version/s: 2.9 3.0 This issue is still valid, ShingleMatrixFilter still sets its class name as type attribute for all tokens and resets flags to 0. Furthermore, ShingleMatrixFilter does not respect custom/new attributes at all (like KeywordAttribute). > Token type and flags values get lost when using ShingleMatrixFilter > --- > > Key: LUCENE-1391 > URL: https://issues.apache.org/jira/browse/LUCENE-1391 > Project: Lucene - Java > Issue Type: Bug > Components: contrib/analyzers >Affects Versions: 2.4, 2.9, 3.0 >Reporter: Wouter Heijke >Assignee: Karl Wettin > Fix For: 3.1, 4.0 > > > While using the new ShingleMatrixFilter I noticed that a token's type and > flags get lost while using this filter. ShingleFilter does respect these > values like the other filters I know. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1391) Token type and flags values get lost when using ShingleMatrixFilter
[ https://issues.apache.org/jira/browse/LUCENE-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1391: -- Fix Version/s: 4.0 3.1 > Token type and flags values get lost when using ShingleMatrixFilter > --- > > Key: LUCENE-1391 > URL: https://issues.apache.org/jira/browse/LUCENE-1391 > Project: Lucene - Java > Issue Type: Bug > Components: contrib/analyzers >Affects Versions: 2.4, 2.9, 3.0 >Reporter: Wouter Heijke >Assignee: Karl Wettin > Fix For: 3.1, 4.0 > > > While using the new ShingleMatrixFilter I noticed that a token's type and > flags get lost while using this filter. ShingleFilter does respect these > values like the other filters I know. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-1391) Token type and flags values get lost when using ShingleMatrixFilter
[ https://issues.apache.org/jira/browse/LUCENE-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uwe Schindler updated LUCENE-1391: -- Attachment: LUCENE-1391.patch Here is just a funny rewrite of this filter, not 100% working (but tests pass). Problems occur, when you define your own matrix, but the AttributeSources representing the Tokens are not compatible with copyTo() on the actual TokenStream (e.g. use different AttributeFactory, have additional attributes,...). Also the filter was not yet optimized. Currently it always adds all 6 basic attributes. To get around the TokenType problem, we can add the setter method to explicitely set the type for shingles (currently its the class name). The FlagsAttribute is itsself used by the Filter to manage internal Token state. It should be replaced by a filter-internal ShingleMatrixStateAttribute containing an enum. > Token type and flags values get lost when using ShingleMatrixFilter > --- > > Key: LUCENE-1391 > URL: https://issues.apache.org/jira/browse/LUCENE-1391 > Project: Lucene - Java > Issue Type: Bug > Components: contrib/analyzers >Affects Versions: 2.4, 2.9, 3.0 >Reporter: Wouter Heijke >Assignee: Uwe Schindler > Fix For: 3.1, 4.0 > > Attachments: LUCENE-1391.patch > > > While using the new ShingleMatrixFilter I noticed that a token's type and > flags get lost while using this filter. ShingleFilter does respect these > values like the other filters I know. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
