I have a Solr 3.6 deployment I inherited. The schema.xml specifies the use of StandardTokenizerFactory like so ...
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100"> ... <tokenizer class="solr.StandardTokenizerFactory"/> ... According to this reference guide ( https://home.apache.org/~ctargett/RefGuidePOC/jekyll/Tokenizers.html) ... the StandardTokenizer will treat punctuation as a delimiters. However, here is my content that gets indexed: "IOM-1:BA9ATS0FAB,\"Company Name Module\",8.1.0.16.0.2,B-A,000006KB09029932,PASS,,0,0,0,Y:0,0,0,0,0:BA9AUT0FAB,\"Company CM Rear Module\",B-6,000009XP12133407," This piece `B-A,000006KB09029932` gets tokenized into two words ... `|B-A|` and `|000006KB09029932|`. But this piece `B-6,000009XP12133407` gets tokenized into one word ... `|B-6,000009XP12133407|`. What I've observed is the comma is not considered a delimiter when it is proceeded by a digit ... almost like it considers "6,000" to be currency or something? QUESTION: Is this a bug in StandardTokenizer, or do I misunderstand how commas are used as delimiters? Rob