The two largest problems in the re module are splitting on zero-width
patterns and complete and correct support of the Unicode standard. These
problems are solved in regex. regex has many other features, but they
are less important.
I want to tell the problem of splitting on zero-width patterns. It
already was discussed on Python-Dev 13 years ago [3] and maybe later.
See also issues: [4], [5], [6], [7], [8].
In short it doesn't work. Splitting on the pattern r'\b' doesn't split
the text at boundaries of words, and splitting on the pattern
r'\s+|(?<=-)' will split the text on whitespaces, but will not split
words with hypens as expected.
In Python 3.4 and earlier:
>>> re.split(r'\b', 'Self-Defence Class')
['Self-Defence Class']
>>> re.split(r'\s+|(?<=-)', 'Self-Defence Class')
['Self-Defence', 'Class']
>>> re.split(r'\s*', 'Self-Defence Class')
['Self-Defence', 'Class']
Note that splitting on r'\s*' (0 or more whitespaces) actually split on
r'\s+' (1 or more whitespaces). Splitting on patterns that only can
match the empty string (like r'\b' or r'(?<=-)') never worked, while
splitting
Starting since Python 3.5 splitting on a pattern that only can match the
empty string raises a ValueError (this never worked), and splitting a
pattern that can match the empty string but not only emits a
FutureWarning. This taken developers a time for replacing their patterns
r'\s*' to r'\s+' as they should be.
Now I have created a final patch [9] that makes re.split() splitting on
zero-width patterns.
>>> re.split(r'\b', 'Self-Defence Class')
['', 'Self', '-', 'Defence', ' ', 'Class', '']
>>> re.split(r'\s+|(?<=-)', 'Self-Defence Class')
['Self-', 'Defence', 'Class']
>>> re.split(r'\s*', 'Self-Defence Class')
['', 'S', 'e', 'l', 'f', '-', 'D', 'e', 'f', 'e', 'n', 'c', 'e', 'C',
'l', 'a', 's', 's', '']
The latter case the result is differ too much from the previous result,
and this likely not what the author wanted to get. But users had two
Python releases for fixing their code. FutureWarning is not silent by
default.
Because these patterns produced errors or warnings in the recent two
releases, we don't need an additional parameter for compatibility.
But the problem was not just with re.split(). Other functions also
worked not good with patterns that can match the empty string.
>>> re.findall(r'^|\w+', 'Self-Defence Class')
['', 'elf', 'Defence', 'Class']
>>> list(re.finditer(r'^|\w+', 'Self-Defence Class'))
[<re.Match object; span=(0, 0), match=''>, <re.Match object; span=(1,
4), match='elf'>, <re.Match object; span=(5, 12), match='Defence'>,
<re.Match object; span=(13, 18), match='Class'>]
>>> re.sub(r'(^|\w+)', r'<\1>', 'Self-Defence Class')
'<>S<elf>-<Defence> <Class>'
After matching the empty string the following character will be skipped
and will be not included in the next match. My patch fixes these
functions too.
>>> re.findall(r'^|\w+', 'Self-Defence Class')
['', 'Self', 'Defence', 'Class']
>>> list(re.finditer(r'^|\w+', 'Self-Defence Class'))
[<re.Match object; span=(0, 0), match=''>, <re.Match object; span=(0,
4), match='Self'>, <re.Match object; span=(5, 12), match='Defence'>,
<re.Match object; span=(13, 18), match='Class'>]
>>> re.sub(r'(^|\w+)', r'<\1>', 'Self-Defence Class')
'<><Self>-<Defence> <Class>'
I think this change don't need preliminary warnings, because it change
the behavior of more rarely used patterns. No re tests have been broken.
I was needed to add new tests for detecting the behavior change.
But there is one spoonful of tar in a barrel of honey. I didn't expect
this, but this change have broken a pattern used with re.sub() in the
doctest module: r'(?m)^\s*?$'. This was fixed by replacing it with
r'(?m)^[^\S\n]+?$'). I hope that such cases are pretty rare and think
this is an avoidable breakage.
The new behavior of re.split() matches the behavior of regex.split()
with the VERSION1 flag, the new behavior of re.findall() and
re.finditer() matches the behavior of corresponding functions in the
regex module (independently from the version flag). But the new behavior
of re.sub() doesn't match exactly the behavior of regex.sub() with any
version flag. It differs from the old behavior as you can see in the
example above, but is closer to it that to regex.sub() with VERSION1.
This allowed to avoid braking existing tests for re.sub().
>>> regex.sub(r'(\W+|(?<=-))', r':', 'Self-Defence Class')
'Self:Defence:Class'
>>> regex.sub(r'(?V1)(\W+|(?<=-))', r':', 'Self-Defence Class')
'Self::Defence:Class'
>>> re.sub(r'(\W+|(?<=-))', r':', 'Self-Defence Class')
'Self:Defence:Class'
As re.split() it never matches the empty string adjacent to the previous
match. re.findall() and re.finditer() only don't match the empty string
adjacent to the previous empty string match. In the regex module
regex.sub() is mutually consistent with regex.findall() and
regex.finditer() (with the VERSION1 flag), but regex.split() is not
consistent with them. In the re module re.split() and re.sub() will be
mutually consistent, as well as re.findall() and re.finditer(). This is
more backward compatible. And I don't know reasons for preferring the
behavior of re.findall() and re.finditer() over the behavior of
re.split() in this corner case.
