Vincent Lefevre wrote:
[...] Note that this option can also be passed to pcre_exec() and pcre_dfa_exec(), to suppress the validity checking of subject strings only. If the same string is being matched many times, the option can be safely set for the second and subsequent matchings to improve performance.The last sentence would imply that the UTF8 checking is done on the whole input buffer before matching is done.
That's pretty subtle, and perhaps too subtle. A plausible interpretation of the phrase "same string is being matched" is that libpcre checks only the matched string, and that bytes after the match (which did not need to be examined to do the match) are not checked. Can you confirm with the libpcre authors that this plausible interpretation is incorrect, i.e., that the entire input string is checked, even the unmatched part? If that's what is intended, the documentation should state so clearly, so at least there's a documentation bug there.
If there are many invalid UTF8 bytes, this would be slow, IMHO
That's OK. We don't need grep -P to be fast on invalid input.
But is the copy of the buffer really needed? Couldn't the invalid UTF8 sequences just be replaced by null bytes?
I'd rather not, because that changes the semantics of matching. The null byte is valid input data that might get matched.
in case of invalid UTF8 bytes, in some (many?) cases, the cause is a binary file (possibly with some text in it), where lines can be very long. So, wouldn't it mean that it can take significantly more memory?
Sure. But that's the same for -P as it is for plain grep.
