[
https://issues.apache.org/jira/browse/CAMEL-25015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-25015 started by Claus Ibsen.
-------------------------------------------
> Tokenize language: a token with regex=false is still compiled as a regular
> expression, so tokenize("|") splits every character and tokenize("*") fails
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25015
> URL: https://issues.apache.org/jira/browse/CAMEL-25015
> Project: Camel
> Issue Type: Bug
> Components: camel-core
> Reporter: shashank
> Assignee: Claus Ibsen
> Priority: Minor
>
> The tokenize language has a {{regex}} option ("If the token is a regular
> expression pattern", default {{false}}), but a token is always compiled as a
> regular expression. {{TokenizeLanguage.createExpression}} picks
> {{tokenizeExpression}} or {{regexTokenizeExpression}}, and both end in
> {{ExchangeHelper.getScanner(exchange, value, token)}}, which passes the token
> unchanged to {{org.apache.camel.util.Scanner}}, which calls
> {{Pattern.compile}}. With {{regex=false}} the only difference is that a
> Simple function in the token is evaluated.
> So common delimiters that are regex metacharacters give wrong splits, or
> fail, with no hint that the token is a regex:
> {noformat}
> split().tokenize("|") of "a|b|c" -> [a, |, b, |, c] (one message per
> character)
> split().tokenize(".") of "a.b.c" -> [, , , ]
> split().tokenize("$") of "a$b$c" -> [a$b$c]
> split().tokenize("*") of "a*b*c" -> PatternSyntaxException: Dangling meta
> character '*'
> split().tokenize("|", false, 2) of "a|b|c|d" -> [a||, b||, c||, d]
> tokenizePair("(", ")") of "(a)(b)" -> PatternSyntaxException: Unmatched
> closing ')'
> tokenizePair("[rec]", "END.") of "[rec]1 ENDS HERE END.[rec]2END." -> [1 ]
> (expected [1 ENDS HERE , 2])
> controls: tokenize(newline), tokenize(";"), tokenize(escaped pipe,
> regex=true) -> [a, b, c]
> {noformat}
> Pipe-delimited files are the most visible case. This has been the behaviour
> since at least 2.25.
> *The fix has a compatibility question, so I would like to ask before sending
> a PR.* The obvious fix is to quote the token ({{Pattern.quote}}) when
> {{regex=false}} (and the end token of {{tokenizePair}}, and to use the
> unquoted token as the group delimiter). Tokens without regex metacharacters
> would behave exactly as before. But routes that rely on a regex token without
> {{regex=true}} would change. For example, in the XML or YAML DSL a token of
> backslash-r backslash-n is four characters, and it only works today because
> it is compiled as a regex ({{TokenizerExpressionReifier}} only converts a
> leading backslash-n); a whitespace pattern such as backslash-s-plus works the
> same way.
> Options I can see:
> # quote the token when {{regex=false}}, and document it in the upgrade guide
> (routes with regex tokens must set {{regex=true}}); maybe also convert the
> backslash-r and backslash-t escapes in the reifier as it already does for
> backslash-n;
> # keep compiling the token, but quote it when it is a single character (the
> common delimiter case: pipe, dot, dollar, star, plus), which changes no
> multi-character token;
> # keep the behaviour, and document that the token is always a regular
> expression (and what {{regex}} really means).
> Which would the committers prefer? I can send a PR with tests for either.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
