Jackie-Jiang commented on issue #19696:
URL: https://github.com/apache/pinot/issues/19696#issuecomment-6000839918
Thanks for the report. The leading `\` and the alternatives you tried all
have the same root cause.
Table config values are resolved against environment variables and system
properties when the config is read: a string value that is entirely of the form
`${NAME:default}` is replaced by the value of `NAME`, or by `default` when
`NAME` is not set. `${filePathPattern:\1}` has exactly that shape, so it is
replaced by `\1` before the segment name generator sees it:
| `segment.name.template` in the table config (JSON) | Value the generator
receives | Segment name |
|---|---|---|
| `"\\${filePathPattern:\\1}"` | `\${filePathPattern:\1}` (not resolved,
does not start with `${`) | `\<file name>` |
| `"${filePathPattern:\\1}"` | `\1` | `\1` for every file |
| `"${filePathPattern:1}"` | `1` | `1` for every file |
| `"${filePathPattern:$1}"` | `$1` | `$1` for every file |
The `\${filePathPattern:\1}` form in the job specification docs is specific
to the ingestion job spec YAML, which is rendered with Groovy's
`SimpleTemplateEngine` where `\$` is the escape. Table configs don't go through
Groovy, so the backslash ends up in the segment name.
Workarounds:
- Released versions (up to 1.5.1): make sure the value is not entirely a
`${...}` expression by adding a literal prefix or suffix, e.g.
`"segmentNameGenerator.configs.segment.name.template":
"myTable_${filePathPattern:\\1}"`.
- Current master (not yet released): a leading `$$` skips the resolution
(#19202), so `"$${filePathPattern:\\1}"` reaches the generator as
`${filePathPattern:\1}`.
I'll also make the input file segment name generator reject a template that
doesn't reference any match group of `file.path.pattern`, so this fails with a
clear error instead of silently giving every segment the same name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
