swyxio opened a new pull request, #16544:
URL: https://github.com/apache/pinot/pull/16544
- I read existing schema conversion utilities and commands, and implemented
a unified command to handle both Avro and Parquet schema inputs.
What I added next:
- New command: `InputSchemaToPinotSchema`
- Location:
`pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/InputSchemaToPinotSchema.java`
- Detects type from file extension (.avsc → AVRO_SCHEMA, .avro →
AVRO_DATA, .parquet → PARQUET) or via `-inputType`.
- Converts using:
- Avro schema file: `AvroUtils.getPinotSchemaFromAvroSchemaFile(...)`
- Avro data file: `AvroUtils.getPinotSchemaFromAvroDataFile(...)`
- Parquet file: `ParquetUtils.getParquetAvroSchema(...)` then
`AvroUtils` to Pinot schema
- Supports complex-type handling flags consistent with
`AvroSchemaToPinotSchema`
- Adds logs to validate assumptions:
- detected input type
- Parquet embedded Avro schema presence
- Registration:
- Added to `PinotAdministrator`:
`SUBCOMMAND_MAP.put("InputSchemaToPinotSchema", new
InputSchemaToPinotSchema());`
- Kept a dedicated Parquet command:
- New command: `ParquetSchemaToPinotSchema`
- Location:
`pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/ParquetSchemaToPinotSchema.java`
- Registered in `PinotAdministrator`
- This complements the existing `AvroSchemaToPinotSchema` and provides a
simple entry for Parquet-only workflows.
How to use (examples):
- pinot-admin InputSchemaToPinotSchema -inputFile /path/to/schema.avsc
-outputDir /tmp -pinotSchemaName MySchema -dimensions colA,colB
- pinot-admin InputSchemaToPinotSchema -inputFile /path/to/data.avro
-outputDir /tmp -pinotSchemaName MySchema -metrics m1,m2
- pinot-admin InputSchemaToPinotSchema -inputFile /path/to/file.parquet
-outputDir /tmp -pinotSchemaName MySchema -timeColumnName ts -timeUnit HOURS
- Optional: -inputType AVRO_SCHEMA|AVRO_DATA|PARQUET, -complexType,
-fieldsToUnnest a.b,c.d, -delimiter ., -collectionNotUnnestedToJson
NONE|NON_PRIMITIVE|ALL
- Logs added to validate key assumptions:
- Input type detection and Parquet embedded-avro metadata detection.
- No linter errors introduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
