jonathansudhakar1 opened a new issue, #40994:
URL: https://github.com/apache/beam/issues/40994
### What happened?
A Beam SQL Parquet table (`CREATE EXTERNAL TABLE ... TYPE parquet`) with a
`TIME` column can't be written. On master (b0c68c4a4f7), the INSERT fails with:
```
java.lang.RuntimeException: Unhandled logical type beam:logical_type:time:v1
```
Minimal repro, run as a JUnit test in `sdks/java/extensions/sql` with the
DirectRunner (`p` is a `TestPipeline`):
```java
BeamSqlEnv env = BeamSqlEnv.inMemory(new ParquetTableProvider());
env.executeDdl("CREATE EXTERNAL TABLE t (c TIME) TYPE parquet LOCATION
'/tmp/time-table/'");
BeamSqlRelUtils.toPCollection(p, env.parseQuery("INSERT INTO t VALUES (TIME
'12:34:56')"));
p.run().waitUntilFinish(); // RuntimeException: Unhandled logical type
beam:logical_type:time:v1
```
Calling `AvroUtils.toAvroSchema` on the table's schema, which is what
`ParquetTable` does ([L60, L69,
L111](https://github.com/apache/beam/blob/b0c68c4a4f7f9a87f87220b7413d14e9d0a17b9b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java#L60)),
throws the same exception.
**Cause.** The two sides use different identifiers for TIME:
- Beam SQL maps `TIME` to `FieldType.logicalType(SqlTypes.TIME)`, whose
identifier is `beam:logical_type:time:v1` ([CalciteUtils
L114](https://github.com/apache/beam/blob/b0c68c4a4f7f9a87f87220b7413d14e9d0a17b9b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/utils/CalciteUtils.java#L114)).
- `AvroUtils.getFieldSchema` only recognizes the legacy `"TIME"` identifier
([L1224](https://github.com/apache/beam/blob/b0c68c4a4f7f9a87f87220b7413d14e9d0a17b9b/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java#L1224)).
Every other identifier falls through to the `Unhandled logical type` exception
([L1242](https://github.com/apache/beam/blob/b0c68c4a4f7f9a87f87220b7413d14e9d0a17b9b/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java#L1242)).
- `genericFromBeamField` has the same gap
([L1389](https://github.com/apache/beam/blob/b0c68c4a4f7f9a87f87220b7413d14e9d0a17b9b/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java#L1389),
[L1406](https://github.com/apache/beam/blob/b0c68c4a4f7f9a87f87220b7413d14e9d0a17b9b/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java#L1406)).
DATE does handle both identifiers: `"DATE".equals(identifier) ||
SqlTypes.DATE.getIdentifier().equals(identifier)`
([L1220](https://github.com/apache/beam/blob/b0c68c4a4f7f9a87f87220b7413d14e9d0a17b9b/sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java#L1220)).
That's why DATE columns work and TIME columns don't.
**Expected:** `TIME` columns can be written and read back. The [CREATE
EXTERNAL TABLE
docs](https://github.com/apache/beam/blob/b0c68c4a4f7f9a87f87220b7413d14e9d0a17b9b/website/www/site/content/en/documentation/dsls/sql/extensions/create-external-table.md?plain=1#L566)
list `DATE, TIME, TIMESTAMP` as supported Parquet types.
A fix probably needs a design call. Should `SqlTypes.TIME` (a
`java.time.LocalTime`) map to Avro `time-micros` (long) or `time-millis` (int)?
Reading the column back also has to convert to `LocalTime`, and the
`time-millis`/`time-micros` branches of the Avro-to-Beam conversion currently
return the raw int/long. I'm happy to send a fix once a maintainer confirms
which mapping they'd like.
For reference, here is what `AvroUtils.toAvroSchema` produces today for each
type, from a temporary test that runs `CREATE EXTERNAL TABLE ... TYPE parquet`
for each column type:
| Beam SQL type | Beam field type | Avro type |
|---|---|---|
| TINYINT, SMALLINT, INTEGER | BYTE, INT16, INT32 | int |
| BIGINT | INT64 | long |
| FLOAT | FLOAT | float |
| DOUBLE | DOUBLE | double |
| DECIMAL | DECIMAL | bytes (decimal) |
| BOOLEAN | BOOLEAN | boolean |
| DATE | SqlTypes.DATE | int (date) |
| TIME | SqlTypes.TIME | **exception** |
| TIMESTAMP | DATETIME | long (timestamp-millis) |
| CHAR, VARCHAR | STRING | string |
| BINARY, VARBINARY | BYTES | bytes |
Several rows of the docs' Parquet type table don't match this: the int
types, FLOAT, DATE and TIME. I'll file that docs correction separately.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Prism Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
