kz930 opened a new issue, #7666: URL: https://github.com/apache/texera/issues/7666
### What happened? Reading an Arrow file, a TIMESTAMP column comes out different depending on the timezone of the machine the server runs on. The file is unchanged and the workflow is unchanged, so the same run on two deployments reports different data, with nothing to say so. Arrow stores a timestamp as an instant. On the way in, `ArrowUtils.getTexeraTuple` takes the vector's value, which for a zoned field is epoch milliseconds, and passes it to `AttributeTypeUtils.parseField` as a TIMESTAMP, reaching `new Timestamp(millis)`. That renders the instant in the JVM's default zone, and Texera's TIMESTAMP carries no zone of its own to record which one was used. The zone is therefore read off the server rather than off the file. Expected: one file yields one set of values, on every server. ### How to reproduce? Write a one row `.arrow` file holding a timestamp column. Read it with the Arrow File Scan operator on servers in different zones. Below is one file, written once, then read by four JVMs started with different `-Duser.timezone`. | Server timezone | Value the operator produces | | --- | --- | | America/Los_Angeles | 2024-01-07 00:00:00.0 | | UTC | 2024-01-07 08:00:00.0 | | Asia/Tokyo | 2024-01-07 17:00:00.0 | | Europe/Berlin | 2024-01-07 09:00:00.0 | The file holds a single instant, 1704614400000 milliseconds, which is 2024-01-07 08:00 UTC. Reading the same file with `pandas.read_feather` reports 2024-01-07 08:00:00+00:00 in every case, so the file itself is unambiguous. ### Version/Branch 1.3.0-incubating-SNAPSHOT (main) ### Commit Hash (Optional) 557b84a77 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
