ghaarsma commented on issue #50667: URL: https://github.com/apache/arrow/issues/50667#issuecomment-5239013029
@jorisvandenbossche and others, I have finally been able to produce a **MRE** (see attached [timeseries_test.py](https://github.com/user-attachments/files/30893842/timeseries_test.py)). Whether the bug/performance degradation shows up depends on 3 factors: 1. Do the existing parquet files have nanosecond timestamps (holdover from Pandas < 3.0 default) 2. Is python pytz dependency installed (this was always the case for Pandas < 3.0) 3. Do you run pyarrow 24.0.0 or pyarrow 25.0.0 - With pyarrow 24.0.0 all is good, provided that you have pytz installed (no need to import). Both `us` and `ns` files work - With pyarrow 24.0.0, but no pytz, concatenating a 'ns' with a new 'us' dataframe becomes very slow (300~400 times slower for large timeseries) - With pyarrow 25.0.0, (irrespective of pytz installed), concatenating a 'ns' dataframe with a new 'us' dataframe becomes very slow ``` pyarrow 24.0.0 stats: ------------------------------------------------- read append write unit_in unit_out tz_in tz_out file syn_data_us.parquet 0.501423 0.081193 1.506878 us us datetime.timezone.utc datetime.timezone.utc syn_data_ns.parquet 0.504985 0.079682 1.768292 ns ns datetime.timezone.utc datetime.timezone.utc pyarrow 25.0.0 stats: ------------------------------------------------- read append write unit_in unit_out tz_in tz_out file syn_data_us.parquet 0.480525 0.078757 1.589404 us us zoneinfo.ZoneInfo(key='UTC') zoneinfo.ZoneInfo(key='UTC') syn_data_ns.parquet 0.492546 28.473683 1.583237 ns ns zoneinfo.ZoneInfo(key='UTC') datetime.timezone.utc ``` This was all run with the latest packages (where only pyarrow was changed from 24.0.0 to 25.0.0): ``` (venv) PS C:\Users\gabe\Python\pyarrow> pip list Package Version --------------- ------------ numpy 2.5.2 pandas 3.0.5 pip 26.1.2 pyarrow 24.0.0 <-> 25.0.0 python-dateutil 2.9.0.post0 pytz 2026.3.post1 six 1.17.0 tzdata 2026.3 ``` To be clear, the majority of the Parquet files were originally created under Pandas 2.x so it had `ns` timestamps. Concatenating these with a newer pandas 3.0 dataframe with `us` timestamps and writing back to parquet, retained the ns timestamps. Because I occasionally get some cache poisoning, I manually over time deleted 56 (see prior post) parquet files. This will now generate a brand new parquet file from scratch with only us timestamps. I'm still trying to wrap my mind around all of this, but at this time, I'm not convinced that the newer pyarrow 25.0.0 behavior of UTC timezone handling is an improvement over 24.0.0 I'm still trying to wrap my head around all of this. I think the majority of the fixes/cleanup will/should be on the pandas side of things. However -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
