Thanks, Joris.
I was misled by the pandas -> Arrow Conversion table:
https://arrow.apache.org/docs/python/pandas.html#pandas-arrow-conversion
which does not show datetime.time -> time64 conversion.
A followup question is:
I have a table with a timestamp column in pandas/arrow and saved as
parquet format.
Is there a way to filter on the timestamp.time when reading the parquet?
For example ('x.time', '=', '10:00:00') or ('x.time', '=',
datetime.time(10)).
Currently, I am doing so by saving both a timestamp column and a time
column, which is kind of duplicate in information.
Thanks,
Wenyi
On Mon, Dec 6, 2021 at 7:10 AM Joris Van den Bossche <
[email protected]> wrote:
> On Fri, 3 Dec 2021 at 23:36, Wenyi Huang <[email protected]> wrote:
> >
> > Hi Arrow Team,
> >
> > What is the best data type to save Time of the Day if I want to use
> Pandas Datafrome, but dump data to parquet (or other formats) via PyArrow?
> >
> > I see that pandas Arrow conversion does not convert datetime.time nor
> timedelta.
>
> Can you show an example? Because I think that this conversion should
> handle both cases:
>
> >>> import datetime
> >>> df = pd.DataFrame({"time": [datetime.time(9)], "timedelta":
> [pd.Timedelta("9 hours")]})
> >>> df
> time timedelta
> 0 09:00:00 0 days 09:00:00
>
> >>> pa.table(df)
> pyarrow.Table
> time: time64[us]
> timedelta: duration[ns]
> ----
> time: [[09:00:00.000000]]
> timedelta: [[32400000000000]]
>
> The resulting Arrow table has columns with time and duration type
> (duration is the Arrow equivalent for timedelta).
>
> Joris
>
> >
> > The use case is that I want to save the time of the day column. So that
> while reading (parquet or other formats), I can filter by time.
> >
> > Best,
> > Wenyi
> >
> > --
> > Wenyi Huang
> > LinkedIn: https://www.linkedin.com/in/harrywy/
> > Google Scholar:
> https://scholar.google.com/citations?user=K-RWg7gAAAAJ&hl=en
> > Email: [email protected]
> >
> >
>
--
Wenyi Huang
AI & Quantitative Researcher @Citadel, LLC,
LinkedIn: https://www.linkedin.com/in/harrywy/
Google Scholar: *https://scholar.google.com/citations?user=K-RWg7gAAAAJ&hl=en
<https://scholar.google.com/citations?user=K-RWg7gAAAAJ&hl=en>*
Email: [email protected]
