Gah found a bug with my code.. Here's a corrected python version..
# iterate through possible nested columns
def _convert_to_arrow_type(field, obj):
"""
:param field:
:param obj:
:returns: pyarrow datatype
"""
if isinstance(obj, list):
for child_obj in obj:
pa_type = _convert_to_arrow_type(field, child_obj)
return pa.list_(pa_type)
elif isinstance(obj, dict):
items = []
for k, child_obj in obj.items():
pa_type = _convert_to_arrow_type(k, child_obj)
items.append((k, pa_type))
return pa.struct(items)
else:
if isinstance(obj, str):
if obj == "timestamp":
# default timestamp to microsecond precision
obj = "timestamp[us]"
elif obj == "date":
# default date to date32 which is an alias for date32[day]
obj = "date32"
elif obj == "int":
# default int to int32
obj = "int32"
obj = pa.type_for_alias(obj)
return obj
# iterate through columns to create a schema
def _convert_to_arrow_schema(fields_dict):
"""
:param fields_dict:
:returns: pyarrow schema
"""
columns = []
for field, typ in fields_dict.items():
pa_type = _convert_to_arrow_type(field, typ)
columns.append(pa.field(field, pa_type))
schema = pa.schema(columns)
return schema
-----Original Message-----
From: Lee, David (PAG) <[email protected]>
Sent: Monday, July 8, 2024 11:58 AM
To: [email protected]
Subject: RE: [DISCUSS] Approach to generic schema representation
External Email: Use caution with links and attachments
I came up with my own json representation that I could put into json / yaml
config files with some python code to convert this into a pyarrow schema
object..
------------- yaml flat example-------------
fields:
cusip: string
start_date: date32
end_date: date32
purpose: string
source: string
flow: float32
flow_usd: float32
currency: string
-------------yaml nested example-------------
fields:
cusip: string
start_date: date32
regions:
[string] << list of strings
primary_benchmark: << struct
id: string
name: string
all_benchmarks: << list of structs
-
id: string
name: string
Code:
def _convert_to_arrow_type(field, obj):
"""
:param field:
:param obj:
:returns: pyarrow datatype
"""
if isinstance(obj, list):
for child_obj in obj:
pa_type = _convert_to_arrow_type(field, child_obj)
return pa.list_(pa_type)
elif isinstance(obj, dict):
items = []
for k, child_obj in obj.items():
pa_type = _convert_to_arrow_type(k, child_obj)
items.append((k, pa_type))
return pa.struct(items)
else:
if isinstance(obj, str):
obj = pa.type_for_alias(obj)
return obj
def _convert_to_arrow_schema(fields_dict):
"""
:param fields_dict:
:returns: pyarrow schema
"""
columns = []
for field, typ in fields_dict.items():
if typ == "timestamp":
# default timestamp to microsecond precision
typ = "timestamp[us]"
elif typ == "date":
# default date to date32 which is an alias for date32[day]
typ = "date32"
elif typ == "int":
# default int to int32
typ = "int32"
pa_type = _convert_to_arrow_type(field, typ)
columns.append(pa.field(field, pa_type))
schema = pa.schema(columns)
return schema
-----Original Message-----
From: Weston Pace <[email protected]>
Sent: Monday, July 8, 2024 9:43 AM
To: [email protected]
Subject: Re: [DISCUSS] Approach to generic schema representation
External Email: Use caution with links and attachments
+1 for empty stream/file as schema serialization. I have used this
approach myself on more than one occasion and it works well. It can even be
useful for transmitting schemas between different arrow-native libraries in the
same language (e.g. rust->rust) since it allows the different libraries to use
different arrow versions.
There is one other approach if you only need intra-process serialization (e.g.
between threads / libraries in the same process). You can use the C data
interface
(https://urldefense.com/v3/__https://arrow.apache.org/docs/format/CDataInterface.html__;!!KSjYCgUGsB4!ZpcpNRWAd5SeffO0-cFZpVsg1ze7lbt7Btmp3SdyCqvZcsa1naBsVkk2SXPTgQpHRR-fJd_bupsM0-v2oXAljCk$
).
It is maybe a slightly more complex API (because of the release callback) and I
think it is unlikely to be significantly faster (unless you have an abnormally
large schema). However, it has the same advantages and might be useful if you
are already using the C data interface elsewhere.
On Mon, Jul 8, 2024 at 8:27 AM Matt Topol <[email protected]> wrote:
> Hey Jeremy,
>
> Currently the first message of an IPC stream is a Schema message which
> consists solely of a flatbuffer message and defined in the Schema.fbs
> file of the Arrow repo. All of the libraries that can read Arrow IPC
> should be able to also handle converting a single IPC schema message
> back into an Arrow schema without issue. Would that be sufficient for you?
>
> On Mon, Jul 8, 2024 at 11:12 AM Jeremy Leibs <[email protected]> wrote:
>
> > I'm looking for any advice folks may have on a generic way to
> > document
> and
> > represent expected arrow schemas as part of an interface definition.
> >
> > For context, our library provides a cross-language (python, c++,
> > rust)
> SDK
> > for logging semantic multi-modal data (point clouds, images,
> > geometric transforms, bounding boxes, etc.). Each of these primitive
> > types has an associated arrow schema, but to date we have largely
> > abstracted that from our users through language-native object types,
> > and a bunch of generated code to "serialize" stuff into the arrow
> > buffers before transmitting via our IPC.
> >
> > We're trying to take steps in the direction of making it easier for
> > advanced users to write and read data from the store directly using
> arrow,
> > without needing to go in-and-out of an intermediate object-oriented
> > representation. However, doing this means documenting to users, for
> > example: "This is the arrow schema to use when sending a point cloud
> with a
> > color channel".
> >
> > I would love it if, eventually, the arrow project had a way of
> > defining a spec file similar to a .proto or a .fbs, with all
> > libraries supporting loading of a schema object by directly parsing
> > the spec. Has anyone taken steps in this direction?
> >
> > The best alternative I have at the moment is to redundantly define
> > the schema for each of the 3 languages implicitly by directly
> > providing the code to construct a datatype instance with the correct
> > schema. But this feels unfortunately messy and hard to maintain.
> >
> > Thanks,
> > Jeremy
> >
>
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
http://www.blackrock.com/corporate/compliance/email-disclaimers for further
information. Please refer to
http://www.blackrock.com/corporate/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2024 BlackRock, Inc. All rights reserved.
