I’ve done some testing using pyarrow.jvm as part of jdbc optimization.
https://arrow.apache.org/docs/python/integration/python_java.html
Pull a dataset using a JVM environment with JDBC.
Map JDBC arrow result set to a python pyarrow result set.
ra = jpype.JPackage("org").apache.arrow.memory.RootAllocator(sys.maxsize)
batch = jpype.JPackage("org").apache.arrow.adapter.jdbc.JdbcToArrow.sqlToArrow(
jdbc_connection,
sql,
ra
)
arrow_table = pyarrow.jvm.record_batch(batch)
From: Chang She <[email protected]>
Sent: Monday, October 24, 2022 11:44 AM
To: [email protected]
Subject: Re: Arrow in jupyter
External Email: Use caution with links and attachments
Awesome, I will continue on that JIRA.
Thanks!
On Mon, Oct 24, 2022 at 6:23 AM Neal Richardson
<[email protected]<mailto:[email protected]>> wrote:
Hi Chang,
I don't recall seeing an Arrow issue about this specifically, or any examples
of doing that, but that sounds nice.
https://issues.apache.org/jira/browse/ARROW-17608<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/ARROW-17608__;!!KSjYCgUGsB4!fr_nJ6s_jfTijlw-O2aBexeefqDuRbFByff-p8Zi9Y8VGZrzsqSy-tCzdikV_lCdBGz9Yl-nhYFCIrA$>
(C interface support in JS) is probably a prerequisite.
Neal
On Sun, Oct 23, 2022 at 2:45 PM Chang She <[email protected]<mailto:[email protected]>>
wrote:
couldn’t*
On Sun, Oct 23, 2022 at 11:42 AM Chang She <[email protected]<mailto:[email protected]>>
wrote:
Hi, just curious if anyone’s played with using Arrow to ship data between
languages in jupyter/ipython, specifically python and js (that doesn’t require
a flight server or writing the data to disk).
Eg, can I call do something like IPython.kernel.execute(“table.filter(…)”) and
read the results directly into arrowjs instead of converting from plain text /
json?
Could find an existing issue and figure I’d ask before I go chase this down.
Thanks!
Chang
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
http://www.blackrock.com/corporate/compliance/email-disclaimers for further
information. Please refer to
http://www.blackrock.com/corporate/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
http://www.blackrock.com/corporate/about-us/contacts-locations.
© 2022 BlackRock, Inc. All rights reserved.
