Re: Independent releases and format version

[Micah Kornfield]

>
> A way to alleviate this is to change our status table from "pass / not
> pass" to a "since version X.Y". This would offer the transparency into
> which version it was implemented since, and would also align well with the
> idea that Rust X.Y.Z is bundled as part of the arrow release.


Great idea we should make this change (at least reset all check marks to
4.0).


> This does not address backward incompatible changes of the format, which is
> a whole different beast (e.g. do we require all implementations to change
> prior to releasing the new version of the format?)


When versioning (more specifically release 1.0 of the spec) came up, we
were hoping (and I think most people) would be opposed to breaking changes
at this point.   So this is a possibility, but I'd rather cross this road
when we come to it.

-Micah

On Wed, Apr 28, 2021 at 9:56 PM Jorge Cardoso Leitão <
jorgecarlei...@gmail.com> wrote:

> Hi,
>
> Thanks for bringing this up.
>
> AFAIK, this has always been an issue because we do not require all
> implementations to support a type prior to releasing that new type in the
> spec. Concrete example: map has been released for a while, but Rust does
> not support it yet. Same for interval, where we still do not test in
> integration.
>
> My understanding is that once we accept that we can bump the spec without
> having all implementations implement it, people will need to consult *some
> table*.
>
> A related issue is the *some table* bit: someone using pyarrow==2 can't
> tell whether they can send type X to Rust v2 via IPC. If they go to the
> latest table [1], they can only see the latest status (4.0 in this case).
> AFAIK they would need to download the source, go to the table in .rst for
> that release, and check whether type X is supported by Rust on that
> release.
>
> A way to alleviate this is to change our status table from "pass / not
> pass" to a "since version X.Y". This would offer the transparency into
> which version it was implemented since, and would also align well with the
> idea that Rust X.Y.Z is bundled as part of the arrow release.
>
> This does not address backward incompatible changes of the format, which is
> a whole different beast (e.g. do we require all implementations to change
> prior to releasing the new version of the format?)
>
> [1] https://arrow.apache.org/docs/status.html
>
>
>
> On Thu, Apr 29, 2021 at 2:27 AM Weston Pace  wrote:
>
> > We now have independent releases.  There has been some discussion (not
> > sure if it was formalized) around aligning major release versions
> > across the languages.
> >
> > There is also a potential format change coming up (new interval type).
> >
> > I think this brings up a few questions...
> >
> > Can an arrow library adopt a new format version in a minor release?
> > Semantic versioning would imply yes but then I think we could get in a
> > situation where Arrow-C++ X.Y.Z Might not be able to communicate via
> > flight using Arrow-C++ X.A.B.
> >
> > In addition, is there value in aligning format adoption across languages?
> >
> > For example, if Rust adopts format version 1.1 in version 5 and
> > pyarrow does not then users will need to consult a table to figure out
> > which versions are interoperable.
> >
> > -Weston
> >
>

Re: Independent releases and format version

[Jorge Cardoso Leitão]

Hi,

Thanks for bringing this up.

AFAIK, this has always been an issue because we do not require all
implementations to support a type prior to releasing that new type in the
spec. Concrete example: map has been released for a while, but Rust does
not support it yet. Same for interval, where we still do not test in
integration.

My understanding is that once we accept that we can bump the spec without
having all implementations implement it, people will need to consult *some
table*.

A related issue is the *some table* bit: someone using pyarrow==2 can't
tell whether they can send type X to Rust v2 via IPC. If they go to the
latest table [1], they can only see the latest status (4.0 in this case).
AFAIK they would need to download the source, go to the table in .rst for
that release, and check whether type X is supported by Rust on that release.

A way to alleviate this is to change our status table from "pass / not
pass" to a "since version X.Y". This would offer the transparency into
which version it was implemented since, and would also align well with the
idea that Rust X.Y.Z is bundled as part of the arrow release.

This does not address backward incompatible changes of the format, which is
a whole different beast (e.g. do we require all implementations to change
prior to releasing the new version of the format?)

[1] https://arrow.apache.org/docs/status.html



On Thu, Apr 29, 2021 at 2:27 AM Weston Pace  wrote:

> We now have independent releases.  There has been some discussion (not
> sure if it was formalized) around aligning major release versions
> across the languages.
>
> There is also a potential format change coming up (new interval type).
>
> I think this brings up a few questions...
>
> Can an arrow library adopt a new format version in a minor release?
> Semantic versioning would imply yes but then I think we could get in a
> situation where Arrow-C++ X.Y.Z Might not be able to communicate via
> flight using Arrow-C++ X.A.B.
>
> In addition, is there value in aligning format adoption across languages?
>
> For example, if Rust adopts format version 1.1 in version 5 and
> pyarrow does not then users will need to consult a table to figure out
> which versions are interoperable.
>
> -Weston
>

Re: Independent releases and format version

[Micah Kornfield]

Hi Weston,
I think one other issue that we might want to discuss is releasing the
format separately (i.e. as soon as we make a change to it).

My thoughts:

Can an arrow library adopt a new format version in a minor release?


I agree the answer is yes, and some issues might arise if minor releases
are handled differently among different languages.   last time releases
where discussed it sounded like minor releases where on any implementations
near term road map though.

In addition, is there value in aligning format adoption across languages?


The current standard for accepting a specification update is there needs to
be working integration tests between the C++ and Java versions
demonstrating the feature.  I think the current model has worked fairly
well.  We demonstrate interop with two languages and then the others might
lag a little bit (or unfortunately a lot).  We might want to revisit the
exact standard for adoption, but I think a minimum of two language bindings
is important.  The more languages we add into the mix means the slower the
adoption of a new feature.  Personally, I think 2 or 3 should be the goal,
anything more would be too cumbersome.

So in summary, I think for better or worse users will likely need to
consult the table (which already exists).

Cheers,
Micah

On Wed, Apr 28, 2021 at 5:27 PM Weston Pace  wrote:

> We now have independent releases.  There has been some discussion (not
> sure if it was formalized) around aligning major release versions
> across the languages.
>
> There is also a potential format change coming up (new interval type).
>
> I think this brings up a few questions...
>
> Can an arrow library adopt a new format version in a minor release?
> Semantic versioning would imply yes but then I think we could get in a
> situation where Arrow-C++ X.Y.Z Might not be able to communicate via
> flight using Arrow-C++ X.A.B.
>
> In addition, is there value in aligning format adoption across languages?
>
> For example, if Rust adopts format version 1.1 in version 5 and
> pyarrow does not then users will need to consult a table to figure out
> which versions are interoperable.
>
> -Weston
>

Re: Arrow sync call April 28 at 12:00 US/Eastern, 16:00 UTC

[Neal Richardson]

Attendees:

Nate Bauernfeind
Ian Cook
Nic Crane
Jonathan Keane
Micah Kornfield
Andrew Lamb
Rok Mihevc
Ruan Pearce-Authers
Prudhvi Porandla
Neal Richardson
Ying Zhou

Discussion:

* 4.0 release: done! Watch out for issues that might merit a patch release,
raise on the mailing list. Note that Rust also may be releasing more
frequently, will need to work out how that works in practice. Also, still
need to fill in the release blog post draft.
* Interval format addition: PR up, please comment. Some questions about
whether this is a format change (looks like it), format version bumping (we
probably should have done so with Decimal256 too) and how that works in
practice, and solidifying the criteria for when something should be an
extension type vs. a format addition.

On Tue, Apr 27, 2021 at 6:56 PM Neal Richardson 
wrote:

> Hi all,
> Our biweekly call is coming up tomorrow at
> https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will
> be shared with the mailing list afterward.
>
> Neal
>

Re: Please Review: Application for a Media Type

[Weston Pace]

+1 for .arrows from me.  I agree that .stream is too generic.


On Thu, Apr 22, 2021 at 7:42 PM Sutou Kouhei  wrote:
>
> Hi,
>
> I feel that '.stream' is too generic. How about '.arrows'?
> JSON Lines uses 'l' suffix for extension: '.jsonl'
>
> https://jsonlines.org/#conventions
>
>
> Thanks,
> --
> kou
>
> In 
>   "Re: Please Review: Application for a Media Type" on Thu, 22 Apr 2021 
> 06:44:51 +0200,
>   Jorge Cardoso Leitão  wrote:
>
> > Thanks for driving this, exciting stuff!
> >
> > I went through it, left minor comments, it looks good to me.
> >
> > wrt to the extension: imo they should be different as the formats are not
> > interchangeable.
> >
> > AFAIK `.stream` is not taken: it was used by Adobe shockwave but it was
> > discontinued [1].
> > So, .arrow and .stream may be sufficient.
> >
> > [1] https://helpx.adobe.com/shockwave/shockwave-end-of-life-faq.html
> >
> > Best,
> > Jorge
> >
> >
> > On Thu, Apr 22, 2021 at 3:35 AM Sutou Kouhei  wrote:
> >
> >> Hi,
> >>
> >> Thanks for updating the draft.
> >>
> >> I want to wait for at least a weak before we start a vote.
> >> Does anyone have an opinion about file extension of Apache
> >> Arrow format data? What do you think about ".arrow"?
> >>
> >>
> >> Thanks,
> >> --
> >> kou
> >>
> >> In 
> >>   "Re: Please Review: Application for a Media Type" on Wed, 21 Apr 2021
> >> 08:17:40 -1000,
> >>   Weston Pace  wrote:
> >>
> >> > Thank you for reviewing.  I have added your suggestions to the draft.
> >> > Are we ready for a vote?  If so I will clean up the comments and send
> >> > out a clean version of the draft.
> >> >
> >> > On Mon, Apr 19, 2021 at 3:10 PM Sutou Kouhei  wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> Sorry for not responding this...
> >> >>
> >> >> Weston, thanks for writing up the draft!
> >> >>
> >> https://docs.google.com/document/d/1PmZFoSifV_TX4vXnv775WiOtqCgz5zLF5ryFRWio3HQ/edit?usp=sharing
> >> >>
> >> >> Here are items we need to discuss before we apply a media
> >> >> type to IANA:
> >> >>
> >> >> 1. Interoperability Considerations
> >> >>
> >> >> Draft:
> >> >>
> >> >> > The Apache arrow format is intended to be a language
> >> >> > independent columnar memory format for flat and
> >> >> > hierarchical data.  It has been shown to work in a variety
> >> >> > of languages and applications.  Arrow files can be
> >> >> > provided in two different formats, a streaming format
> >> >> > (vnd.apache.arrow.stream) and a random access format
> >> >> > (vnd.apache.arrow.file).  Applications should be aware of
> >> >> > which format they are processing as the two are not
> >> >> > interchangeable.
> >> >>
> >> >> Note in draft:
> >> >>
> >> >> > Should we mention something like "applications should
> >> >> > make sure to check the 'version' field to ensure they
> >> >> > can process the file"?
> >> >>
> >> >> How about referring our format document for further
> >> >> information instead of mention the 'version' field?
> >> >> https://arrow.apache.org/docs/format/Columnar.html
> >> >>
> >> >> XML Media Types also refers the XML specification for
> >> >> further information:
> >> >>
> >> >> https://tools.ietf.org/html/rfc7303#section-9.1
> >> >>
> >> >> > For further information, see Section 2.9 "Standalone
> >> >> > Document Declaration" and Section 5 "Conformance" of [XML].
> >> >>
> >> >>
> >> >> 2. File extension(s)
> >> >>
> >> >> Draft:
> >> >>
> >> >> > N/A
> >> >>
> >> >> Note in draft:
> >> >>
> >> >> > Again, there are no formal extensions that have been
> >> >> > recommended before.  Do we want to introduce any?  I'm
> >> >> > pretty sure this is in no way binding (and it's unlikely
> >> >> > anyone will ever see it).
> >> >>
> >> >> I want recommended extensions to avoid spreading various
> >> >> extensions for Apache Arrow formats.
> >> >>
> >> >> How about the followings?
> >> >>
> >> >>   * vnd.apache.arrow.file: .arrow
> >> >>   * vnd.apache.arrow.stream: NA
> >> >> (Generally, this format isn't saved as file. This format
> >> >> is used for pipe, sending/receiving via socket and so on.)
> >> >>
> >> >> FYI: Here is a list that shows used extensions in our code
> >> >> base.
> >> >>
> >> >> Our integration test uses the following extensions:
> >> >>
> >> >>   * vnd.apache.arrow.file: .arrow_file
> >> >>   * vnd.apache.arrow.stream: .stream
> >> >>
> >> >>
> >> https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/runner.py#L250-L257
> >> >>
> >> >> log('-- Validating file')
> >> >> producer_file_path = os.path.join(
> >> >> gold_dir, "generated_" + test_case.name + ".arrow_file")
> >> >> consumer.validate(json_path, producer_file_path)
> >> >>
> >> >> log('-- Validating stream')
> >> >> consumer_stream_path = os.path.join(
> >> >> gold_dir, "generated_" + test_case.name + ".stream")
> >> >>
> >> >> Our C++ tests use the following extensions:
> >> >>
> >> >>   * vnd.apache.arrow.file: Not used (in-memory buffer is used)
> >> >>   * vnd.apache.arrow.stream: Not

Independent releases and format version

[Weston Pace]

We now have independent releases.  There has been some discussion (not
sure if it was formalized) around aligning major release versions
across the languages.

There is also a potential format change coming up (new interval type).

I think this brings up a few questions...

Can an arrow library adopt a new format version in a minor release?
Semantic versioning would imply yes but then I think we could get in a
situation where Arrow-C++ X.Y.Z Might not be able to communicate via
flight using Arrow-C++ X.A.B.

In addition, is there value in aligning format adoption across languages?

For example, if Rust adopts format version 1.1 in version 5 and
pyarrow does not then users will need to consult a table to figure out
which versions are interoperable.

-Weston

Re: [Python] Who has been able to use PyArrow 4.0.0?

[Wes McKinney]

In general, it isn't a good idea to install into /usr (or whatever is the
default path for your system) when you are doing development. I strongly
recommend using isolated install paths and using LD_LIBRARY_PATH when you
want to use your particular development version in an application.

On Wed, Apr 28, 2021 at 7:55 AM Ying Zhou  wrote:

> Well, I do have my own dev version of libarrow (with my own modifications)
> manually installed. I can verify that the pip install went smoothly on my
> work computer which has none of the Arrow development I do after work.
> Moreover I did find that ORC has been reenabled in the wheel and have used
> both the reader and the writer without issues.
>
> As for Conda I did manage to get the pyarrow 4.0.0 but there is no ORC
> functionality since any attempt to import from pyarrow.orc lead to an error
> caused by 'pyarrow._orc isn’t found'.
>
> Ying
>
>
>
> > On Apr 28, 2021, at 5:06 AM, Alessandro Molina <
> alessan...@ursacomputing.com> wrote:
> >
> > Are you sure you haven't installed `libarrow` (the CPP one) manually
> > independently from pyarrow?
> >
> > In your traceback you have that the symbol has not been found in
> > "/usr/local/lib/libarrow.400.dylib"
> >
> > But that smells like an independently installed libarrow, as the libarrow
> > provided by pyarrow should exist in the pytnon environment (in my case
> for
> > example
> /usr/local/lib/python3.9/site-packages/pyarrow/libarrow.400.dylib
> > ) I suspect your system installed libarrow is taking precedence over the
> > one provided by pyarrow and the two might not match.
> >
> > On Wed, Apr 28, 2021 at 10:05 AM Ying Zhou  wrote:
> >
> >> Hi,
> >>
> >> It turns out that I haven’t been able to use PyArrow 4.0.0 either in
> Conda
> >> environments or python venvs. PyArrow does install using pip. However
> this
> >> is what I get if I ever want to use it:
> >>
> > import pyarrow as pa
> >> Traceback (most recent call last):
> >>  File "", line 1, in 
> >>  File
> >>
> "/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/__init__.py",
> >> line 63, in 
> >>import pyarrow.lib as _lib
> >> ImportError:
> >> dlopen(/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> >> lib.cpython-38-darwin.so, 2): Symbol not found:
> >> __ZN5arrow10StopSource5tokenEv
> >>  Referenced from:
> >> /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> >> lib.cpython-38-darwin.so
> >>  Expected in: /usr/local/lib/libarrow.400.dylib
> >> in /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> >> lib.cpython-38-darwin.so
> > pa
> >> Traceback (most recent call last):
> >>  File "", line 1, in 
> >> NameError: name 'pa' is not defined
> >>
> >> On the other hand a Conda installation is not even possible. Does anyone
> >> know what’s going on?
> >>
> >> Ying
>
>

Re: [ANNOUNCE] New Arrow committer: Ian Cook

[paddy horan]

Congrats Ian!



From: Jorge Cardoso Leit?o 
Sent: Wednesday, April 28, 2021 4:56:12 PM
To: dev@arrow.apache.org 
Subject: Re: [ANNOUNCE] New Arrow committer: Ian Cook

Congratulations and thank you for your contributions :)

On Wed, Apr 28, 2021 at 10:37 PM Neal Richardson <
neal.p.richard...@gmail.com> wrote:

> On behalf of the Arrow PMC, I'm happy to announce that Ian has accepted an
> invitation to become a committer on Apache Arrow. Welcome, and thank you
> for your contributions!
>
> Neal
>

Re: [ANNOUNCE] New Arrow committer: Ian Cook

[David Li]

Congrats Ian!

-David

On Wed, Apr 28, 2021, at 16:56, Jorge Cardoso Leitão wrote:
> Congratulations and thank you for your contributions :)
> 
> On Wed, Apr 28, 2021 at 10:37 PM Neal Richardson <
> neal.p.richard...@gmail.com > wrote:
> 
> > On behalf of the Arrow PMC, I'm happy to announce that Ian has accepted an
> > invitation to become a committer on Apache Arrow. Welcome, and thank you
> > for your contributions!
> >
> > Neal
> >
> 

Re: [ANNOUNCE] New Arrow committer: Jonathan Keane

[David Li]

Congrats Jonathan!

-David

On Wed, Apr 28, 2021, at 16:55, Jorge Cardoso Leitão wrote:
> Congratulations and thank you for your contributions :)
> 
> On Wed, Apr 28, 2021 at 10:37 PM Neal Richardson <
> neal.p.richard...@gmail.com > wrote:
> 
> > On behalf of the Arrow PMC, I'm happy to announce that Jonathan has
> > accepted an invitation to become a committer on Apache Arrow. Welcome, and
> > thank you for your contributions!
> >
> > Neal
> >
> 

Re: [ANNOUNCE] New Arrow committer: Ian Cook

[Jorge Cardoso Leitão]

Congratulations and thank you for your contributions :)

On Wed, Apr 28, 2021 at 10:37 PM Neal Richardson <
neal.p.richard...@gmail.com> wrote:

> On behalf of the Arrow PMC, I'm happy to announce that Ian has accepted an
> invitation to become a committer on Apache Arrow. Welcome, and thank you
> for your contributions!
>
> Neal
>

Re: [ANNOUNCE] New Arrow committer: Jonathan Keane

[Jorge Cardoso Leitão]

Congratulations and thank you for your contributions :)

On Wed, Apr 28, 2021 at 10:37 PM Neal Richardson <
neal.p.richard...@gmail.com> wrote:

> On behalf of the Arrow PMC, I'm happy to announce that Jonathan has
> accepted an invitation to become a committer on Apache Arrow. Welcome, and
> thank you for your contributions!
>
> Neal
>

Re: [ANNOUNCE] New Arrow committer: Ian Cook

[Wes McKinney]

Congrats Ian!

On Wed, Apr 28, 2021 at 3:37 PM Neal Richardson 
wrote:

> On behalf of the Arrow PMC, I'm happy to announce that Ian has accepted an
> invitation to become a committer on Apache Arrow. Welcome, and thank you
> for your contributions!
>
> Neal
>

Re: [ANNOUNCE] New Arrow committer: Jonathan Keane

[Wes McKinney]

Congrats Jonathan!

On Wed, Apr 28, 2021 at 3:37 PM Neal Richardson 
wrote:

> On behalf of the Arrow PMC, I'm happy to announce that Jonathan has
> accepted an invitation to become a committer on Apache Arrow. Welcome, and
> thank you for your contributions!
>
> Neal
>

[ANNOUNCE] New Arrow committer: Jonathan Keane

[Neal Richardson]

On behalf of the Arrow PMC, I'm happy to announce that Jonathan has
accepted an invitation to become a committer on Apache Arrow. Welcome, and
thank you for your contributions!

Neal

[ANNOUNCE] New Arrow committer: Ian Cook

[Neal Richardson]

On behalf of the Arrow PMC, I'm happy to announce that Ian has accepted an
invitation to become a committer on Apache Arrow. Welcome, and thank you
for your contributions!

Neal

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[David Li]

Congrats Daniël!

Best,
David

On Wed, Apr 28, 2021, at 14:51, Micah Kornfield wrote:
> Congrats!
> 
> On Wed, Apr 28, 2021 at 11:44 AM Fernando Herrera <
> fernando.j.herr...@gmail.com > wrote:
> 
> > Congrats Daniël,
> > It is nice seeing all the code you apport to datafusion and ballista
> >
> > On Wed, Apr 28, 2021 at 7:39 PM QP Hou  > > wrote:
> >
> > > Congrats Daniël, well deserved!
> > >
> > > Thanks,
> > > QP Hou
> > >
> > > On Wed, Apr 28, 2021 at 6:25 AM Andy Grove  > > >
> > wrote:
> > > >
> > > > On behalf of the Arrow PMC, I'm happy to announce that Daniël has
> > > >
> > > > accepted an invitation to become a committer on Apache Arrow.
> > > >
> > > > Welcome, and thank you for your contributions!
> > >
> >
> 

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[Micah Kornfield]

Congrats!

On Wed, Apr 28, 2021 at 11:44 AM Fernando Herrera <
fernando.j.herr...@gmail.com> wrote:

> Congrats Daniël,
> It is nice seeing all the code you apport to datafusion and ballista
>
> On Wed, Apr 28, 2021 at 7:39 PM QP Hou  wrote:
>
> > Congrats Daniël, well deserved!
> >
> > Thanks,
> > QP Hou
> >
> > On Wed, Apr 28, 2021 at 6:25 AM Andy Grove 
> wrote:
> > >
> > > On behalf of the Arrow PMC, I'm happy to announce that Daniël has
> > >
> > > accepted an invitation to become a committer on Apache Arrow.
> > >
> > > Welcome, and thank you for your contributions!
> >
>

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[Fernando Herrera]

Congrats Daniël,
It is nice seeing all the code you apport to datafusion and ballista

On Wed, Apr 28, 2021 at 7:39 PM QP Hou  wrote:

> Congrats Daniël, well deserved!
>
> Thanks,
> QP Hou
>
> On Wed, Apr 28, 2021 at 6:25 AM Andy Grove  wrote:
> >
> > On behalf of the Arrow PMC, I'm happy to announce that Daniël has
> >
> > accepted an invitation to become a committer on Apache Arrow.
> >
> > Welcome, and thank you for your contributions!
>

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[QP Hou]

Congrats Daniël, well deserved!

Thanks,
QP Hou

On Wed, Apr 28, 2021 at 6:25 AM Andy Grove  wrote:
>
> On behalf of the Arrow PMC, I'm happy to announce that Daniël has
>
> accepted an invitation to become a committer on Apache Arrow.
>
> Welcome, and thank you for your contributions!

Re: Including JS patch in 4.0.1 if released

[Adam Lippai]

Really good question, it should throw upon Builder() creation.
The change in the PR fixes an exception when importing the Arrow lib.
I'll test it later today or tomorrow.

Best regards,
Adam Lippai


On Wed, Apr 28, 2021 at 6:38 PM Dominik Moritz  wrote:

> Thank you for the pull request.
>
> I’m curious regarding, CSP (which restricts the use of eval), did you try
> that Arrow 3 works? I’m wondering since Arrow already uses a Function
> constructor in
>
> https://github.com/apache/arrow/blob/0d11014ee8e6ce408ddbbdfb788d901dd6c6374f/js/src/builder/valid.ts#L66
> .
>
> On Apr 28, 2021 at 00:55:27, Adam Lippai  wrote:
>
> > Hi,
> >
> > I'd want to propose including https://github.com/apache/arrow/pull/10181
> > in
> > 4.0.1 release, if it happens.
> > While the issue is barely a bugfix, it's still a minor regression since
> > 3.0.0. It happens only in special circumstances, eg. using Rollup bundler
> > or restricting eval() for security reasons. It doesn't warrant a new
> > release on it's own as it affects a marginal fraction of the consumers
> > only.
> >
> > The change shouldn't affect the API or the required NodeJS versions.
> >
> > Best regards,
> > Adam Lippai
> >
>

Re: Including JS patch in 4.0.1 if released

[Dominik Moritz]

Thank you for the pull request.

I’m curious regarding, CSP (which restricts the use of eval), did you try
that Arrow 3 works? I’m wondering since Arrow already uses a Function
constructor in
https://github.com/apache/arrow/blob/0d11014ee8e6ce408ddbbdfb788d901dd6c6374f/js/src/builder/valid.ts#L66
.

On Apr 28, 2021 at 00:55:27, Adam Lippai  wrote:

> Hi,
>
> I'd want to propose including https://github.com/apache/arrow/pull/10181
> in
> 4.0.1 release, if it happens.
> While the issue is barely a bugfix, it's still a minor regression since
> 3.0.0. It happens only in special circumstances, eg. using Rollup bundler
> or restricting eval() for security reasons. It doesn't warrant a new
> release on it's own as it affects a marginal fraction of the consumers
> only.
>
> The change shouldn't affect the API or the required NodeJS versions.
>
> Best regards,
> Adam Lippai
>

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[Ying Zhou]

Congrats Daniël! 

> On Apr 28, 2021, at 9:24 AM, Andy Grove  wrote:
> 
> On behalf of the Arrow PMC, I'm happy to announce that Daniël has
> 
> accepted an invitation to become a committer on Apache Arrow.
> 
> Welcome, and thank you for your contributions!

RE: [ANNOUNCE] New Arrow committer: Daniël Heres

[Ruan Pearce-Authers]

Awesome stuff Daniël, well-deserved :D

> -Original Message-
> From: Daniël Heres 
> Sent: 28 April 2021 17:26
> To: dev@arrow.apache.org
> Subject: Re: [ANNOUNCE] New Arrow committer: Daniël Heres
> 
> Thank you all!
> 
> It has been an amazing experience working with you! Looking forward to
> more collaboration, let's take Arrow to new heights! :)
> 
> Best, Daniël
> 
> On Wed, Apr 28, 2021, 18:21 Rok Mihevc  wrote:
> 
> > Congrats Daniël!
> >
> > Rok
> >
> > On Wed, Apr 28, 2021 at 6:16 PM Jorge Cardoso Leitão <
> > jorgecarlei...@gmail.com> wrote:
> >
> > > Congratulations, and thank you for your contributions; I have been
> > learning
> > > a lot from you :)
> > >
> > > Best,
> > > Jorge
> > >
> > >
> > > On Wed, Apr 28, 2021 at 6:10 PM Andrew Lamb
> 
> > wrote:
> > >
> > > > Congratulations Daniël!
> > > >
> > > > On Wed, Apr 28, 2021 at 9:25 AM Andy Grove
> 
> > > wrote:
> > > >
> > > > > On behalf of the Arrow PMC, I'm happy to announce that Daniël
> > > > > has
> > > > >
> > > > > accepted an invitation to become a committer on Apache Arrow.
> > > > >
> > > > > Welcome, and thank you for your contributions!
> > > > >
> > > >
> > >
> >

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[Daniël Heres]

Thank you all!

It has been an amazing experience working with you! Looking forward to more
collaboration, let's take Arrow to new heights! :)

Best, Daniël

On Wed, Apr 28, 2021, 18:21 Rok Mihevc  wrote:

> Congrats Daniël!
>
> Rok
>
> On Wed, Apr 28, 2021 at 6:16 PM Jorge Cardoso Leitão <
> jorgecarlei...@gmail.com> wrote:
>
> > Congratulations, and thank you for your contributions; I have been
> learning
> > a lot from you :)
> >
> > Best,
> > Jorge
> >
> >
> > On Wed, Apr 28, 2021 at 6:10 PM Andrew Lamb 
> wrote:
> >
> > > Congratulations Daniël!
> > >
> > > On Wed, Apr 28, 2021 at 9:25 AM Andy Grove 
> > wrote:
> > >
> > > > On behalf of the Arrow PMC, I'm happy to announce that Daniël has
> > > >
> > > > accepted an invitation to become a committer on Apache Arrow.
> > > >
> > > > Welcome, and thank you for your contributions!
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[Rok Mihevc]

Congrats Daniël!

Rok

On Wed, Apr 28, 2021 at 6:16 PM Jorge Cardoso Leitão <
jorgecarlei...@gmail.com> wrote:

> Congratulations, and thank you for your contributions; I have been learning
> a lot from you :)
>
> Best,
> Jorge
>
>
> On Wed, Apr 28, 2021 at 6:10 PM Andrew Lamb  wrote:
>
> > Congratulations Daniël!
> >
> > On Wed, Apr 28, 2021 at 9:25 AM Andy Grove 
> wrote:
> >
> > > On behalf of the Arrow PMC, I'm happy to announce that Daniël has
> > >
> > > accepted an invitation to become a committer on Apache Arrow.
> > >
> > > Welcome, and thank you for your contributions!
> > >
> >
>

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[Jorge Cardoso Leitão]

Congratulations, and thank you for your contributions; I have been learning
a lot from you :)

Best,
Jorge


On Wed, Apr 28, 2021 at 6:10 PM Andrew Lamb  wrote:

> Congratulations Daniël!
>
> On Wed, Apr 28, 2021 at 9:25 AM Andy Grove  wrote:
>
> > On behalf of the Arrow PMC, I'm happy to announce that Daniël has
> >
> > accepted an invitation to become a committer on Apache Arrow.
> >
> > Welcome, and thank you for your contributions!
> >
>

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[paddy horan]

Congrats Daniël!



From: Andy Grove 
Sent: Wednesday, April 28, 2021 9:24:41 AM
To: dev 
Subject: [ANNOUNCE] New Arrow committer: Daniël Heres

On behalf of the Arrow PMC, I'm happy to announce that Daniël has

accepted an invitation to become a committer on Apache Arrow.

Welcome, and thank you for your contributions!

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[Wes McKinney]

Congrats!

On Wed, Apr 28, 2021 at 11:10 AM Andrew Lamb  wrote:

> Congratulations Daniël!
>
> On Wed, Apr 28, 2021 at 9:25 AM Andy Grove  wrote:
>
> > On behalf of the Arrow PMC, I'm happy to announce that Daniël has
> >
> > accepted an invitation to become a committer on Apache Arrow.
> >
> > Welcome, and thank you for your contributions!
> >
>

Re: [ANNOUNCE] New Arrow committer: Daniël Heres

[Andrew Lamb]

Congratulations Daniël!

On Wed, Apr 28, 2021 at 9:25 AM Andy Grove  wrote:

> On behalf of the Arrow PMC, I'm happy to announce that Daniël has
>
> accepted an invitation to become a committer on Apache Arrow.
>
> Welcome, and thank you for your contributions!
>

Re: [DISCUSS] Moving the format directory to arrow-format repository

[Jorge Cardoso Leitão]

Hi,

imo the time-scale of changes in the format is too large to justify the
complexity. I also think that we should not force users to clone or
submodule the repo to even compile the crate.

What if we just do not have the format files there at all, and instead just
keep the generated code? Updates to the format are manually performed (i.e.
instead of copying the file, we run the build against latest). This is what
we already do in practice, anyways: whenever the format changed, tonic
would auto-generate new rust code that we would commit before any change.

This avoids having copies of files between repos, which imo confuses people
into which one is the official one (even when they are all equal, people
will not do diffs in their heads).

Best,
Jorge



On Wed, Apr 28, 2021 at 12:13 PM Andrew Lamb  wrote:

> I also think manually copying the format .fbs files to arrow-rs is probably
> ok for the time being.
>
> Once Arrow gets to the point where many implementations that need
> format.fbs live in many different repos, pulling out the format files into
> their own repo might be worth reconsidering.
>
> Andrew
>
> On Tue, Apr 27, 2021 at 5:45 PM Wes McKinney  wrote:
>
> > I wouldn't be too excited about this. Here are my thoughts:
> >
> > 1. Having the format/ directory in apache/arrow be a submodule would be
> > cumbersome and error-prone for developers. The only submodules we have
> > right now are optional testing dependencies — not having these
> initialized
> > and updated does not result in a broken project, whereas this change
> would.
> > We have a copy of parquet.thrift from apache/parquet-format for similar
> > reasons.
> >
> > 2. So based on #1, we would want to maintain a copy of the format files
> in
> > apache/arrow, even if there were a separate apache/arrow-format
> repository.
> > The format files are slow-moving enough that I don't think it's
> burdensome
> > to mirror these into satellite repositories like arrow/arrow-rs.
> >
> > On Tue, Apr 27, 2021 at 10:54 AM Neville Dipale 
> > wrote:
> >
> > > Hi Arrow devs,
> > >
> > > Andy noticed that we carry a copy of the format directory in arrow-rs,
> > > which
> > > is bound to get outdated in the future.
> > >
> > > We would like to propose creating an arrow-format repository, similar
> to
> > > parquet-format, so that arrow-rs and other future separate repositories
> > > could
> > > add this as a submodule.
> > >
> > > What are your thoughts?
> > >
> > > Regards
> > > Neville
> > >
> >
>

[Rust] Helping wiih release notes for 4.0.0

[Andy Grove]

Neal has created a PR for release notes for 4.0.0 [1] and we need to help
with the Rust section of this.

I have created a Google doc for the Rust maintainers and contributors to
collaborate on since this is the usual process, and then we can submit as a
suggested change to the PR once ready.

Thanks,

Andy.

[1] https://github.com/apache/arrow-site/pull/104
[2]
https://docs.google.com/document/d/1ISVaVXfKnJnzAzLt_ivIYvM6MaRBbDJzi2D44x-lcPE/edit?usp=sharing

[ANNOUNCE] New Arrow committer: Daniël Heres

[Andy Grove]

On behalf of the Arrow PMC, I'm happy to announce that Daniël has

accepted an invitation to become a committer on Apache Arrow.

Welcome, and thank you for your contributions!

Re: [Python] Who has been able to use PyArrow 4.0.0?

[Ying Zhou]

Well, I do have my own dev version of libarrow (with my own modifications) 
manually installed. I can verify that the pip install went smoothly on my work 
computer which has none of the Arrow development I do after work. Moreover I 
did find that ORC has been reenabled in the wheel and have used both the reader 
and the writer without issues.

As for Conda I did manage to get the pyarrow 4.0.0 but there is no ORC 
functionality since any attempt to import from pyarrow.orc lead to an error 
caused by 'pyarrow._orc isn’t found'.

Ying



> On Apr 28, 2021, at 5:06 AM, Alessandro Molina  
> wrote:
> 
> Are you sure you haven't installed `libarrow` (the CPP one) manually
> independently from pyarrow?
> 
> In your traceback you have that the symbol has not been found in
> "/usr/local/lib/libarrow.400.dylib"
> 
> But that smells like an independently installed libarrow, as the libarrow
> provided by pyarrow should exist in the pytnon environment (in my case for
> example  /usr/local/lib/python3.9/site-packages/pyarrow/libarrow.400.dylib
> ) I suspect your system installed libarrow is taking precedence over the
> one provided by pyarrow and the two might not match.
> 
> On Wed, Apr 28, 2021 at 10:05 AM Ying Zhou  wrote:
> 
>> Hi,
>> 
>> It turns out that I haven’t been able to use PyArrow 4.0.0 either in Conda
>> environments or python venvs. PyArrow does install using pip. However this
>> is what I get if I ever want to use it:
>> 
> import pyarrow as pa
>> Traceback (most recent call last):
>>  File "", line 1, in 
>>  File
>> "/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/__init__.py",
>> line 63, in 
>>import pyarrow.lib as _lib
>> ImportError:
>> dlopen(/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
>> lib.cpython-38-darwin.so, 2): Symbol not found:
>> __ZN5arrow10StopSource5tokenEv
>>  Referenced from:
>> /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
>> lib.cpython-38-darwin.so
>>  Expected in: /usr/local/lib/libarrow.400.dylib
>> in /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
>> lib.cpython-38-darwin.so
> pa
>> Traceback (most recent call last):
>>  File "", line 1, in 
>> NameError: name 'pa' is not defined
>> 
>> On the other hand a Conda installation is not even possible. Does anyone
>> know what’s going on?
>> 
>> Ying

Re: [DISCUSS] Moving the format directory to arrow-format repository

[Andrew Lamb]

I also think manually copying the format .fbs files to arrow-rs is probably
ok for the time being.

Once Arrow gets to the point where many implementations that need
format.fbs live in many different repos, pulling out the format files into
their own repo might be worth reconsidering.

Andrew

On Tue, Apr 27, 2021 at 5:45 PM Wes McKinney  wrote:

> I wouldn't be too excited about this. Here are my thoughts:
>
> 1. Having the format/ directory in apache/arrow be a submodule would be
> cumbersome and error-prone for developers. The only submodules we have
> right now are optional testing dependencies — not having these initialized
> and updated does not result in a broken project, whereas this change would.
> We have a copy of parquet.thrift from apache/parquet-format for similar
> reasons.
>
> 2. So based on #1, we would want to maintain a copy of the format files in
> apache/arrow, even if there were a separate apache/arrow-format repository.
> The format files are slow-moving enough that I don't think it's burdensome
> to mirror these into satellite repositories like arrow/arrow-rs.
>
> On Tue, Apr 27, 2021 at 10:54 AM Neville Dipale 
> wrote:
>
> > Hi Arrow devs,
> >
> > Andy noticed that we carry a copy of the format directory in arrow-rs,
> > which
> > is bound to get outdated in the future.
> >
> > We would like to propose creating an arrow-format repository, similar to
> > parquet-format, so that arrow-rs and other future separate repositories
> > could
> > add this as a submodule.
> >
> > What are your thoughts?
> >
> > Regards
> > Neville
> >
>

[NIGHTLY] Arrow Build Report for Job nightly-2021-04-28-0

[Crossbow]

Arrow Build Report for Job nightly-2021-04-28-0

All tasks: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0

Failed Tasks:
- conda-linux-gcc-py36-arm64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-drone-conda-linux-gcc-py36-arm64
- conda-linux-gcc-py37-arm64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-drone-conda-linux-gcc-py37-arm64
- conda-linux-gcc-py38-arm64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-drone-conda-linux-gcc-py38-arm64
- conda-linux-gcc-py39-arm64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-drone-conda-linux-gcc-py39-arm64
- test-conda-python-3.7-pandas-0.24:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-github-test-conda-python-3.7-pandas-0.24
- test-conda-python-3.7-turbodbc-latest:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-github-test-conda-python-3.7-turbodbc-latest
- test-conda-python-3.7-turbodbc-master:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-github-test-conda-python-3.7-turbodbc-master
- test-conda-python-3.8-jpype:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-github-test-conda-python-3.8-jpype
- test-conda-python-3.9:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-github-test-conda-python-3.9
- test-ubuntu-20.10-docs:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-test-ubuntu-20.10-docs

Succeeded Tasks:
- centos-7-amd64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-github-centos-7-amd64
- centos-8-amd64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-github-centos-8-amd64
- centos-8-arm64:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-travis-centos-8-arm64
- conda-clean:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-clean
- conda-linux-gcc-py36-cpu-r36:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-linux-gcc-py36-cpu-r36
- conda-linux-gcc-py36-cuda:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-linux-gcc-py36-cuda
- conda-linux-gcc-py37-cpu-r40:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-linux-gcc-py37-cpu-r40
- conda-linux-gcc-py37-cuda:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-linux-gcc-py37-cuda
- conda-linux-gcc-py38-cpu:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-linux-gcc-py38-cpu
- conda-linux-gcc-py38-cuda:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-linux-gcc-py38-cuda
- conda-linux-gcc-py39-cpu:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-linux-gcc-py39-cpu
- conda-linux-gcc-py39-cuda:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-linux-gcc-py39-cuda
- conda-osx-arm64-clang-py38:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-osx-arm64-clang-py38
- conda-osx-arm64-clang-py39:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-osx-arm64-clang-py39
- conda-osx-clang-py36-r36:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-osx-clang-py36-r36
- conda-osx-clang-py37-r40:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-osx-clang-py37-r40
- conda-osx-clang-py38:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-osx-clang-py38
- conda-osx-clang-py39:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-osx-clang-py39
- conda-win-vs2017-py36-r36:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-win-vs2017-py36-r36
- conda-win-vs2017-py37-r40:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-win-vs2017-py37-r40
- conda-win-vs2017-py38:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-win-vs2017-py38
- conda-win-vs2017-py39:
  URL: 
https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-04-28-0-azure-conda-win-vs2017-py39
- debian-bullseye-amd64:
  URL: 
https://

Re: [Python] Who has been able to use PyArrow 4.0.0?

[Alessandro Molina]

Are you sure you haven't installed `libarrow` (the CPP one) manually
independently from pyarrow?

In your traceback you have that the symbol has not been found in
"/usr/local/lib/libarrow.400.dylib"

But that smells like an independently installed libarrow, as the libarrow
provided by pyarrow should exist in the pytnon environment (in my case for
example  /usr/local/lib/python3.9/site-packages/pyarrow/libarrow.400.dylib
) I suspect your system installed libarrow is taking precedence over the
one provided by pyarrow and the two might not match.

On Wed, Apr 28, 2021 at 10:05 AM Ying Zhou  wrote:

> Hi,
>
> It turns out that I haven’t been able to use PyArrow 4.0.0 either in Conda
> environments or python venvs. PyArrow does install using pip. However this
> is what I get if I ever want to use it:
>
> >>> import pyarrow as pa
> Traceback (most recent call last):
>   File "", line 1, in 
>   File
> "/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/__init__.py",
> line 63, in 
> import pyarrow.lib as _lib
> ImportError:
> dlopen(/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> lib.cpython-38-darwin.so, 2): Symbol not found:
> __ZN5arrow10StopSource5tokenEv
>   Referenced from:
> /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> lib.cpython-38-darwin.so
>   Expected in: /usr/local/lib/libarrow.400.dylib
>  in /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> lib.cpython-38-darwin.so
> >>> pa
> Traceback (most recent call last):
>   File "", line 1, in 
> NameError: name 'pa' is not defined
>
> On the other hand a Conda installation is not even possible. Does anyone
> know what’s going on?
>
> Ying

Re: [Python] Who has been able to use PyArrow 4.0.0?

[Joris Van den Bossche]

On Wed, 28 Apr 2021 at 10:05, Ying Zhou  wrote:

>
> On the other hand a Conda installation is not even possible. Does anyone
> know what’s going on?
>

For conda installation: the conda packages for pyarrow 4.0 were uploaded
around 1 hour ago, so this should now be possible.

Joris



>
> Ying

Re: [Python] Who has been able to use PyArrow 4.0.0?

[Jorge Cardoso Leitão]

Hi,

I am unable to reproduce it on Mac OS 11.2

```
python3 --version
> Python 3.7.4
python3 -m venv venv
source venv/bin/activate
pip --version
> pip 19.0.3

pip install pyarrow==4.0.0
> downloading [...]/pyarrow-4.0.0-cp37-cp37m-macosx_10_13_x86_64.whl
> ...

pip freeze
> numpy==1.20.2
> pyarrow==4.0.0

python -c "import pyarrow; print(pyarrow.__file__)"
> [...]/venv/lib/python3.7/site-packages/pyarrow/__init__.py
```

Best,
Jorge


On Wed, Apr 28, 2021 at 10:08 AM Ying Zhou  wrote:

> In case you guys wonder I’m on MacOS 10.15.7. Due to my environment being
> pretty dirty I didn’t announce it when my verification attempt failed back
> then.
>
> > On Apr 28, 2021, at 4:04 AM, Ying Zhou  wrote:
> >
> > Hi,
> >
> > It turns out that I haven’t been able to use PyArrow 4.0.0 either in
> Conda environments or python venvs. PyArrow does install using pip. However
> this is what I get if I ever want to use it:
> >
> > >>> import pyarrow as pa
> > Traceback (most recent call last):
> >   File "", line 1, in 
> >   File
> "/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/__init__.py",
> line 63, in 
> > import pyarrow.lib as _lib
> > ImportError:
> dlopen(/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> lib.cpython-38-darwin.so, 2): Symbol not found:
> __ZN5arrow10StopSource5tokenEv
> >   Referenced from:
> /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> lib.cpython-38-darwin.so
> >   Expected in: /usr/local/lib/libarrow.400.dylib
> >  in /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/
> lib.cpython-38-darwin.so
> > >>> pa
> > Traceback (most recent call last):
> >   File "", line 1, in 
> > NameError: name 'pa' is not defined
> >
> > On the other hand a Conda installation is not even possible. Does anyone
> know what’s going on?
> >
> > Ying
>
>

Re: [Python] Who has been able to use PyArrow 4.0.0?

[Ying Zhou]

In case you guys wonder I’m on MacOS 10.15.7. Due to my environment being 
pretty dirty I didn’t announce it when my verification attempt failed back then.

> On Apr 28, 2021, at 4:04 AM, Ying Zhou  wrote:
> 
> Hi,
> 
> It turns out that I haven’t been able to use PyArrow 4.0.0 either in Conda 
> environments or python venvs. PyArrow does install using pip. However this is 
> what I get if I ever want to use it:
> 
> >>> import pyarrow as pa
> Traceback (most recent call last):
>   File "", line 1, in 
>   File 
> "/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/__init__.py",
>  line 63, in 
> import pyarrow.lib as _lib
> ImportError: 
> dlopen(/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so,
>  2): Symbol not found: __ZN5arrow10StopSource5tokenEv
>   Referenced from: 
> /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so
>   Expected in: /usr/local/lib/libarrow.400.dylib
>  in 
> /Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so
> >>> pa
> Traceback (most recent call last):
>   File "", line 1, in 
> NameError: name 'pa' is not defined
> 
> On the other hand a Conda installation is not even possible. Does anyone know 
> what’s going on?
> 
> Ying 

[Python] Who has been able to use PyArrow 4.0.0?

[Ying Zhou]

Hi,

It turns out that I haven’t been able to use PyArrow 4.0.0 either in Conda 
environments or python venvs. PyArrow does install using pip. However this is 
what I get if I ever want to use it:

>>> import pyarrow as pa
Traceback (most recent call last):
  File "", line 1, in 
  File 
"/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/__init__.py", 
line 63, in 
import pyarrow.lib as _lib
ImportError: 
dlopen(/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so,
 2): Symbol not found: __ZN5arrow10StopSource5tokenEv
  Referenced from: 
/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so
  Expected in: /usr/local/lib/libarrow.400.dylib
 in 
/Users/karlkatzen/anaconda3/lib/python3.8/site-packages/pyarrow/lib.cpython-38-darwin.so
>>> pa
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'pa' is not defined

On the other hand a Conda installation is not even possible. Does anyone know 
what’s going on?

Ying 

Including JS patch in 4.0.1 if released

[Adam Lippai]

Hi,

I'd want to propose including https://github.com/apache/arrow/pull/10181 in
4.0.1 release, if it happens.
While the issue is barely a bugfix, it's still a minor regression since
3.0.0. It happens only in special circumstances, eg. using Rollup bundler
or restricting eval() for security reasons. It doesn't warrant a new
release on it's own as it affects a marginal fraction of the consumers
only.

The change shouldn't affect the API or the required NodeJS versions.

Best regards,
Adam Lippai