[jira] [Commented] (ARROW-2335) [Go] Remove extra directory nesting from go/ directory
[
https://issues.apache.org/jira/browse/ARROW-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409001#comment-16409001
]
Stuart Carnie commented on ARROW-2335:
--
That makes sense to me
> [Go] Remove extra directory nesting from go/ directory
> --
>
> Key: ARROW-2335
> URL: https://issues.apache.org/jira/browse/ARROW-2335
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Go
>Reporter: Wes McKinney
>Assignee: Stuart Carnie
>Priority: Major
> Fix For: 0.10.0
>
>
> I noticed this after merging. I am not sure we need the {{arrow/go/arrow}}
> directory structure if simply {{arrow/go}} would suffice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2335) [Go] Remove extra directory nesting from go/ directory
[
https://issues.apache.org/jira/browse/ARROW-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408945#comment-16408945
]
Wes McKinney commented on ARROW-2335:
-
Got it. It may make sense, then, to promote some files out of the {{arrow}}
directory (like the README) in the event that you decide to leave it like that.
It seemed odd to me to have the {{go}} directory not have at least a README
(that's the only thing that Vitess has in its go directory
https://github.com/vitessio/vitess/tree/master/go)
> [Go] Remove extra directory nesting from go/ directory
> --
>
> Key: ARROW-2335
> URL: https://issues.apache.org/jira/browse/ARROW-2335
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Go
>Reporter: Wes McKinney
>Assignee: Stuart Carnie
>Priority: Major
> Fix For: 0.10.0
>
>
> I noticed this after merging. I am not sure we need the {{arrow/go/arrow}}
> directory structure if simply {{arrow/go}} would suffice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-2340) [Website] Add blog post about Go codebase donation
[ https://issues.apache.org/jira/browse/ARROW-2340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-2340: -- Labels: pull-request-available (was: ) > [Website] Add blog post about Go codebase donation > -- > > Key: ARROW-2340 > URL: https://issues.apache.org/jira/browse/ARROW-2340 > Project: Apache Arrow > Issue Type: Improvement > Components: Website >Reporter: Wes McKinney >Assignee: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > Adding blog post from the Arrow PMC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2340) [Website] Add blog post about Go codebase donation
Wes McKinney created ARROW-2340: --- Summary: [Website] Add blog post about Go codebase donation Key: ARROW-2340 URL: https://issues.apache.org/jira/browse/ARROW-2340 Project: Apache Arrow Issue Type: Improvement Components: Website Reporter: Wes McKinney Assignee: Wes McKinney Fix For: 0.10.0 Adding blog post from the Arrow PMC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2336) [Website] Blog post for 0.9.0 release
[ https://issues.apache.org/jira/browse/ARROW-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-2336: -- Labels: pull-request-available (was: ) > [Website] Blog post for 0.9.0 release > - > > Key: ARROW-2336 > URL: https://issues.apache.org/jira/browse/ARROW-2336 > Project: Apache Arrow > Issue Type: Improvement > Components: Website >Reporter: Wes McKinney >Assignee: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Release highlights blog post as in past major releases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2336) [Website] Blog post for 0.9.0 release
[ https://issues.apache.org/jira/browse/ARROW-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408926#comment-16408926 ] ASF GitHub Bot commented on ARROW-2336: --- wesm opened a new pull request #1776: ARROW-2336: [Website] Add 0.9.0 release blog post URL: https://github.com/apache/arrow/pull/1776 Short and sweet. I know it's late, but if anyone would like to add anything please push directly to this branch or post a commit for me to cherry-pick This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Website] Blog post for 0.9.0 release > - > > Key: ARROW-2336 > URL: https://issues.apache.org/jira/browse/ARROW-2336 > Project: Apache Arrow > Issue Type: Improvement > Components: Website >Reporter: Wes McKinney >Assignee: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > > Release highlights blog post as in past major releases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-640) [Python] Arrow scalar values should have a sensible __hash__ and comparison
[
https://issues.apache.org/jira/browse/ARROW-640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408924#comment-16408924
]
ASF GitHub Bot commented on ARROW-640:
--
AlexHagerman commented on a change in pull request #1765: ARROW-640: [Python]
Implement __hash__ and equality for Array scalar values Arrow scalar values
URL: https://github.com/apache/arrow/pull/1765#discussion_r176290582
##
File path: python/pyarrow/scalar.pxi
##
@@ -73,6 +73,12 @@ cdef class ArrayValue(Scalar):
raise NotImplementedError(
"Cannot compare Arrow values that don't support as_py()")
+def __hash__(self):
Review comment:
Ticket created https://issues.apache.org/jira/browse/ARROW-2339. Anything
else you would suggest for this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Arrow scalar values should have a sensible __hash__ and comparison
> ---
>
> Key: ARROW-640
> URL: https://issues.apache.org/jira/browse/ARROW-640
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Miki Tebeka
>Assignee: Alex Hagerman
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> {noformat}
> In [86]: arr = pa.from_pylist([1, 1, 1, 2])
> In [87]: set(arr)
> Out[87]: {1, 2, 1, 1}
> In [88]: arr[0] == arr[1]
> Out[88]: False
> In [89]: arr
> Out[89]:
>
> [
> 1,
> 1,
> 1,
> 2
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-2339) [Python] Add a fast path for int hashing
Alex Hagerman created ARROW-2339: Summary: [Python] Add a fast path for int hashing Key: ARROW-2339 URL: https://issues.apache.org/jira/browse/ARROW-2339 Project: Apache Arrow Issue Type: Improvement Components: Python Reporter: Alex Hagerman Assignee: Alex Hagerman Fix For: 0.10.0 Create a __hash__ fast path for Int scalars that avoids using as_py(). https://issues.apache.org/jira/browse/ARROW-640 [https://github.com/apache/arrow/pull/1765/files/4497b69db8039cfeaa7a25f593f3a3e6c7984604] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2335) [Go] Remove extra directory nesting from go/ directory
[
https://issues.apache.org/jira/browse/ARROW-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408665#comment-16408665
]
Stuart Carnie commented on ARROW-2335:
--
[Damien Gryski|https://twitter.com/dgryski/status/976565595209519104] has
started a poll to see what other Go devs think, as there isn't a good precedent
for Go residing in multi-language repos.
For some background, the current reason for using the {{.../go/arrow}} path is
so the Go package name of files in the {{arrow}} directory match. If we move
the files back to {{arrow/go}}, the directory of the files will be {{go}} and
the package name {{arrow}}, which is not the norm.
> [Go] Remove extra directory nesting from go/ directory
> --
>
> Key: ARROW-2335
> URL: https://issues.apache.org/jira/browse/ARROW-2335
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Go
>Reporter: Wes McKinney
>Assignee: Stuart Carnie
>Priority: Major
> Fix For: 0.10.0
>
>
> I noticed this after merging. I am not sure we need the {{arrow/go/arrow}}
> directory structure if simply {{arrow/go}} would suffice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (ARROW-2335) [Go] Remove extra directory nesting from go/ directory
[
https://issues.apache.org/jira/browse/ARROW-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408665#comment-16408665
]
Stuart Carnie edited comment on ARROW-2335 at 3/21/18 10:10 PM:
[Damien Gryski|https://twitter.com/dgryski/status/976565595209519104] has
started a poll to see what other Go devs think, as there isn't a good precedent
for Go residing in multi-language repos. I still don't mind moving it back, but
I am interested in the results.
For some background, the current reason for using the {{.../go/arrow}} path is
so the Go package name of files in the {{arrow}} directory match. If we move
the files back to {{arrow/go}}, the directory of the files will be {{go}} and
the package name {{arrow}}, which is not the norm.
was (Author: stuartcarnie):
[Damien Gryski|https://twitter.com/dgryski/status/976565595209519104] has
started a poll to see what other Go devs think, as there isn't a good precedent
for Go residing in multi-language repos.
For some background, the current reason for using the {{.../go/arrow}} path is
so the Go package name of files in the {{arrow}} directory match. If we move
the files back to {{arrow/go}}, the directory of the files will be {{go}} and
the package name {{arrow}}, which is not the norm.
> [Go] Remove extra directory nesting from go/ directory
> --
>
> Key: ARROW-2335
> URL: https://issues.apache.org/jira/browse/ARROW-2335
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Go
>Reporter: Wes McKinney
>Assignee: Stuart Carnie
>Priority: Major
> Fix For: 0.10.0
>
>
> I noticed this after merging. I am not sure we need the {{arrow/go/arrow}}
> directory structure if simply {{arrow/go}} would suffice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-2337) [Scripts] Windows release verification script should use boost DSOs instead of static linkage
[ https://issues.apache.org/jira/browse/ARROW-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phillip Cloud updated ARROW-2337: - Component/s: Packaging > [Scripts] Windows release verification script should use boost DSOs instead > of static linkage > - > > Key: ARROW-2337 > URL: https://issues.apache.org/jira/browse/ARROW-2337 > Project: Apache Arrow > Issue Type: Task > Components: Packaging >Affects Versions: 0.9.0 >Reporter: Phillip Cloud >Assignee: Phillip Cloud >Priority: Major > Fix For: 0.10.0 > > > Fix up shortly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2338) [Scripts] Windows release verification script should create a conda environment
Phillip Cloud created ARROW-2338: Summary: [Scripts] Windows release verification script should create a conda environment Key: ARROW-2338 URL: https://issues.apache.org/jira/browse/ARROW-2338 Project: Apache Arrow Issue Type: Task Components: Packaging Affects Versions: 0.9.0 Reporter: Phillip Cloud Assignee: Phillip Cloud Fix For: 0.10.0 It should also download the source. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2337) [Scripts] Windows release verification script should use boost DSOs instead of static linkage
Phillip Cloud created ARROW-2337: Summary: [Scripts] Windows release verification script should use boost DSOs instead of static linkage Key: ARROW-2337 URL: https://issues.apache.org/jira/browse/ARROW-2337 Project: Apache Arrow Issue Type: Task Affects Versions: 0.9.0 Reporter: Phillip Cloud Assignee: Phillip Cloud Fix For: 0.10.0 Fix up shortly -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2335) [Go] Remove extra directory nesting from go/ directory
[
https://issues.apache.org/jira/browse/ARROW-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408551#comment-16408551
]
Stuart Carnie commented on ARROW-2335:
--
github.com/google/flatbuffers/tree/master/go is arranged this way, so I don't
have a problem with that.
> [Go] Remove extra directory nesting from go/ directory
> --
>
> Key: ARROW-2335
> URL: https://issues.apache.org/jira/browse/ARROW-2335
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Go
>Reporter: Wes McKinney
>Assignee: Stuart Carnie
>Priority: Major
> Fix For: 0.10.0
>
>
> I noticed this after merging. I am not sure we need the {{arrow/go/arrow}}
> directory structure if simply {{arrow/go}} would suffice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-2335) [Go] Remove extra directory nesting from go/ directory
[
https://issues.apache.org/jira/browse/ARROW-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stuart Carnie reassigned ARROW-2335:
Assignee: Stuart Carnie
> [Go] Remove extra directory nesting from go/ directory
> --
>
> Key: ARROW-2335
> URL: https://issues.apache.org/jira/browse/ARROW-2335
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Go
>Reporter: Wes McKinney
>Assignee: Stuart Carnie
>Priority: Major
> Fix For: 0.10.0
>
>
> I noticed this after merging. I am not sure we need the {{arrow/go/arrow}}
> directory structure if simply {{arrow/go}} would suffice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-2336) [Website] Blog post for 0.9.0 release
Wes McKinney created ARROW-2336: --- Summary: [Website] Blog post for 0.9.0 release Key: ARROW-2336 URL: https://issues.apache.org/jira/browse/ARROW-2336 Project: Apache Arrow Issue Type: Improvement Components: Website Reporter: Wes McKinney Assignee: Wes McKinney Fix For: 0.9.0 Release highlights blog post as in past major releases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408508#comment-16408508 ] ASF GitHub Bot commented on ARROW-2299: --- wesm commented on issue #1739: ARROW-2299: [Go] Import Go arrow implementation from influxdata/arrow URL: https://github.com/apache/arrow/pull/1739#issuecomment-375080481 Great, thanks, I added you to the contributor list (so you can assign yourself JIRAs) and assigned this one to you This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Assignee: Stuart Carnie >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2333) [Python] boost bundling fails in setup.py
[
https://issues.apache.org/jira/browse/ARROW-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408506#comment-16408506
]
ASF GitHub Bot commented on ARROW-2333:
---
cpcloud commented on issue #1773: ARROW-2333: [Python] Fix bundling boost with
default namespace
URL: https://github.com/apache/arrow/pull/1773#issuecomment-375080279
@wesm Not sure, I'll look.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] boost bundling fails in setup.py
> -
>
> Key: ARROW-2333
> URL: https://issues.apache.org/jira/browse/ARROW-2333
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.9.0
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> {code}
> [...]
>File "setup.py", line 88, in run
> self._run_cmake()
>File "setup.py", line 280, in _run_cmake
> "{}_filesystem".format(self.boost_namespace))
>File "setup.py", line 386, in move_shared_libs
> _move_shared_libs_unix(build_prefix, build_lib, lib_name)
>File "setup.py", line 408, in _move_shared_libs_unix
> ' in ' + build_prefix)
> Exception: Could not find library:libNone_filesystem.so in
> release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney reassigned ARROW-2299: --- Assignee: Stuart Carnie > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Assignee: Stuart Carnie >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408504#comment-16408504 ] ASF GitHub Bot commented on ARROW-2299: --- stuartcarnie commented on issue #1739: ARROW-2299: [Go] Import Go arrow implementation from influxdata/arrow URL: https://github.com/apache/arrow/pull/1739#issuecomment-375079827 @wesm done – username is same as GH This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408500#comment-16408500 ] ASF GitHub Bot commented on ARROW-2299: --- wesm commented on issue #1739: ARROW-2299: [Go] Import Go arrow implementation from influxdata/arrow URL: https://github.com/apache/arrow/pull/1739#issuecomment-375078570 Please let me know if you have opinions on https://issues.apache.org/jira/browse/ARROW-2335 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2335) [Go] Remove extra directory nesting from go/ directory
Wes McKinney created ARROW-2335:
---
Summary: [Go] Remove extra directory nesting from go/ directory
Key: ARROW-2335
URL: https://issues.apache.org/jira/browse/ARROW-2335
Project: Apache Arrow
Issue Type: Improvement
Components: Go
Reporter: Wes McKinney
Fix For: 0.10.0
I noticed this after merging. I am not sure we need the {{arrow/go/arrow}}
directory structure if simply {{arrow/go}} would suffice
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408497#comment-16408497 ] ASF GitHub Bot commented on ARROW-2299: --- wesm commented on issue #1739: ARROW-2299: [Go] Import Go arrow implementation from influxdata/arrow URL: https://github.com/apache/arrow/pull/1739#issuecomment-375077863 thanks @stuartcarnie! Would you mind creating an ID on JIRA so I can assign this to you https://issues.apache.org/jira/browse/ARROW-2299? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-2299. - Resolution: Fixed Issue resolved by pull request 1739 [https://github.com/apache/arrow/pull/1739] > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408466#comment-16408466 ] ASF GitHub Bot commented on ARROW-2299: --- wesm commented on issue #1739: ARROW-2299: [Go] Import Go arrow implementation from influxdata/arrow URL: https://github.com/apache/arrow/pull/1739#issuecomment-375068231 +1. Merging on green build This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] cpcloud commented on issue #23: Build wheels for 0.9.0
cpcloud commented on issue #23: Build wheels for 0.9.0 URL: https://github.com/apache/arrow-dist/pull/23#issuecomment-375068142 @xhochy @kou I need some help with this. There's a ton of stuff here that I'm unfamiliar with especially the details of yum and apt-get packages. I don't have access to a Mac either. Can you take a look at what I've done here and help debug? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408463#comment-16408463 ] ASF GitHub Bot commented on ARROW-2299: --- wesm commented on issue #1739: DONOTMERGE ARROW-2299: [Go] Import Go arrow implementation from influxdata/arrow URL: https://github.com/apache/arrow/pull/1739#issuecomment-375067024 I'm working on fixing the RAT checks (and also the fact that the release audit wasn't run in this CI build -- since it is a part of the C++ build at the moment), and then will merge this This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2182) [Python] ASV benchmark setup does not account for C++ library changing
[ https://issues.apache.org/jira/browse/ARROW-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408452#comment-16408452 ] ASF GitHub Bot commented on ARROW-2182: --- wesm commented on issue #1775: [WIP] ARROW-2182: [Python] Build C++ libraries in benchmarks build step URL: https://github.com/apache/arrow/pull/1775#issuecomment-375065316 I'll take this for a spin when I can, tomorrow or Friday most likely This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Python] ASV benchmark setup does not account for C++ library changing > -- > > Key: ARROW-2182 > URL: https://issues.apache.org/jira/browse/ARROW-2182 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > See https://github.com/apache/arrow/blob/master/python/README-benchmarks.md > Perhaps we could create a helper script that will run all the > currently-defined benchmarks for a specific commit, and ensure that we are > running against pristine, up-to-date release builds of Arrow (and any other > dependencies, like parquet-cpp) at that commit? > cc [~pitrou] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2299) [Go] Go language implementation
[ https://issues.apache.org/jira/browse/ARROW-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408453#comment-16408453 ] ASF GitHub Bot commented on ARROW-2299: --- wesm commented on issue #1739: DONOTMERGE ARROW-2299: [Go] Import Go arrow implementation from influxdata/arrow URL: https://github.com/apache/arrow/pull/1739#issuecomment-372526683 Thanks @stuartcarnie! Steps from here: - [x] PMC vote to accept code donation - [x] Receive software grant from InfluxData, Inc. - [x] IP Clearance vote on Incubator general mailing list See http://incubator.apache.org/ip-clearance/arrow-go-library.html for status on the IP Clearance process This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Go] Go language implementation > --- > > Key: ARROW-2299 > URL: https://issues.apache.org/jira/browse/ARROW-2299 > Project: Apache Arrow > Issue Type: New Feature > Components: Go >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2182) [Python] ASV benchmark setup does not account for C++ library changing
[ https://issues.apache.org/jira/browse/ARROW-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408438#comment-16408438 ] ASF GitHub Bot commented on ARROW-2182: --- pitrou commented on issue #1775: [WIP] ARROW-2182: [Python] Build C++ libraries in benchmarks build step URL: https://github.com/apache/arrow/pull/1775#issuecomment-375062304 Note this doesn't handle parquet-cpp. Currently the benchmarks don't exercise Parquet at all. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Python] ASV benchmark setup does not account for C++ library changing > -- > > Key: ARROW-2182 > URL: https://issues.apache.org/jira/browse/ARROW-2182 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > See https://github.com/apache/arrow/blob/master/python/README-benchmarks.md > Perhaps we could create a helper script that will run all the > currently-defined benchmarks for a specific commit, and ensure that we are > running against pristine, up-to-date release builds of Arrow (and any other > dependencies, like parquet-cpp) at that commit? > cc [~pitrou] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2182) [Python] ASV benchmark setup does not account for C++ library changing
[ https://issues.apache.org/jira/browse/ARROW-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408435#comment-16408435 ] ASF GitHub Bot commented on ARROW-2182: --- pitrou commented on issue #1775: [WIP] ARROW-2182: [Python] Build C++ libraries in benchmarks build step URL: https://github.com/apache/arrow/pull/1775#issuecomment-375061404 @wesm it would be nice if you could give this a try. It seems to work here (i.e. `asv run` rebuilds the arrow C++ libraries on the given changeset) but perhaps it doesn't on other setups. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Python] ASV benchmark setup does not account for C++ library changing > -- > > Key: ARROW-2182 > URL: https://issues.apache.org/jira/browse/ARROW-2182 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > See https://github.com/apache/arrow/blob/master/python/README-benchmarks.md > Perhaps we could create a helper script that will run all the > currently-defined benchmarks for a specific commit, and ensure that we are > running against pristine, up-to-date release builds of Arrow (and any other > dependencies, like parquet-cpp) at that commit? > cc [~pitrou] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2182) [Python] ASV benchmark setup does not account for C++ library changing
[ https://issues.apache.org/jira/browse/ARROW-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-2182: -- Labels: pull-request-available (was: ) > [Python] ASV benchmark setup does not account for C++ library changing > -- > > Key: ARROW-2182 > URL: https://issues.apache.org/jira/browse/ARROW-2182 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > See https://github.com/apache/arrow/blob/master/python/README-benchmarks.md > Perhaps we could create a helper script that will run all the > currently-defined benchmarks for a specific commit, and ensure that we are > running against pristine, up-to-date release builds of Arrow (and any other > dependencies, like parquet-cpp) at that commit? > cc [~pitrou] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2182) [Python] ASV benchmark setup does not account for C++ library changing
[ https://issues.apache.org/jira/browse/ARROW-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408431#comment-16408431 ] ASF GitHub Bot commented on ARROW-2182: --- pitrou opened a new pull request #1775: [WIP] ARROW-2182: [Python] Build C++ libraries in benchmarks build step URL: https://github.com/apache/arrow/pull/1775 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Python] ASV benchmark setup does not account for C++ library changing > -- > > Key: ARROW-2182 > URL: https://issues.apache.org/jira/browse/ARROW-2182 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Reporter: Wes McKinney >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > See https://github.com/apache/arrow/blob/master/python/README-benchmarks.md > Perhaps we could create a helper script that will run all the > currently-defined benchmarks for a specific commit, and ensure that we are > running against pristine, up-to-date release builds of Arrow (and any other > dependencies, like parquet-cpp) at that commit? > cc [~pitrou] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2182) [Python] ASV benchmark setup does not account for C++ library changing
[ https://issues.apache.org/jira/browse/ARROW-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408430#comment-16408430 ] Antoine Pitrou commented on ARROW-2182: --- Given the current absence of response on the ASV PRs, I think we're going to have to use our own fork. > [Python] ASV benchmark setup does not account for C++ library changing > -- > > Key: ARROW-2182 > URL: https://issues.apache.org/jira/browse/ARROW-2182 > Project: Apache Arrow > Issue Type: Bug > Components: Python >Reporter: Wes McKinney >Priority: Major > Fix For: 0.10.0 > > > See https://github.com/apache/arrow/blob/master/python/README-benchmarks.md > Perhaps we could create a helper script that will run all the > currently-defined benchmarks for a specific commit, and ensure that we are > running against pristine, up-to-date release builds of Arrow (and any other > dependencies, like parquet-cpp) at that commit? > cc [~pitrou] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2333) [Python] boost bundling fails in setup.py
[
https://issues.apache.org/jira/browse/ARROW-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408428#comment-16408428
]
ASF GitHub Bot commented on ARROW-2333:
---
wesm commented on issue #1773: ARROW-2333: [Python] Fix bundling boost with
default namespace
URL: https://github.com/apache/arrow/pull/1773#issuecomment-375059165
I see. @cpcloud does this impact the packaging builds?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] boost bundling fails in setup.py
> -
>
> Key: ARROW-2333
> URL: https://issues.apache.org/jira/browse/ARROW-2333
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.9.0
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> {code}
> [...]
>File "setup.py", line 88, in run
> self._run_cmake()
>File "setup.py", line 280, in _run_cmake
> "{}_filesystem".format(self.boost_namespace))
>File "setup.py", line 386, in move_shared_libs
> _move_shared_libs_unix(build_prefix, build_lib, lib_name)
>File "setup.py", line 408, in _move_shared_libs_unix
> ' in ' + build_prefix)
> Exception: Could not find library:libNone_filesystem.so in
> release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2333) [Python] boost bundling fails in setup.py
[
https://issues.apache.org/jira/browse/ARROW-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408429#comment-16408429
]
ASF GitHub Bot commented on ARROW-2333:
---
wesm commented on issue #1773: ARROW-2333: [Python] Fix bundling boost with
default namespace
URL: https://github.com/apache/arrow/pull/1773#issuecomment-375059348
Probably goes without saying that finding out about packaging problems
post-release shows that something is wrong with our development process
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] boost bundling fails in setup.py
> -
>
> Key: ARROW-2333
> URL: https://issues.apache.org/jira/browse/ARROW-2333
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.9.0
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> {code}
> [...]
>File "setup.py", line 88, in run
> self._run_cmake()
>File "setup.py", line 280, in _run_cmake
> "{}_filesystem".format(self.boost_namespace))
>File "setup.py", line 386, in move_shared_libs
> _move_shared_libs_unix(build_prefix, build_lib, lib_name)
>File "setup.py", line 408, in _move_shared_libs_unix
> ' in ' + build_prefix)
> Exception: Could not find library:libNone_filesystem.so in
> release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2333) [Python] boost bundling fails in setup.py
[
https://issues.apache.org/jira/browse/ARROW-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408426#comment-16408426
]
ASF GitHub Bot commented on ARROW-2333:
---
pitrou commented on issue #1773: ARROW-2333: [Python] Fix bundling boost with
default namespace
URL: https://github.com/apache/arrow/pull/1773#issuecomment-375058333
No, I think it's simply `PYARROW_BUNDLE_ARROW_CPP=1`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] boost bundling fails in setup.py
> -
>
> Key: ARROW-2333
> URL: https://issues.apache.org/jira/browse/ARROW-2333
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.9.0
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> {code}
> [...]
>File "setup.py", line 88, in run
> self._run_cmake()
>File "setup.py", line 280, in _run_cmake
> "{}_filesystem".format(self.boost_namespace))
>File "setup.py", line 386, in move_shared_libs
> _move_shared_libs_unix(build_prefix, build_lib, lib_name)
>File "setup.py", line 408, in _move_shared_libs_unix
> ' in ' + build_prefix)
> Exception: Could not find library:libNone_filesystem.so in
> release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (ARROW-2332) [Python] Provide API for reading multiple Feather files
[ https://issues.apache.org/jira/browse/ARROW-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408384#comment-16408384 ] Dhruv Madeka edited comment on ARROW-2332 at 3/21/18 6:32 PM: -- As mentioned in the Github issue - IMHO this requires a few steps. * In `FeatherReader`, separate the extraction of the table from the call to convert it `to_pandas` * Create a `FeatherDataset` class which takes a list of feather files and creates a table for each one * Validate that the schemas for each of the files match, the `validate_schema` for the `ParquetDataset` seems to work. Maybe we can create an abstract dataset class and inherit from there * call `concat_tables` on the extract tables and return them as a pandas dataframe was (Author: madeka): As mentioned in the Github issue - IMHO this requires a few steps. * In `FeatherReader`, separate the extraction of the table from the call to convert it `to_pandas` * Create a `FeatherDataset` class which takes a list of featherfiles and creates a table for each one * Validate that the schemas for each of the files match, the `validate_schema` for the `ParquetDataset` seems to work. Maybe we can create an abstract dataset class and inherit from there > [Python] Provide API for reading multiple Feather files > --- > > Key: ARROW-2332 > URL: https://issues.apache.org/jira/browse/ARROW-2332 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Reporter: Wes McKinney >Priority: Major > Fix For: 0.10.0 > > > See discussion in > https://github.com/wesm/feather/issues/273#issuecomment-374093374 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2332) [Python] Provide API for reading multiple Feather files
[ https://issues.apache.org/jira/browse/ARROW-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408384#comment-16408384 ] Dhruv Madeka commented on ARROW-2332: - As mentioned in the Github issue - IMHO this requires a few steps. * In `FeatherReader`, separate the extraction of the table from the call to convert it `to_pandas` * Create a `FeatherDataset` class which takes a list of featherfiles and creates a table for each one * Validate that the schemas for each of the files match, the `validate_schema` for the `ParquetDataset` seems to work. Maybe we can create an abstract dataset class and inherit from there > [Python] Provide API for reading multiple Feather files > --- > > Key: ARROW-2332 > URL: https://issues.apache.org/jira/browse/ARROW-2332 > Project: Apache Arrow > Issue Type: Improvement > Components: Python >Reporter: Wes McKinney >Priority: Major > Fix For: 0.10.0 > > > See discussion in > https://github.com/wesm/feather/issues/273#issuecomment-374093374 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2334) [C++] Update boost to 1.66.0
[ https://issues.apache.org/jira/browse/ARROW-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408381#comment-16408381 ] ASF GitHub Bot commented on ARROW-2334: --- cpcloud commented on issue #1774: ARROW-2334: [C++] Update boost to 1.66.0 URL: https://github.com/apache/arrow/pull/1774#issuecomment-375049181 Yes, I was trying to make the conda-forge builds and the wheels' boost versions match each other. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [C++] Update boost to 1.66.0 > > > Key: ARROW-2334 > URL: https://issues.apache.org/jira/browse/ARROW-2334 > Project: Apache Arrow > Issue Type: Task >Affects Versions: 0.9.0 >Reporter: Phillip Cloud >Assignee: Phillip Cloud >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2334) [C++] Update boost to 1.66.0
[ https://issues.apache.org/jira/browse/ARROW-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408353#comment-16408353 ] ASF GitHub Bot commented on ARROW-2334: --- wesm commented on issue #1774: ARROW-2334: [C++] Update boost to 1.66.0 URL: https://github.com/apache/arrow/pull/1774#issuecomment-375045011 Is this needed for packaging? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [C++] Update boost to 1.66.0 > > > Key: ARROW-2334 > URL: https://issues.apache.org/jira/browse/ARROW-2334 > Project: Apache Arrow > Issue Type: Task >Affects Versions: 0.9.0 >Reporter: Phillip Cloud >Assignee: Phillip Cloud >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2334) [C++] Update boost to 1.66.0
[ https://issues.apache.org/jira/browse/ARROW-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated ARROW-2334: -- Labels: pull-request-available (was: ) > [C++] Update boost to 1.66.0 > > > Key: ARROW-2334 > URL: https://issues.apache.org/jira/browse/ARROW-2334 > Project: Apache Arrow > Issue Type: Task >Affects Versions: 0.9.0 >Reporter: Phillip Cloud >Assignee: Phillip Cloud >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ARROW-2333) [Python] boost bundling fails in setup.py
[
https://issues.apache.org/jira/browse/ARROW-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney reassigned ARROW-2333:
---
Assignee: Antoine Pitrou
> [Python] boost bundling fails in setup.py
> -
>
> Key: ARROW-2333
> URL: https://issues.apache.org/jira/browse/ARROW-2333
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.9.0
>Reporter: Antoine Pitrou
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> {code}
> [...]
>File "setup.py", line 88, in run
> self._run_cmake()
>File "setup.py", line 280, in _run_cmake
> "{}_filesystem".format(self.boost_namespace))
>File "setup.py", line 386, in move_shared_libs
> _move_shared_libs_unix(build_prefix, build_lib, lib_name)
>File "setup.py", line 408, in _move_shared_libs_unix
> ' in ' + build_prefix)
> Exception: Could not find library:libNone_filesystem.so in
> release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-2333) [Python] boost bundling fails in setup.py
[
https://issues.apache.org/jira/browse/ARROW-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-2333:
Fix Version/s: 0.10.0
> [Python] boost bundling fails in setup.py
> -
>
> Key: ARROW-2333
> URL: https://issues.apache.org/jira/browse/ARROW-2333
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.9.0
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> {code}
> [...]
>File "setup.py", line 88, in run
> self._run_cmake()
>File "setup.py", line 280, in _run_cmake
> "{}_filesystem".format(self.boost_namespace))
>File "setup.py", line 386, in move_shared_libs
> _move_shared_libs_unix(build_prefix, build_lib, lib_name)
>File "setup.py", line 408, in _move_shared_libs_unix
> ' in ' + build_prefix)
> Exception: Could not find library:libNone_filesystem.so in
> release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-2334) [C++] Update boost to 1.66.0
Phillip Cloud created ARROW-2334: Summary: [C++] Update boost to 1.66.0 Key: ARROW-2334 URL: https://issues.apache.org/jira/browse/ARROW-2334 Project: Apache Arrow Issue Type: Task Affects Versions: 0.9.0 Reporter: Phillip Cloud Assignee: Phillip Cloud Fix For: 0.9.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2333) [Python] boost bundling fails in setup.py
[
https://issues.apache.org/jira/browse/ARROW-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408215#comment-16408215
]
ASF GitHub Bot commented on ARROW-2333:
---
pitrou opened a new pull request #1773: ARROW-2333: [Python] Fix bundling boost
with default namespace
URL: https://github.com/apache/arrow/pull/1773
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] boost bundling fails in setup.py
> -
>
> Key: ARROW-2333
> URL: https://issues.apache.org/jira/browse/ARROW-2333
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.9.0
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
>
> {code}
> [...]
>File "setup.py", line 88, in run
> self._run_cmake()
>File "setup.py", line 280, in _run_cmake
> "{}_filesystem".format(self.boost_namespace))
>File "setup.py", line 386, in move_shared_libs
> _move_shared_libs_unix(build_prefix, build_lib, lib_name)
>File "setup.py", line 408, in _move_shared_libs_unix
> ' in ' + build_prefix)
> Exception: Could not find library:libNone_filesystem.so in
> release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-2333) [Python] boost bundling fails in setup.py
[
https://issues.apache.org/jira/browse/ARROW-2333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-2333:
--
Labels: pull-request-available (was: )
> [Python] boost bundling fails in setup.py
> -
>
> Key: ARROW-2333
> URL: https://issues.apache.org/jira/browse/ARROW-2333
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Affects Versions: 0.9.0
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
>
> {code}
> [...]
>File "setup.py", line 88, in run
> self._run_cmake()
>File "setup.py", line 280, in _run_cmake
> "{}_filesystem".format(self.boost_namespace))
>File "setup.py", line 386, in move_shared_libs
> _move_shared_libs_unix(build_prefix, build_lib, lib_name)
>File "setup.py", line 408, in _move_shared_libs_unix
> ' in ' + build_prefix)
> Exception: Could not find library:libNone_filesystem.so in
> release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-2333) [Python] boost bundling fails in setup.py
Antoine Pitrou created ARROW-2333:
-
Summary: [Python] boost bundling fails in setup.py
Key: ARROW-2333
URL: https://issues.apache.org/jira/browse/ARROW-2333
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.9.0
Reporter: Antoine Pitrou
{code}
[...]
File "setup.py", line 88, in run
self._run_cmake()
File "setup.py", line 280, in _run_cmake
"{}_filesystem".format(self.boost_namespace))
File "setup.py", line 386, in move_shared_libs
_move_shared_libs_unix(build_prefix, build_lib, lib_name)
File "setup.py", line 408, in _move_shared_libs_unix
' in ' + build_prefix)
Exception: Could not find library:libNone_filesystem.so in release
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-2332) [Python] Provide API for reading multiple Feather files
Wes McKinney created ARROW-2332: --- Summary: [Python] Provide API for reading multiple Feather files Key: ARROW-2332 URL: https://issues.apache.org/jira/browse/ARROW-2332 Project: Apache Arrow Issue Type: Improvement Components: Python Reporter: Wes McKinney Fix For: 0.10.0 See discussion in https://github.com/wesm/feather/issues/273#issuecomment-374093374 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] wesm commented on issue #23: Build wheels for 0.9.0
wesm commented on issue #23: Build wheels for 0.9.0 URL: https://github.com/apache/arrow-dist/pull/23#issuecomment-374986924 Conceivably there would be the possibility of ABI conflicts, but that would be true if we were statically linking also. We could go the namespaced Boost route like in the manylinux1 wheels, but that's quite a bit of effort This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (ARROW-2327) [JS] Table.fromStruct missing from externs
[
https://issues.apache.org/jira/browse/ARROW-2327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-2327:
Fix Version/s: JS-0.4.0
> [JS] Table.fromStruct missing from externs
> --
>
> Key: ARROW-2327
> URL: https://issues.apache.org/jira/browse/ARROW-2327
> Project: Apache Arrow
> Issue Type: Bug
> Components: JavaScript
>Reporter: Brian Hulette
>Priority: Major
> Fix For: JS-0.4.0
>
>
> {{Table.fromStruct}} is not listed in externs, so its obfuscated by the
> closure compiler
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-2328) Writing a slice with feather ignores the offset
[ https://issues.apache.org/jira/browse/ARROW-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-2328: Fix Version/s: (was: 0.9.0) 0.10.0 > Writing a slice with feather ignores the offset > --- > > Key: ARROW-2328 > URL: https://issues.apache.org/jira/browse/ARROW-2328 > Project: Apache Arrow > Issue Type: Bug > Components: C++ >Affects Versions: 0.8.0 >Reporter: Adrian >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > Writing a slice from row n of length m of an array to feather would write the > first m rows, instead of the rows starting at n. > The null bitmap also ends up misaligned. Also tested and fixed in the pull > request below. > I've created a pull request with tests and fix here: > [Pullrequest#1766|https://github.com/apache/arrow/pull/1766] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2330) [C++] Optimize delta buffer creation with partially finishable array builders
[ https://issues.apache.org/jira/browse/ARROW-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-2330: Fix Version/s: (was: 0.9.0) 0.10.0 > [C++] Optimize delta buffer creation with partially finishable array builders > - > > Key: ARROW-2330 > URL: https://issues.apache.org/jira/browse/ARROW-2330 > Project: Apache Arrow > Issue Type: New Feature > Components: C++ >Affects Versions: 0.8.0 >Reporter: Dimitri Vorona >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > The main aim of this change is to optimize the building of delta > dictionaries. In the current version delta dictionaries are built using an > additional "overflow" buffer which leads to complicated and potentially > error-prone code and subpar performance by doubling the number of lookups. > I solve this problem by introducing the notion of partially finishable array > builders, i.e. builder which are able to retain the state on calling Finish. > The interface is based on RecordBatchBuilder::Flush, i.e. Finish is > overloaded with additional signature Finish(bool reset_builder, > std::shared_ptr* out). The resulting Arrays point to the same data > buffer with different offsets. > I'm aware that the change is kind of biggish, but I'd like to discuss it > here. The solution makes the code more straight forward, doesn't bloat the > code base too much and leaves the API more or less untouched. Additionally, > the new way to make delta dictionaries by using a different call signature to > Finish feel cleaner to me. > I'm looking forward to your critic and improvement ideas. > The pull request is available at: https://github.com/apache/arrow/pull/1769 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] wesm commented on issue #23: Build wheels for 0.9.0
wesm commented on issue #23: Build wheels for 0.9.0 URL: https://github.com/apache/arrow-dist/pull/23#issuecomment-374970680 LGTM, but do we want to bundle the required Boost shared libraries with the Windows and macOS wheels? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (ARROW-2328) Writing a slice with feather ignores the offset
[
https://issues.apache.org/jira/browse/ARROW-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408021#comment-16408021
]
ASF GitHub Bot commented on ARROW-2328:
---
Adriandorr commented on a change in pull request #1766: ARROW-2328: [C++]
Writing a slice with feather ignores the offset.
URL: https://github.com/apache/arrow/pull/1766#discussion_r176107139
##
File path: cpp/src/arrow/ipc/feather.cc
##
@@ -603,7 +602,12 @@ class TableWriter::TableWriterImpl : public ArrayVisitor {
}
if (prim_values.values()) {
-values_buffer = prim_values.values()->data();
+if (prim_values.offset() != 0 && (fw_type.bit_width() % 8 != 0)) {
+ return arrow::Status::Invalid(
Review comment:
I've now made these changes and more.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Writing a slice with feather ignores the offset
> ---
>
> Key: ARROW-2328
> URL: https://issues.apache.org/jira/browse/ARROW-2328
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 0.8.0
>Reporter: Adrian
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Writing a slice from row n of length m of an array to feather would write the
> first m rows, instead of the rows starting at n.
> The null bitmap also ends up misaligned. Also tested and fixed in the pull
> request below.
> I've created a pull request with tests and fix here:
> [Pullrequest#1766|https://github.com/apache/arrow/pull/1766]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2281) [Python] Expose MakeArray to construct arrays from buffers
[
https://issues.apache.org/jira/browse/ARROW-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407873#comment-16407873
]
ASF GitHub Bot commented on ARROW-2281:
---
pitrou commented on a change in pull request #1772: ARROW-2281: [Python] Add
Array.from_buffers()
URL: https://github.com/apache/arrow/pull/1772#discussion_r176073851
##
File path: cpp/src/arrow/array.h
##
@@ -111,6 +111,11 @@ struct ARROW_EXPORT ArrayData {
int64_t null_count =
kUnknownNullCount,
int64_t offset = 0);
+ static std::shared_ptr Make(
Review comment:
This addition avoids having to spell the `std::move` declaration in Cython...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Expose MakeArray to construct arrays from buffers
> --
>
> Key: ARROW-2281
> URL: https://issues.apache.org/jira/browse/ARROW-2281
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Uwe L. Korn
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> To create new arrays from existing buffers in Python, we would need to call
> into the C++ {{MakeArray}} method. This would then construct the Array and we
> would only wrap it in Python to have construction support for all Array types.
> This would also mean that we need to have a Python representation of
> {{ArrayData}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2281) [Python] Expose MakeArray to construct arrays from buffers
[
https://issues.apache.org/jira/browse/ARROW-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407872#comment-16407872
]
ASF GitHub Bot commented on ARROW-2281:
---
pitrou opened a new pull request #1772: ARROW-2281: [Python] Add
Array.from_buffers()
URL: https://github.com/apache/arrow/pull/1772
Note this is shadowed by the specialized StringArray.from_buffers().
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Expose MakeArray to construct arrays from buffers
> --
>
> Key: ARROW-2281
> URL: https://issues.apache.org/jira/browse/ARROW-2281
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Uwe L. Korn
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> To create new arrays from existing buffers in Python, we would need to call
> into the C++ {{MakeArray}} method. This would then construct the Array and we
> would only wrap it in Python to have construction support for all Array types.
> This would also mean that we need to have a Python representation of
> {{ArrayData}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-2281) [Python] Expose MakeArray to construct arrays from buffers
[
https://issues.apache.org/jira/browse/ARROW-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated ARROW-2281:
--
Labels: pull-request-available (was: )
> [Python] Expose MakeArray to construct arrays from buffers
> --
>
> Key: ARROW-2281
> URL: https://issues.apache.org/jira/browse/ARROW-2281
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Uwe L. Korn
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> To create new arrays from existing buffers in Python, we would need to call
> into the C++ {{MakeArray}} method. This would then construct the Array and we
> would only wrap it in Python to have construction support for all Array types.
> This would also mean that we need to have a Python representation of
> {{ArrayData}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (ARROW-2281) [Python] Expose MakeArray to construct arrays from buffers
[
https://issues.apache.org/jira/browse/ARROW-2281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antoine Pitrou reassigned ARROW-2281:
-
Assignee: Antoine Pitrou (was: Uwe L. Korn)
> [Python] Expose MakeArray to construct arrays from buffers
> --
>
> Key: ARROW-2281
> URL: https://issues.apache.org/jira/browse/ARROW-2281
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Uwe L. Korn
>Assignee: Antoine Pitrou
>Priority: Major
> Fix For: 0.10.0
>
>
> To create new arrays from existing buffers in Python, we would need to call
> into the C++ {{MakeArray}} method. This would then construct the Array and we
> would only wrap it in Python to have construction support for all Array types.
> This would also mean that we need to have a Python representation of
> {{ArrayData}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1886) [Python] Add function to "flatten" structs within tables
[
https://issues.apache.org/jira/browse/ARROW-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407666#comment-16407666
]
ASF GitHub Bot commented on ARROW-1886:
---
pitrou commented on a change in pull request #1768: ARROW-1886: [C++/Python]
Flatten struct columns in table
URL: https://github.com/apache/arrow/pull/1768#discussion_r176019996
##
File path: python/pyarrow/tests/test_table.py
##
@@ -335,12 +348,34 @@ def test_table_remove_column_empty():
table = pa.Table.from_arrays(data, names=['a'])
t2 = table.remove_column(0)
+t2._validate()
assert len(t2) == len(table)
t3 = t2.add_column(0, table[0])
+t3._validate()
assert t3.equals(table)
+def test_table_flatten():
+ty1 = pa.struct([pa.field('x', pa.int16()),
+ pa.field('y', pa.float32())])
+ty2 = pa.struct([pa.field('nest', ty1)])
+a = pa.array([(1, 2.5), (3, 4.5)], type=ty1)
+b = pa.array([((11, 12.5),), ((13, 14.5),)], type=ty2)
+c = pa.array([False, True], type=pa.bool_())
+
+table = pa.Table.from_arrays([a, b, c], names=['a', 'b', 'c'])
+t2 = table.flatten()
+t2._validate()
+expected = pa.Table.from_arrays([
+pa.array([1, 3], type=pa.int16()),
+pa.array([2.5, 4.5], type=pa.float32()),
+pa.array([(11, 12.5), (13, 14.5)], type=ty1),
+c],
+names=['a.x', 'a.y', 'b.nest', 'c'])
Review comment:
I'm not sure we should deal with them ourselves? I *think* users can
reimplement the table-flattening functionality themselves using `array.flatten`
(if they want another naming scheme), though I'm not sure how easy that is with
the currently exposed APIs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Add function to "flatten" structs within tables
>
>
> Key: ARROW-1886
> URL: https://issues.apache.org/jira/browse/ARROW-1886
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Wes McKinney
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> See discussion in https://issues.apache.org/jira/browse/ARROW-1873
> When a user has a struct column, it may be more efficient to flatten the
> struct into multiple columns of the form {{struct_name.field_name}} for each
> field in the struct. Then when you call {{to_pandas}}, Python dictionaries do
> not have to be created, and the conversion will be much more efficient
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-2328) Writing a slice with feather ignores the offset
[ https://issues.apache.org/jira/browse/ARROW-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407661#comment-16407661 ] ASF GitHub Bot commented on ARROW-2328: --- Adriandorr commented on issue #1766: ARROW-2328: [C++] Writing a slice with feather ignores the offset. URL: https://github.com/apache/arrow/pull/1766#issuecomment-374875952 In the latest commit I have unit tested and fixed the null bitmap bug. I ended up adding the memory pool argument with a nullptr default and fail if an array is sliced and no pool was passed in. I also copy and pasted GetTruncatedBitmap from writer.cpp, I wasn't sure how to share it without it leaking into the public api. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Writing a slice with feather ignores the offset > --- > > Key: ARROW-2328 > URL: https://issues.apache.org/jira/browse/ARROW-2328 > Project: Apache Arrow > Issue Type: Bug > Components: C++ >Affects Versions: 0.8.0 >Reporter: Adrian >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > Writing a slice from row n of length m of an array to feather would write the > first m rows, instead of the rows starting at n. > The null bitmap also ends up misaligned. Also tested and fixed in the pull > request below. > I've created a pull request with tests and fix here: > [Pullrequest#1766|https://github.com/apache/arrow/pull/1766] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1886) [Python] Add function to "flatten" structs within tables
[
https://issues.apache.org/jira/browse/ARROW-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407660#comment-16407660
]
ASF GitHub Bot commented on ARROW-1886:
---
pitrou commented on a change in pull request #1768: ARROW-1886: [C++/Python]
Flatten struct columns in table
URL: https://github.com/apache/arrow/pull/1768#discussion_r176019222
##
File path: cpp/src/arrow/util/bit-util.cc
##
@@ -172,4 +172,58 @@ bool BitmapEquals(const uint8_t* left, int64_t
left_offset, const uint8_t* right
return true;
}
+namespace {
+
+void AlignedBitmapAnd(const uint8_t* left, int64_t left_offset, const uint8_t*
right,
+ int64_t right_offset, uint8_t* out, int64_t out_offset,
+ int64_t length) {
+ DCHECK_EQ(left_offset % 8, right_offset % 8);
+ DCHECK_EQ(left_offset % 8, out_offset % 8);
+
+ const int64_t nbytes = BitUtil::BytesForBits(length + left_offset);
+ left += left_offset / 8;
+ right += right_offset / 8;
+ out += out_offset / 8;
+ for (int64_t i = 0; i < nbytes; ++i) {
+out[i] = left[i] & right[i];
+ }
+}
+
+void UnalignedBitmapAnd(const uint8_t* left, int64_t left_offset, const
uint8_t* right,
+int64_t right_offset, uint8_t* out, int64_t out_offset,
+int64_t length) {
+ auto left_reader = internal::BitmapReader(left, left_offset, length);
+ auto right_reader = internal::BitmapReader(right, right_offset, length);
+ auto writer = internal::BitmapWriter(out, out_offset, length);
+ for (int64_t i = 0; i < length; ++i) {
+if (left_reader.IsSet() && right_reader.IsSet()) {
+ writer.Set();
+}
+left_reader.Next();
+right_reader.Next();
+writer.Next();
+ }
+ writer.Finish();
+}
+
+} // namespace
+
+Status BitmapAnd(MemoryPool* pool, const uint8_t* left, int64_t left_offset,
+ const uint8_t* right, int64_t right_offset, int64_t length,
+ int64_t out_offset, std::shared_ptr* out_buffer) {
Review comment:
Not sure what you mean with "tightly coupled". It seems to me that packed
bitmaps are a rather generic notion. The only thing peculiar here is passing a
bit offset.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Add function to "flatten" structs within tables
>
>
> Key: ARROW-1886
> URL: https://issues.apache.org/jira/browse/ARROW-1886
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Wes McKinney
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> See discussion in https://issues.apache.org/jira/browse/ARROW-1873
> When a user has a struct column, it may be more efficient to flatten the
> struct into multiple columns of the form {{struct_name.field_name}} for each
> field in the struct. Then when you call {{to_pandas}}, Python dictionaries do
> not have to be created, and the conversion will be much more efficient
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1886) [Python] Add function to "flatten" structs within tables
[
https://issues.apache.org/jira/browse/ARROW-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407657#comment-16407657
]
ASF GitHub Bot commented on ARROW-1886:
---
pitrou commented on a change in pull request #1768: ARROW-1886: [C++/Python]
Flatten struct columns in table
URL: https://github.com/apache/arrow/pull/1768#discussion_r176018821
##
File path: cpp/src/arrow/util/bit-util-test.cc
##
@@ -41,6 +44,59 @@ static void EnsureCpuInfoInitialized() {
}
}
+void WriteVectorToWriter(internal::BitmapWriter& writer, const
std::vector values) {
+ for (const auto& value : values) {
+if (value) {
+ writer.Set();
+} else {
+ writer.Clear();
+}
+writer.Next();
+ }
+ writer.Finish();
+}
+
+void BitmapFromVector(const std::vector& values, int64_t bit_offset,
+ std::shared_ptr* out_buffer, int64_t*
out_length) {
+ const int64_t length = values.size();
+ *out_length = length;
+ ASSERT_OK(GetEmptyBitmap(default_memory_pool(), length + bit_offset,
out_buffer));
+ auto writer = internal::BitmapWriter((*out_buffer)->mutable_data(),
bit_offset, length);
+ WriteVectorToWriter(writer, values);
+}
+
+#define ASSERT_READER_SET(reader)\
+ do { \
+ASSERT_TRUE(reader.IsSet()); \
+ASSERT_FALSE(reader.IsNotSet()); \
+reader.Next(); \
+ } while (0)
Review comment:
Ok, will do.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Add function to "flatten" structs within tables
>
>
> Key: ARROW-1886
> URL: https://issues.apache.org/jira/browse/ARROW-1886
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Wes McKinney
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> See discussion in https://issues.apache.org/jira/browse/ARROW-1873
> When a user has a struct column, it may be more efficient to flatten the
> struct into multiple columns of the form {{struct_name.field_name}} for each
> field in the struct. Then when you call {{to_pandas}}, Python dictionaries do
> not have to be created, and the conversion will be much more efficient
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1886) [Python] Add function to "flatten" structs within tables
[
https://issues.apache.org/jira/browse/ARROW-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407658#comment-16407658
]
ASF GitHub Bot commented on ARROW-1886:
---
pitrou commented on a change in pull request #1768: ARROW-1886: [C++/Python]
Flatten struct columns in table
URL: https://github.com/apache/arrow/pull/1768#discussion_r176018968
##
File path: cpp/src/arrow/util/bit-util-test.cc
##
@@ -41,6 +44,59 @@ static void EnsureCpuInfoInitialized() {
}
}
+void WriteVectorToWriter(internal::BitmapWriter& writer, const
std::vector values) {
+ for (const auto& value : values) {
+if (value) {
+ writer.Set();
+} else {
+ writer.Clear();
+}
+writer.Next();
+ }
+ writer.Finish();
+}
+
+void BitmapFromVector(const std::vector& values, int64_t bit_offset,
+ std::shared_ptr* out_buffer, int64_t*
out_length) {
+ const int64_t length = values.size();
+ *out_length = length;
+ ASSERT_OK(GetEmptyBitmap(default_memory_pool(), length + bit_offset,
out_buffer));
+ auto writer = internal::BitmapWriter((*out_buffer)->mutable_data(),
bit_offset, length);
+ WriteVectorToWriter(writer, values);
+}
+
+#define ASSERT_READER_SET(reader)\
+ do { \
+ASSERT_TRUE(reader.IsSet()); \
+ASSERT_FALSE(reader.IsNotSet()); \
+reader.Next(); \
+ } while (0)
+
+#define ASSERT_READER_NOT_SET(reader) \
+ do {\
+ASSERT_FALSE(reader.IsSet()); \
+ASSERT_TRUE(reader.IsNotSet()); \
+reader.Next();\
+ } while (0)
+
+// Assert that a BitmapReader yields the given bit values
+void ASSERT_READER_VALUES(internal::BitmapReader& reader, std::vector
values) {
+ for (const auto& value : values) {
+if (value) {
+ ASSERT_READER_SET(reader);
+} else {
+ ASSERT_READER_NOT_SET(reader);
+}
+ }
+}
+
+// Assert equal contents of a memory area and a vector of bytes
+void ASSERT_BYTES_EQ(const uint8_t* left, const std::vector& right) {
+ // No direct conversion from (T*, size) to vector
Review comment:
Hmm... you may be right, I'll give it a try.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Add function to "flatten" structs within tables
>
>
> Key: ARROW-1886
> URL: https://issues.apache.org/jira/browse/ARROW-1886
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Wes McKinney
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> See discussion in https://issues.apache.org/jira/browse/ARROW-1873
> When a user has a struct column, it may be more efficient to flatten the
> struct into multiple columns of the form {{struct_name.field_name}} for each
> field in the struct. Then when you call {{to_pandas}}, Python dictionaries do
> not have to be created, and the conversion will be much more efficient
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1886) [Python] Add function to "flatten" structs within tables
[
https://issues.apache.org/jira/browse/ARROW-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407654#comment-16407654
]
ASF GitHub Bot commented on ARROW-1886:
---
pitrou commented on a change in pull request #1768: ARROW-1886: [C++/Python]
Flatten struct columns in table
URL: https://github.com/apache/arrow/pull/1768#discussion_r176018578
##
File path: cpp/src/arrow/table.cc
##
@@ -127,6 +127,38 @@ std::shared_ptr ChunkedArray::Slice(int64_t
offset) const {
return Slice(offset, length_);
}
+Status ChunkedArray::Flatten(MemoryPool* pool,
+ std::vector* out)
const {
+ std::vector flattened;
+ if (type()->id() != Type::STRUCT) {
+// Emulate non-existent copy constructor
+flattened.emplace_back(std::make_shared(chunks_));
+*out = flattened;
+return Status::OK();
+ }
+ std::vector flattened_chunks;
+ for (const auto& chunk : chunks_) {
+ArrayVector res;
+RETURN_NOT_OK(dynamic_cast(*chunk).Flatten(pool,
));
Review comment:
I thought the `dynamic_cast` would help protect against bugs (though I'm not
sure how costly it is).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Add function to "flatten" structs within tables
>
>
> Key: ARROW-1886
> URL: https://issues.apache.org/jira/browse/ARROW-1886
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Wes McKinney
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> See discussion in https://issues.apache.org/jira/browse/ARROW-1873
> When a user has a struct column, it may be more efficient to flatten the
> struct into multiple columns of the form {{struct_name.field_name}} for each
> field in the struct. Then when you call {{to_pandas}}, Python dictionaries do
> not have to be created, and the conversion will be much more efficient
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1886) [Python] Add function to "flatten" structs within tables
[
https://issues.apache.org/jira/browse/ARROW-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407656#comment-16407656
]
ASF GitHub Bot commented on ARROW-1886:
---
pitrou commented on a change in pull request #1768: ARROW-1886: [C++/Python]
Flatten struct columns in table
URL: https://github.com/apache/arrow/pull/1768#discussion_r176018790
##
File path: cpp/src/arrow/type.cc
##
@@ -42,6 +42,21 @@ std::shared_ptr Field::RemoveMetadata() const {
return std::make_shared(name_, type_, nullable_);
}
+std::vector Field::Flatten() const {
+ std::vector flattened;
+ if (type_->id() == Type::STRUCT) {
+for (const auto& child : type_->children()) {
+ auto flattened_child = std::make_shared(*child);
+ flattened.push_back(flattened_child);
+ flattened_child->name_.insert(0, name() + ".");
Review comment:
I'm not sure we should do anything about it. Names are indicative in any
case, there can even be duplicate names ;-)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Add function to "flatten" structs within tables
>
>
> Key: ARROW-1886
> URL: https://issues.apache.org/jira/browse/ARROW-1886
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Wes McKinney
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> See discussion in https://issues.apache.org/jira/browse/ARROW-1873
> When a user has a struct column, it may be more efficient to flatten the
> struct into multiple columns of the form {{struct_name.field_name}} for each
> field in the struct. Then when you call {{to_pandas}}, Python dictionaries do
> not have to be created, and the conversion will be much more efficient
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-640) [Python] Arrow scalar values should have a sensible __hash__ and comparison
[
https://issues.apache.org/jira/browse/ARROW-640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407653#comment-16407653
]
ASF GitHub Bot commented on ARROW-640:
--
pitrou commented on a change in pull request #1765: ARROW-640: [Python]
Implement __hash__ and equality for Array scalar values Arrow scalar values
URL: https://github.com/apache/arrow/pull/1765#discussion_r176018434
##
File path: python/pyarrow/scalar.pxi
##
@@ -73,6 +73,12 @@ cdef class ArrayValue(Scalar):
raise NotImplementedError(
"Cannot compare Arrow values that don't support as_py()")
+def __hash__(self):
Review comment:
> Per the JIRA ticket can we consider a separate ticket for a fast path
custom hash on ints?
Definitely.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Arrow scalar values should have a sensible __hash__ and comparison
> ---
>
> Key: ARROW-640
> URL: https://issues.apache.org/jira/browse/ARROW-640
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
>Reporter: Miki Tebeka
>Assignee: Alex Hagerman
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> {noformat}
> In [86]: arr = pa.from_pylist([1, 1, 1, 2])
> In [87]: set(arr)
> Out[87]: {1, 2, 1, 1}
> In [88]: arr[0] == arr[1]
> Out[88]: False
> In [89]: arr
> Out[89]:
>
> [
> 1,
> 1,
> 1,
> 2
> ]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARROW-1886) [Python] Add function to "flatten" structs within tables
[
https://issues.apache.org/jira/browse/ARROW-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407655#comment-16407655
]
ASF GitHub Bot commented on ARROW-1886:
---
pitrou commented on a change in pull request #1768: ARROW-1886: [C++/Python]
Flatten struct columns in table
URL: https://github.com/apache/arrow/pull/1768#discussion_r176018684
##
File path: cpp/src/arrow/array.cc
##
@@ -365,6 +365,50 @@ std::shared_ptr StructArray::field(int i) const {
return boxed_fields_[i];
}
+Status StructArray::Flatten(MemoryPool* pool, ArrayVector* out) const {
Review comment:
Will do (note this is commenting on a piece of PR #1755).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Python] Add function to "flatten" structs within tables
>
>
> Key: ARROW-1886
> URL: https://issues.apache.org/jira/browse/ARROW-1886
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Python
>Reporter: Wes McKinney
>Assignee: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.10.0
>
>
> See discussion in https://issues.apache.org/jira/browse/ARROW-1873
> When a user has a struct column, it may be more efficient to flatten the
> struct into multiple columns of the form {{struct_name.field_name}} for each
> field in the struct. Then when you call {{to_pandas}}, Python dictionaries do
> not have to be created, and the conversion will be much more efficient
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARROW-2328) Writing a slice with feather ignores the offset
[ https://issues.apache.org/jira/browse/ARROW-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian updated ARROW-2328: -- Description: Writing a slice from row n of length m of an array to feather would write the first m rows, instead of the rows starting at n. The null bitmap also ends up misaligned. Also tested and fixed in the pull request below. I've created a pull request with tests and fix here: [Pullrequest#1766|https://github.com/apache/arrow/pull/1766] was: Writing a slice from row n of length m of an array to feather would write the first m rows, instead of the rows starting at n. I've created a pull request with tests and fix here: [Pullrequest#1766|https://github.com/apache/arrow/pull/1766] > Writing a slice with feather ignores the offset > --- > > Key: ARROW-2328 > URL: https://issues.apache.org/jira/browse/ARROW-2328 > Project: Apache Arrow > Issue Type: Bug > Components: C++ >Affects Versions: 0.8.0 >Reporter: Adrian >Priority: Major > Labels: pull-request-available > Fix For: 0.9.0 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > Writing a slice from row n of length m of an array to feather would write the > first m rows, instead of the rows starting at n. > The null bitmap also ends up misaligned. Also tested and fixed in the pull > request below. > I've created a pull request with tests and fix here: > [Pullrequest#1766|https://github.com/apache/arrow/pull/1766] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
