Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
Rebasing myself seems to have fixed it. On Mon, Apr 20, 2020 at 10:50 PM Micah Kornfield wrote: > Did something go wrong with the rebase of open PRs? I'm seeing a lot of > diffs show up after rebasing off of master? > > Thanks, > Micah > > On Mon, Apr 20, 2020 at 8:59 PM Sutou Kouhei wrote: > >> MSYS2: Merged: https://github.com/msys2/MINGW-packages/pull/6396 >> >> In >> "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 >> 20:09:29 -0700, >> Neal Richardson wrote: >> >> > Homebrew updates submitted: >> > https://github.com/Homebrew/homebrew-core/pull/53475 >> > >> > R packaging hopefully will go through tomorrow. >> > >> > Neal >> > >> > >> > On Mon, Apr 20, 2020 at 7:28 PM Andy Grove >> wrote: >> > >> >> Well, I got trhe crates published, but there's a nasty workaround for >> users >> >> that want to use these crates as a dependency and it means there is no >> real >> >> dependency management on the Flight protocol version. I think the >> answer is >> >> that we need to publish the Flight.proto as part of the arrow-flight >> crate >> >> and make sure that version is used in the custom build script. I'll >> look at >> >> this again tomorrow and try and come up with a solution for the next >> >> release. >> >> >> >> Here's the JIRA to track this specific issue. >> >> >> >> https://issues.apache.org/jira/browse/ARROW-8536 >> >> >> >> On Mon, Apr 20, 2020 at 7:49 PM Andy Grove >> wrote: >> >> >> >> > I've run into issues publishing the Rust crates and I don't think I >> can >> >> > resolve this tonight. I am documenting the issue in >> >> > https://issues.apache.org/jira/browse/ARROW-8535 >> >> > >> >> > >> >> > On Mon, Apr 20, 2020 at 5:02 PM Krisztián Szűcs < >> >> szucs.kriszt...@gmail.com> >> >> > wrote: >> >> > >> >> >> Created a PR with updated docs. >> >> >> >> >> >> Conda post release task is left, it's a bit strange that the >> conda-forge >> >> >> autotick bot has not created the version bump PRs yet. I'm updating >> >> >> them manually tomorrow. >> >> >> >> >> >> 1. [x] rebase >> >> >> 2. [x] upload source >> >> >> 3. [x] upload binaries >> >> >> 4. [x] update website >> >> >> 5. [x] upload ruby gems >> >> >> 6. [x] upload js packages >> >> >> 8. [x] upload C# packages >> >> >> 9. [Andy] upload rust crates >> >> >> 10. [ ] update conda recipes >> >> >> 11. [x] upload wheels to pypi >> >> >> 12. [Neal] update homebrew packages >> >> >> 13. [x] update maven artifacts >> >> >> 14. [kou] update msys2 >> >> >> 15. [Neal] update R packages >> >> >> 16. [x] update docs >> >> >> >> >> >> I'm going to announce 0.17 once the site PRs get merged. >> >> >> > >> >> >> > >> >> >> > Thanks, >> >> >> > -- >> >> >> > kou >> >> >> > >> >> >> > In > >> u+1f3...@mail.gmail.com> >> >> >> > "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr >> 2020 >> >> >> 23:20:48 +0200, >> >> >> > Krisztián Szűcs wrote: >> >> >> > >> >> >> > > On Mon, Apr 20, 2020 at 11:17 PM Andy Grove < >> andygrov...@gmail.com> >> >> >> wrote: >> >> >> > >> >> >> >> > >> Ok, I can look into this after work today (in about 3 hours). >> >> >> > > Great, thanks! >> >> >> > > >> >> >> > > The current status is (`x` means done): >> >> >> > > >> >> >> > > 1. [x] rebase >> >> >> > > 2. [x] upload source >> >> >> > > 3. [x] upload binaries >> >> >> > > 4. [x] update website >> >> >> > > 5. [x] upload ruby gems >> >> >> > > 6. [x] upload js packages >> >> >> > > 8. [ ] upload C# crates >> >> >> > > 9. [Andy] upload rust crates >> >> >> > > 10. [ ] update conda recipes >> >> >> > > 11. [x] upload wheels to pypi >> >> >> > > 12. [Neal] update homebrew packages >> >> >> > > 13. [x] update maven artifacts >> >> >> > > 14. [ ] update msys2 >> >> >> > > 15. [Neal] update R packages >> >> >> > > 16. [Krisztian] update docs >> >> >> > >> >> >> >> > >> On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs < >> >> >> szucs.kriszt...@gmail.com> >> >> >> > >> wrote: >> >> >> > >> >> >> >> > >> > Thanks Andy! I tried to upload the rust packages but >> >> arrow-flight, >> >> >> > >> > but a version pin is missing from the package tree: >> >> >> > >> > >> >> >> > >> > error: all dependencies must have a version specified when >> >> >> publishing. >> >> >> > >> > dependency `arrow-flight` does not specify a version >> >> >> > >> > >> >> >> > >> > Please upload the packages! >> >> >> > >> > >> >> >> > >> > Also added Uwe and Kou to the package owners. >> >> >> > >> > >> >> >> > >> > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove < >> >> andygrov...@gmail.com> >> >> >> wrote: >> >> >> > >> > > >> >> >> > >> > > You should have an invite for the arrow-flight crate. >> Please >> >> >> check >> >> >> > >> > > https://crates.io/me/pending-invites >> >> >> > >> > > >> >> >> > >> > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < >> >> >> > >> > szucs.kriszt...@gmail.com> >> >> >> > >> > > wrote: >> >> >> > >> > > >> >> >> > >> > > > 1. [x] rebase >> >> >> > >> > > > 2. [x] upload source >> >> >> > >> > > > 3. [x] upload
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
Did something go wrong with the rebase of open PRs? I'm seeing a lot of diffs show up after rebasing off of master? Thanks, Micah On Mon, Apr 20, 2020 at 8:59 PM Sutou Kouhei wrote: > MSYS2: Merged: https://github.com/msys2/MINGW-packages/pull/6396 > > In > "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 > 20:09:29 -0700, > Neal Richardson wrote: > > > Homebrew updates submitted: > > https://github.com/Homebrew/homebrew-core/pull/53475 > > > > R packaging hopefully will go through tomorrow. > > > > Neal > > > > > > On Mon, Apr 20, 2020 at 7:28 PM Andy Grove > wrote: > > > >> Well, I got trhe crates published, but there's a nasty workaround for > users > >> that want to use these crates as a dependency and it means there is no > real > >> dependency management on the Flight protocol version. I think the > answer is > >> that we need to publish the Flight.proto as part of the arrow-flight > crate > >> and make sure that version is used in the custom build script. I'll > look at > >> this again tomorrow and try and come up with a solution for the next > >> release. > >> > >> Here's the JIRA to track this specific issue. > >> > >> https://issues.apache.org/jira/browse/ARROW-8536 > >> > >> On Mon, Apr 20, 2020 at 7:49 PM Andy Grove > wrote: > >> > >> > I've run into issues publishing the Rust crates and I don't think I > can > >> > resolve this tonight. I am documenting the issue in > >> > https://issues.apache.org/jira/browse/ARROW-8535 > >> > > >> > > >> > On Mon, Apr 20, 2020 at 5:02 PM Krisztián Szűcs < > >> szucs.kriszt...@gmail.com> > >> > wrote: > >> > > >> >> Created a PR with updated docs. > >> >> > >> >> Conda post release task is left, it's a bit strange that the > conda-forge > >> >> autotick bot has not created the version bump PRs yet. I'm updating > >> >> them manually tomorrow. > >> >> > >> >> 1. [x] rebase > >> >> 2. [x] upload source > >> >> 3. [x] upload binaries > >> >> 4. [x] update website > >> >> 5. [x] upload ruby gems > >> >> 6. [x] upload js packages > >> >> 8. [x] upload C# packages > >> >> 9. [Andy] upload rust crates > >> >> 10. [ ] update conda recipes > >> >> 11. [x] upload wheels to pypi > >> >> 12. [Neal] update homebrew packages > >> >> 13. [x] update maven artifacts > >> >> 14. [kou] update msys2 > >> >> 15. [Neal] update R packages > >> >> 16. [x] update docs > >> >> > >> >> I'm going to announce 0.17 once the site PRs get merged. > >> >> > > >> >> > > >> >> > Thanks, > >> >> > -- > >> >> > kou > >> >> > > >> >> > In >> u+1f3...@mail.gmail.com> > >> >> > "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr > 2020 > >> >> 23:20:48 +0200, > >> >> > Krisztián Szűcs wrote: > >> >> > > >> >> > > On Mon, Apr 20, 2020 at 11:17 PM Andy Grove < > andygrov...@gmail.com> > >> >> wrote: > >> >> > >> > >> >> > >> Ok, I can look into this after work today (in about 3 hours). > >> >> > > Great, thanks! > >> >> > > > >> >> > > The current status is (`x` means done): > >> >> > > > >> >> > > 1. [x] rebase > >> >> > > 2. [x] upload source > >> >> > > 3. [x] upload binaries > >> >> > > 4. [x] update website > >> >> > > 5. [x] upload ruby gems > >> >> > > 6. [x] upload js packages > >> >> > > 8. [ ] upload C# crates > >> >> > > 9. [Andy] upload rust crates > >> >> > > 10. [ ] update conda recipes > >> >> > > 11. [x] upload wheels to pypi > >> >> > > 12. [Neal] update homebrew packages > >> >> > > 13. [x] update maven artifacts > >> >> > > 14. [ ] update msys2 > >> >> > > 15. [Neal] update R packages > >> >> > > 16. [Krisztian] update docs > >> >> > >> > >> >> > >> On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs < > >> >> szucs.kriszt...@gmail.com> > >> >> > >> wrote: > >> >> > >> > >> >> > >> > Thanks Andy! I tried to upload the rust packages but > >> arrow-flight, > >> >> > >> > but a version pin is missing from the package tree: > >> >> > >> > > >> >> > >> > error: all dependencies must have a version specified when > >> >> publishing. > >> >> > >> > dependency `arrow-flight` does not specify a version > >> >> > >> > > >> >> > >> > Please upload the packages! > >> >> > >> > > >> >> > >> > Also added Uwe and Kou to the package owners. > >> >> > >> > > >> >> > >> > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove < > >> andygrov...@gmail.com> > >> >> wrote: > >> >> > >> > > > >> >> > >> > > You should have an invite for the arrow-flight crate. Please > >> >> check > >> >> > >> > > https://crates.io/me/pending-invites > >> >> > >> > > > >> >> > >> > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < > >> >> > >> > szucs.kriszt...@gmail.com> > >> >> > >> > > wrote: > >> >> > >> > > > >> >> > >> > > > 1. [x] rebase > >> >> > >> > > > 2. [x] upload source > >> >> > >> > > > 3. [x] upload binaries > >> >> > >> > > > 4. [x] update website > >> >> > >> > > > 5. [x] upload ruby gems > >> >> > >> > > > 6. [x] upload js packages > >> >> > >> > > > 8. [ ] upload C# crates > >> >> > >> > > > 9. [ ] upload rust crates > >> >> > >> > > > 10. [ ]
Re: [C++] Revamping approach to Arrow compute kernel development
Hi Wes,
I think reducing temporary memory allocation is a great effort and will
show great benefit in compute intensive scenarios.
As we are mainly working with the Rust and Datafusion part of the Arrow
project I was wondering how we could best align the concepts and
implementations on that level.
Is the approach that the compute kernels have to be implemented for every
language separately as well similar to the format (which is my current
understanding) or do you see options for re-use and a common/shared
implementation?
Happy to get your thoughts on that.
Thanks,
Sven
On Sun, Apr 19, 2020 at 2:14 AM Wes McKinney wrote:
> I started a brain dump of some issues that come to mind around kernel
> implementation and array expression evaluation. I'll try to fill this
> out, and it would be helpful to add supporting citations to other
> projects about what kinds of issues come up and what implementation
> strategies may be helpful. If anyone would like edit access please let
> me know, otherwise feel free to add comments
>
>
> https://docs.google.com/document/d/1LFk3WRfWGQbJ9uitWwucjiJsZMqLh8lC1vAUOscLtj8/edit?usp=sharing
>
> On Sat, Apr 18, 2020 at 4:41 PM Wes McKinney wrote:
> >
> > hi folks,
> >
> > This e-mail comes in the context of two C++ data processing
> > subprojects we have discussed in the past
> >
> > * Data Frame API
> >
> https://docs.google.com/document/d/1XHe_j87n2VHGzEbnLe786GHbbcbrzbjgG8D0IXWAeHg/edit
> > * In-memory Query Engine
> >
> https://docs.google.com/document/d/10RoUZmiMQRi_J1FcPeVAUAMJ6d_ZuiEbaM2Y33sNPu4/edit
> >
> > One important area for both of these projects is the evaluation of
> > array expressions that are found inside projections (i.e. "SELECT" --
> > without any aggregate functions), filters (i.e. "WHERE"), and join
> > clauses. In summary, these are mostly elementwise functions that do
> > not alter the size of their input arrays. For conciseness I'll call
> > these kinds of expressions "array-exprs"
> >
> > Gandiva does LLVM code generation for evaluating array-exprs, but I
> > think it makes sense to have both codegen'd expressions as well as
> > X100-style [1] interpreted array-expr evaluation. These modes of query
> > evaluation ideally should share as much common implementation code as
> > possible (taking into account Gandiva's need to compile to LLVM IR)
> >
> > I have reviewed the contents of our arrow/compute/kernels directory
> > and made the following spreadsheet
> >
> >
> https://docs.google.com/spreadsheets/d/1oXVy29GT1apM4e3isNpzk497Re3OqM8APvlBZCDc8B8/edit?usp=sharing
> >
> > There are some problems with our current collection of kernels in the
> > context of array-expr evaluation in query processing:
> >
> > * For efficiency, kernels used for array-expr evaluation should write
> > into preallocated memory as their default mode. This enables the
> > interpreter to avoid temporary memory allocations and improve CPU
> > cache utilization. Almost none of our kernels are implemented this way
> > currently.
> > * The current approach for expr-type kernels of having a top-level
> > memory-allocating function is not scalable for binding developers. I
> > believe instead that kernels should be selected and invoked
> > generically by using the string name of the kernel
> >
> > On this last point, what I am suggesting is that we do something more
> like
> >
> > ASSIGN_OR_RAISE(auto kernel, compute::GetKernel("greater", {type0,
> type1}));
> > ArrayData* out = ... ;
> > RETURN_NOT_OK(kernel->Call({arg0, arg1}, ));
> >
> > In particular, when we reason that successive kernel invocations can
> > reuse memory, we can have code that is doing in essence
> >
> > k1->Call({arg0, arg1}, )
> > k2->Call({out}, ))
> > k3->Call({arg2, out}, )
> >
> > Of course, if you want the kernel to allocate memory then you can do
> > something like
> >
> > ASSIGN_OR_RAISE(out = kernel->Call({arg0, arg1}));
> >
> > And if you want to avoid the "GetKernel" business you should be able to
> do
> >
> > ASSIGN_OR_RAISE(auto result, ExecKernel(name, {arg0, arg1}));
> >
> > I think this "ExecKernel" function should likely replace the public
> > APIs for running specific kernels that we currently have.
> >
> > One detail that isn't addressed above is what to do with
> > kernel-specific configuration options. One way to address that is to
> > have a common base type for all options so that we can do
> >
> > struct MyKernelOptions : public KernelOptions {
> > ...
> > };
> >
> > and then
> >
> > MyKernelOptions options = ...;
> > out = ExecKernel(name, args, options);
> >
> > Maybe there are some other ideas.
> >
> > At some point we will need to have an implementation blitz where we go
> > from our current 20 or so non-codegen'd kernels for array-exprs to
> > several hundred, so I wanted to discuss these issues so we can all get
> > on the same page. I'd like to take a crack at an initial iteration of
> > the above API proposal with a centralized kernel registry (that will
> > also need to
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
MSYS2: Merged: https://github.com/msys2/MINGW-packages/pull/6396 In "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 20:09:29 -0700, Neal Richardson wrote: > Homebrew updates submitted: > https://github.com/Homebrew/homebrew-core/pull/53475 > > R packaging hopefully will go through tomorrow. > > Neal > > > On Mon, Apr 20, 2020 at 7:28 PM Andy Grove wrote: > >> Well, I got trhe crates published, but there's a nasty workaround for users >> that want to use these crates as a dependency and it means there is no real >> dependency management on the Flight protocol version. I think the answer is >> that we need to publish the Flight.proto as part of the arrow-flight crate >> and make sure that version is used in the custom build script. I'll look at >> this again tomorrow and try and come up with a solution for the next >> release. >> >> Here's the JIRA to track this specific issue. >> >> https://issues.apache.org/jira/browse/ARROW-8536 >> >> On Mon, Apr 20, 2020 at 7:49 PM Andy Grove wrote: >> >> > I've run into issues publishing the Rust crates and I don't think I can >> > resolve this tonight. I am documenting the issue in >> > https://issues.apache.org/jira/browse/ARROW-8535 >> > >> > >> > On Mon, Apr 20, 2020 at 5:02 PM Krisztián Szűcs < >> szucs.kriszt...@gmail.com> >> > wrote: >> > >> >> Created a PR with updated docs. >> >> >> >> Conda post release task is left, it's a bit strange that the conda-forge >> >> autotick bot has not created the version bump PRs yet. I'm updating >> >> them manually tomorrow. >> >> >> >> 1. [x] rebase >> >> 2. [x] upload source >> >> 3. [x] upload binaries >> >> 4. [x] update website >> >> 5. [x] upload ruby gems >> >> 6. [x] upload js packages >> >> 8. [x] upload C# packages >> >> 9. [Andy] upload rust crates >> >> 10. [ ] update conda recipes >> >> 11. [x] upload wheels to pypi >> >> 12. [Neal] update homebrew packages >> >> 13. [x] update maven artifacts >> >> 14. [kou] update msys2 >> >> 15. [Neal] update R packages >> >> 16. [x] update docs >> >> >> >> I'm going to announce 0.17 once the site PRs get merged. >> >> > >> >> > >> >> > Thanks, >> >> > -- >> >> > kou >> >> > >> >> > In > u+1f3...@mail.gmail.com> >> >> > "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 >> >> 23:20:48 +0200, >> >> > Krisztián Szűcs wrote: >> >> > >> >> > > On Mon, Apr 20, 2020 at 11:17 PM Andy Grove >> >> wrote: >> >> > >> >> >> > >> Ok, I can look into this after work today (in about 3 hours). >> >> > > Great, thanks! >> >> > > >> >> > > The current status is (`x` means done): >> >> > > >> >> > > 1. [x] rebase >> >> > > 2. [x] upload source >> >> > > 3. [x] upload binaries >> >> > > 4. [x] update website >> >> > > 5. [x] upload ruby gems >> >> > > 6. [x] upload js packages >> >> > > 8. [ ] upload C# crates >> >> > > 9. [Andy] upload rust crates >> >> > > 10. [ ] update conda recipes >> >> > > 11. [x] upload wheels to pypi >> >> > > 12. [Neal] update homebrew packages >> >> > > 13. [x] update maven artifacts >> >> > > 14. [ ] update msys2 >> >> > > 15. [Neal] update R packages >> >> > > 16. [Krisztian] update docs >> >> > >> >> >> > >> On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs < >> >> szucs.kriszt...@gmail.com> >> >> > >> wrote: >> >> > >> >> >> > >> > Thanks Andy! I tried to upload the rust packages but >> arrow-flight, >> >> > >> > but a version pin is missing from the package tree: >> >> > >> > >> >> > >> > error: all dependencies must have a version specified when >> >> publishing. >> >> > >> > dependency `arrow-flight` does not specify a version >> >> > >> > >> >> > >> > Please upload the packages! >> >> > >> > >> >> > >> > Also added Uwe and Kou to the package owners. >> >> > >> > >> >> > >> > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove < >> andygrov...@gmail.com> >> >> wrote: >> >> > >> > > >> >> > >> > > You should have an invite for the arrow-flight crate. Please >> >> check >> >> > >> > > https://crates.io/me/pending-invites >> >> > >> > > >> >> > >> > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < >> >> > >> > szucs.kriszt...@gmail.com> >> >> > >> > > wrote: >> >> > >> > > >> >> > >> > > > 1. [x] rebase >> >> > >> > > > 2. [x] upload source >> >> > >> > > > 3. [x] upload binaries >> >> > >> > > > 4. [x] update website >> >> > >> > > > 5. [x] upload ruby gems >> >> > >> > > > 6. [x] upload js packages >> >> > >> > > > 8. [ ] upload C# crates >> >> > >> > > > 9. [ ] upload rust crates >> >> > >> > > > 10. [ ] update conda recipes >> >> > >> > > > 11. [-] upload wheels to pypi >> >> > >> > > > 12. [ ] update homebrew packages >> >> > >> > > > 13. [-] update maven artifacts >> >> > >> > > > 14. [ ] update msys2 >> >> > >> > > > 15. [ ] update R packages >> >> > >> > > > >> >> > >> > > > @Andy Grove please grant me access to the arrow-flight crate >> >> > >> > > > @Neal Richardson please update the R package >> >> > >> > > > >> >> > >> > > > Currently I'm handling maven and pypi artifacts. >> >> >
[jira] [Created] (ARROW-8538) [Packaging] Remove boost from homebrew formula
Neal Richardson created ARROW-8538: -- Summary: [Packaging] Remove boost from homebrew formula Key: ARROW-8538 URL: https://issues.apache.org/jira/browse/ARROW-8538 Project: Apache Arrow Issue Type: Improvement Components: C++, Packaging Reporter: Neal Richardson Assignee: Neal Richardson -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
Homebrew updates submitted: https://github.com/Homebrew/homebrew-core/pull/53475 R packaging hopefully will go through tomorrow. Neal On Mon, Apr 20, 2020 at 7:28 PM Andy Grove wrote: > Well, I got trhe crates published, but there's a nasty workaround for users > that want to use these crates as a dependency and it means there is no real > dependency management on the Flight protocol version. I think the answer is > that we need to publish the Flight.proto as part of the arrow-flight crate > and make sure that version is used in the custom build script. I'll look at > this again tomorrow and try and come up with a solution for the next > release. > > Here's the JIRA to track this specific issue. > > https://issues.apache.org/jira/browse/ARROW-8536 > > On Mon, Apr 20, 2020 at 7:49 PM Andy Grove wrote: > > > I've run into issues publishing the Rust crates and I don't think I can > > resolve this tonight. I am documenting the issue in > > https://issues.apache.org/jira/browse/ARROW-8535 > > > > > > On Mon, Apr 20, 2020 at 5:02 PM Krisztián Szűcs < > szucs.kriszt...@gmail.com> > > wrote: > > > >> Created a PR with updated docs. > >> > >> Conda post release task is left, it's a bit strange that the conda-forge > >> autotick bot has not created the version bump PRs yet. I'm updating > >> them manually tomorrow. > >> > >> 1. [x] rebase > >> 2. [x] upload source > >> 3. [x] upload binaries > >> 4. [x] update website > >> 5. [x] upload ruby gems > >> 6. [x] upload js packages > >> 8. [x] upload C# packages > >> 9. [Andy] upload rust crates > >> 10. [ ] update conda recipes > >> 11. [x] upload wheels to pypi > >> 12. [Neal] update homebrew packages > >> 13. [x] update maven artifacts > >> 14. [kou] update msys2 > >> 15. [Neal] update R packages > >> 16. [x] update docs > >> > >> I'm going to announce 0.17 once the site PRs get merged. > >> > > >> > > >> > Thanks, > >> > -- > >> > kou > >> > > >> > In u+1f3...@mail.gmail.com> > >> > "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 > >> 23:20:48 +0200, > >> > Krisztián Szűcs wrote: > >> > > >> > > On Mon, Apr 20, 2020 at 11:17 PM Andy Grove > >> wrote: > >> > >> > >> > >> Ok, I can look into this after work today (in about 3 hours). > >> > > Great, thanks! > >> > > > >> > > The current status is (`x` means done): > >> > > > >> > > 1. [x] rebase > >> > > 2. [x] upload source > >> > > 3. [x] upload binaries > >> > > 4. [x] update website > >> > > 5. [x] upload ruby gems > >> > > 6. [x] upload js packages > >> > > 8. [ ] upload C# crates > >> > > 9. [Andy] upload rust crates > >> > > 10. [ ] update conda recipes > >> > > 11. [x] upload wheels to pypi > >> > > 12. [Neal] update homebrew packages > >> > > 13. [x] update maven artifacts > >> > > 14. [ ] update msys2 > >> > > 15. [Neal] update R packages > >> > > 16. [Krisztian] update docs > >> > >> > >> > >> On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs < > >> szucs.kriszt...@gmail.com> > >> > >> wrote: > >> > >> > >> > >> > Thanks Andy! I tried to upload the rust packages but > arrow-flight, > >> > >> > but a version pin is missing from the package tree: > >> > >> > > >> > >> > error: all dependencies must have a version specified when > >> publishing. > >> > >> > dependency `arrow-flight` does not specify a version > >> > >> > > >> > >> > Please upload the packages! > >> > >> > > >> > >> > Also added Uwe and Kou to the package owners. > >> > >> > > >> > >> > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove < > andygrov...@gmail.com> > >> wrote: > >> > >> > > > >> > >> > > You should have an invite for the arrow-flight crate. Please > >> check > >> > >> > > https://crates.io/me/pending-invites > >> > >> > > > >> > >> > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < > >> > >> > szucs.kriszt...@gmail.com> > >> > >> > > wrote: > >> > >> > > > >> > >> > > > 1. [x] rebase > >> > >> > > > 2. [x] upload source > >> > >> > > > 3. [x] upload binaries > >> > >> > > > 4. [x] update website > >> > >> > > > 5. [x] upload ruby gems > >> > >> > > > 6. [x] upload js packages > >> > >> > > > 8. [ ] upload C# crates > >> > >> > > > 9. [ ] upload rust crates > >> > >> > > > 10. [ ] update conda recipes > >> > >> > > > 11. [-] upload wheels to pypi > >> > >> > > > 12. [ ] update homebrew packages > >> > >> > > > 13. [-] update maven artifacts > >> > >> > > > 14. [ ] update msys2 > >> > >> > > > 15. [ ] update R packages > >> > >> > > > > >> > >> > > > @Andy Grove please grant me access to the arrow-flight crate > >> > >> > > > @Neal Richardson please update the R package > >> > >> > > > > >> > >> > > > Currently I'm handling maven and pypi artifacts. > >> > >> > > > > >> > >> > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs > >> > >> > > > wrote: > >> > >> > > > > > >> > >> > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding > >> +1 vote. > >> > >> > > > > > >> > >> > > > > I'm starting the post-release tasks. I'm going to send > >> updates to > >> > >> > this > >>
[jira] [Created] (ARROW-8537) [C++] Performance regression from ARROW-8523
Yibo Cai created ARROW-8537: --- Summary: [C++] Performance regression from ARROW-8523 Key: ARROW-8537 URL: https://issues.apache.org/jira/browse/ARROW-8537 Project: Apache Arrow Issue Type: Bug Reporter: Yibo Cai I optimized BitmapReader in [this PR|https://github.com/apache/arrow/pull/6986] and see performance uplift of BitmapReader test case. I didn't check other test cases as this change looks trivial. I reviewed all test cases just now and see big performance drop of 4 cases, details at [PR link|https://github.com/apache/arrow/pull/6986#issuecomment-616915079]. I also compared performance of code using BitmapReader, no obvious changes found. Looks we should revert that PR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
Well, I got trhe crates published, but there's a nasty workaround for users that want to use these crates as a dependency and it means there is no real dependency management on the Flight protocol version. I think the answer is that we need to publish the Flight.proto as part of the arrow-flight crate and make sure that version is used in the custom build script. I'll look at this again tomorrow and try and come up with a solution for the next release. Here's the JIRA to track this specific issue. https://issues.apache.org/jira/browse/ARROW-8536 On Mon, Apr 20, 2020 at 7:49 PM Andy Grove wrote: > I've run into issues publishing the Rust crates and I don't think I can > resolve this tonight. I am documenting the issue in > https://issues.apache.org/jira/browse/ARROW-8535 > > > On Mon, Apr 20, 2020 at 5:02 PM Krisztián Szűcs > wrote: > >> Created a PR with updated docs. >> >> Conda post release task is left, it's a bit strange that the conda-forge >> autotick bot has not created the version bump PRs yet. I'm updating >> them manually tomorrow. >> >> 1. [x] rebase >> 2. [x] upload source >> 3. [x] upload binaries >> 4. [x] update website >> 5. [x] upload ruby gems >> 6. [x] upload js packages >> 8. [x] upload C# packages >> 9. [Andy] upload rust crates >> 10. [ ] update conda recipes >> 11. [x] upload wheels to pypi >> 12. [Neal] update homebrew packages >> 13. [x] update maven artifacts >> 14. [kou] update msys2 >> 15. [Neal] update R packages >> 16. [x] update docs >> >> I'm going to announce 0.17 once the site PRs get merged. >> > >> > >> > Thanks, >> > -- >> > kou >> > >> > In >> > "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 >> 23:20:48 +0200, >> > Krisztián Szűcs wrote: >> > >> > > On Mon, Apr 20, 2020 at 11:17 PM Andy Grove >> wrote: >> > >> >> > >> Ok, I can look into this after work today (in about 3 hours). >> > > Great, thanks! >> > > >> > > The current status is (`x` means done): >> > > >> > > 1. [x] rebase >> > > 2. [x] upload source >> > > 3. [x] upload binaries >> > > 4. [x] update website >> > > 5. [x] upload ruby gems >> > > 6. [x] upload js packages >> > > 8. [ ] upload C# crates >> > > 9. [Andy] upload rust crates >> > > 10. [ ] update conda recipes >> > > 11. [x] upload wheels to pypi >> > > 12. [Neal] update homebrew packages >> > > 13. [x] update maven artifacts >> > > 14. [ ] update msys2 >> > > 15. [Neal] update R packages >> > > 16. [Krisztian] update docs >> > >> >> > >> On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs < >> szucs.kriszt...@gmail.com> >> > >> wrote: >> > >> >> > >> > Thanks Andy! I tried to upload the rust packages but arrow-flight, >> > >> > but a version pin is missing from the package tree: >> > >> > >> > >> > error: all dependencies must have a version specified when >> publishing. >> > >> > dependency `arrow-flight` does not specify a version >> > >> > >> > >> > Please upload the packages! >> > >> > >> > >> > Also added Uwe and Kou to the package owners. >> > >> > >> > >> > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove >> wrote: >> > >> > > >> > >> > > You should have an invite for the arrow-flight crate. Please >> check >> > >> > > https://crates.io/me/pending-invites >> > >> > > >> > >> > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < >> > >> > szucs.kriszt...@gmail.com> >> > >> > > wrote: >> > >> > > >> > >> > > > 1. [x] rebase >> > >> > > > 2. [x] upload source >> > >> > > > 3. [x] upload binaries >> > >> > > > 4. [x] update website >> > >> > > > 5. [x] upload ruby gems >> > >> > > > 6. [x] upload js packages >> > >> > > > 8. [ ] upload C# crates >> > >> > > > 9. [ ] upload rust crates >> > >> > > > 10. [ ] update conda recipes >> > >> > > > 11. [-] upload wheels to pypi >> > >> > > > 12. [ ] update homebrew packages >> > >> > > > 13. [-] update maven artifacts >> > >> > > > 14. [ ] update msys2 >> > >> > > > 15. [ ] update R packages >> > >> > > > >> > >> > > > @Andy Grove please grant me access to the arrow-flight crate >> > >> > > > @Neal Richardson please update the R package >> > >> > > > >> > >> > > > Currently I'm handling maven and pypi artifacts. >> > >> > > > >> > >> > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs >> > >> > > > wrote: >> > >> > > > > >> > >> > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding >> +1 vote. >> > >> > > > > >> > >> > > > > I'm starting the post-release tasks. I'm going to send >> updates to >> > >> > this >> > >> > > > > thread to coordinate if anyone wants to help. >> > >> > > > > >> > >> > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc < >> rok.mih...@gmail.com> >> > >> > wrote: >> > >> > > > > > >> > >> > > > > > +1 (non-binding) >> > >> > > > > > >> > >> > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went >> fine. >> > >> > > > > > >> > >> > > > > > Rok >> > >> > > > > > >> > >> > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < >> > >> > > > szucs.kriszt...@gmail.com> >> > >> > > > > > wrote: >> > >> > >
[jira] [Created] (ARROW-8536) [Rust] Failed to locate format/Flight.proto in any parent directory
Andy Grove created ARROW-8536: - Summary: [Rust] Failed to locate format/Flight.proto in any parent directory Key: ARROW-8536 URL: https://issues.apache.org/jira/browse/ARROW-8536 Project: Apache Arrow Issue Type: Bug Components: Rust Affects Versions: 0.17.0 Reporter: Andy Grove When using Arrow 0.17.0 as a dependency, it is likely that you will get the error "Failed to locate format/Flight.proto in any parent directory". The workaround is to create a directoy `/format` in the root of your file system and place the Flight.proto file there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [Java] Memory Allocation Tips
Hi Razvan, Arrow Java is based on off-heap memory. So it does not rely on GC. Some of the recommended best practice can be found in https://arrow.apache.org/docs/java/vector.html Best, Liya Fan On Mon, Apr 20, 2020 at 8:05 PM Razvan Chitu wrote: > Hi, > > Does the Arrow community have any tips / recommendations / best practices > on how to manage Arrow memory in Java? Is there a way to rely on the GC > exclusively (i.e. is there support for heap-only allocation)? > > Best, > Razvan >
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
I've run into issues publishing the Rust crates and I don't think I can resolve this tonight. I am documenting the issue in https://issues.apache.org/jira/browse/ARROW-8535 On Mon, Apr 20, 2020 at 5:02 PM Krisztián Szűcs wrote: > Created a PR with updated docs. > > Conda post release task is left, it's a bit strange that the conda-forge > autotick bot has not created the version bump PRs yet. I'm updating > them manually tomorrow. > > 1. [x] rebase > 2. [x] upload source > 3. [x] upload binaries > 4. [x] update website > 5. [x] upload ruby gems > 6. [x] upload js packages > 8. [x] upload C# packages > 9. [Andy] upload rust crates > 10. [ ] update conda recipes > 11. [x] upload wheels to pypi > 12. [Neal] update homebrew packages > 13. [x] update maven artifacts > 14. [kou] update msys2 > 15. [Neal] update R packages > 16. [x] update docs > > I'm going to announce 0.17 once the site PRs get merged. > > > > > > Thanks, > > -- > > kou > > > > In > > "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 > 23:20:48 +0200, > > Krisztián Szűcs wrote: > > > > > On Mon, Apr 20, 2020 at 11:17 PM Andy Grove > wrote: > > >> > > >> Ok, I can look into this after work today (in about 3 hours). > > > Great, thanks! > > > > > > The current status is (`x` means done): > > > > > > 1. [x] rebase > > > 2. [x] upload source > > > 3. [x] upload binaries > > > 4. [x] update website > > > 5. [x] upload ruby gems > > > 6. [x] upload js packages > > > 8. [ ] upload C# crates > > > 9. [Andy] upload rust crates > > > 10. [ ] update conda recipes > > > 11. [x] upload wheels to pypi > > > 12. [Neal] update homebrew packages > > > 13. [x] update maven artifacts > > > 14. [ ] update msys2 > > > 15. [Neal] update R packages > > > 16. [Krisztian] update docs > > >> > > >> On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs < > szucs.kriszt...@gmail.com> > > >> wrote: > > >> > > >> > Thanks Andy! I tried to upload the rust packages but arrow-flight, > > >> > but a version pin is missing from the package tree: > > >> > > > >> > error: all dependencies must have a version specified when > publishing. > > >> > dependency `arrow-flight` does not specify a version > > >> > > > >> > Please upload the packages! > > >> > > > >> > Also added Uwe and Kou to the package owners. > > >> > > > >> > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove > wrote: > > >> > > > > >> > > You should have an invite for the arrow-flight crate. Please check > > >> > > https://crates.io/me/pending-invites > > >> > > > > >> > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < > > >> > szucs.kriszt...@gmail.com> > > >> > > wrote: > > >> > > > > >> > > > 1. [x] rebase > > >> > > > 2. [x] upload source > > >> > > > 3. [x] upload binaries > > >> > > > 4. [x] update website > > >> > > > 5. [x] upload ruby gems > > >> > > > 6. [x] upload js packages > > >> > > > 8. [ ] upload C# crates > > >> > > > 9. [ ] upload rust crates > > >> > > > 10. [ ] update conda recipes > > >> > > > 11. [-] upload wheels to pypi > > >> > > > 12. [ ] update homebrew packages > > >> > > > 13. [-] update maven artifacts > > >> > > > 14. [ ] update msys2 > > >> > > > 15. [ ] update R packages > > >> > > > > > >> > > > @Andy Grove please grant me access to the arrow-flight crate > > >> > > > @Neal Richardson please update the R package > > >> > > > > > >> > > > Currently I'm handling maven and pypi artifacts. > > >> > > > > > >> > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs > > >> > > > wrote: > > >> > > > > > > >> > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 > vote. > > >> > > > > > > >> > > > > I'm starting the post-release tasks. I'm going to send > updates to > > >> > this > > >> > > > > thread to coordinate if anyone wants to help. > > >> > > > > > > >> > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc < > rok.mih...@gmail.com> > > >> > wrote: > > >> > > > > > > > >> > > > > > +1 (non-binding) > > >> > > > > > > > >> > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went > fine. > > >> > > > > > > > >> > > > > > Rok > > >> > > > > > > > >> > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < > > >> > > > szucs.kriszt...@gmail.com> > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > So far we're having 7 +1 binding votes. I'm planning to > close the > > >> > > > > > > vote later today, I anyone else would like to verify the > release > > >> > > > > > > please do it today. > > >> > > > > > > > > >> > > > > > > Thanks, Krisztian > > >> > > > > > > > > >> > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney < > > >> > wesmck...@gmail.com> > > >> > > > wrote: > > >> > > > > > > > > > >> > > > > > > > +1 (binding) > > >> > > > > > > > > > >> > > > > > > > I was able to run the verification of source and wheels > on > > >> > Windows > > >> > > > > > > > after addressing the issues I ran into. The failure > reported in > > >> > > > > > > > ARROW-8510 is worked around by using the UNITY build so >
[jira] [Created] (ARROW-8535) [Rust] Fix issues discovered when releasing 0.17.0
Andy Grove created ARROW-8535:
-
Summary: [Rust] Fix issues discovered when releasing 0.17.0
Key: ARROW-8535
URL: https://issues.apache.org/jira/browse/ARROW-8535
Project: Apache Arrow
Issue Type: Bug
Components: Rust
Affects Versions: 0.17.0
Reporter: Andy Grove
Assignee: Andy Grove
Fix For: 1.0.0
Issues ...
1) Arrow Cargo.toml does not specify a version for Arrow Flight. This is
trivial to fix, we just need to add the "version =" part.
2) "Failed to locate format/Flight.proto in any parent directory" when
publishing Arrow crate
{code:java}
error: failed to run custom build command for `arrow-flight v0.17.0`Caused by:
process didn't exit successfully:
`/home/andy/apache-arrow-0.17.0/rust/target/package/arrow-0.17.0/target/debug/build/arrow-flight-33d3d930b565975b/build-script-build`
(exit code: 1)
--- stderr
Error: "Failed to locate format/Flight.proto in any parent directory"warning:
build failed, waiting for other jobs to finish...
error: failed to verify package tarballCaused by:
build failed
{code}
I'm not sure how to resolve this yet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
Created a PR with updated docs. Conda post release task is left, it's a bit strange that the conda-forge autotick bot has not created the version bump PRs yet. I'm updating them manually tomorrow. 1. [x] rebase 2. [x] upload source 3. [x] upload binaries 4. [x] update website 5. [x] upload ruby gems 6. [x] upload js packages 8. [x] upload C# packages 9. [Andy] upload rust crates 10. [ ] update conda recipes 11. [x] upload wheels to pypi 12. [Neal] update homebrew packages 13. [x] update maven artifacts 14. [kou] update msys2 15. [Neal] update R packages 16. [x] update docs I'm going to announce 0.17 once the site PRs get merged. > > > Thanks, > -- > kou > > In > "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 23:20:48 > +0200, > Krisztián Szűcs wrote: > > > On Mon, Apr 20, 2020 at 11:17 PM Andy Grove wrote: > >> > >> Ok, I can look into this after work today (in about 3 hours). > > Great, thanks! > > > > The current status is (`x` means done): > > > > 1. [x] rebase > > 2. [x] upload source > > 3. [x] upload binaries > > 4. [x] update website > > 5. [x] upload ruby gems > > 6. [x] upload js packages > > 8. [ ] upload C# crates > > 9. [Andy] upload rust crates > > 10. [ ] update conda recipes > > 11. [x] upload wheels to pypi > > 12. [Neal] update homebrew packages > > 13. [x] update maven artifacts > > 14. [ ] update msys2 > > 15. [Neal] update R packages > > 16. [Krisztian] update docs > >> > >> On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs > >> wrote: > >> > >> > Thanks Andy! I tried to upload the rust packages but arrow-flight, > >> > but a version pin is missing from the package tree: > >> > > >> > error: all dependencies must have a version specified when publishing. > >> > dependency `arrow-flight` does not specify a version > >> > > >> > Please upload the packages! > >> > > >> > Also added Uwe and Kou to the package owners. > >> > > >> > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove > >> > wrote: > >> > > > >> > > You should have an invite for the arrow-flight crate. Please check > >> > > https://crates.io/me/pending-invites > >> > > > >> > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < > >> > szucs.kriszt...@gmail.com> > >> > > wrote: > >> > > > >> > > > 1. [x] rebase > >> > > > 2. [x] upload source > >> > > > 3. [x] upload binaries > >> > > > 4. [x] update website > >> > > > 5. [x] upload ruby gems > >> > > > 6. [x] upload js packages > >> > > > 8. [ ] upload C# crates > >> > > > 9. [ ] upload rust crates > >> > > > 10. [ ] update conda recipes > >> > > > 11. [-] upload wheels to pypi > >> > > > 12. [ ] update homebrew packages > >> > > > 13. [-] update maven artifacts > >> > > > 14. [ ] update msys2 > >> > > > 15. [ ] update R packages > >> > > > > >> > > > @Andy Grove please grant me access to the arrow-flight crate > >> > > > @Neal Richardson please update the R package > >> > > > > >> > > > Currently I'm handling maven and pypi artifacts. > >> > > > > >> > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs > >> > > > wrote: > >> > > > > > >> > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. > >> > > > > > >> > > > > I'm starting the post-release tasks. I'm going to send updates to > >> > this > >> > > > > thread to coordinate if anyone wants to help. > >> > > > > > >> > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc > >> > wrote: > >> > > > > > > >> > > > > > +1 (non-binding) > >> > > > > > > >> > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. > >> > > > > > > >> > > > > > Rok > >> > > > > > > >> > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < > >> > > > szucs.kriszt...@gmail.com> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > So far we're having 7 +1 binding votes. I'm planning to close > >> > > > > > > the > >> > > > > > > vote later today, I anyone else would like to verify the > >> > > > > > > release > >> > > > > > > please do it today. > >> > > > > > > > >> > > > > > > Thanks, Krisztian > >> > > > > > > > >> > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney < > >> > wesmck...@gmail.com> > >> > > > wrote: > >> > > > > > > > > >> > > > > > > > +1 (binding) > >> > > > > > > > > >> > > > > > > > I was able to run the verification of source and wheels on > >> > Windows > >> > > > > > > > after addressing the issues I ran into. The failure reported > >> > > > > > > > in > >> > > > > > > > ARROW-8510 is worked around by using the UNITY build so we'll > >> > need > >> > > > to > >> > > > > > > > investigate that separately > >> > > > > > > > > >> > > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney < > >> > wesmck...@gmail.com> > >> > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > Right, sorry about that. > >> > > > > > > > > > >> > > > > > > > > So the difference it turns out is the UNITY build. The > >> > compiler > >> > > > error > >> > > > > > > > > in ARROW-8510 does not occur for some reason when > >> > > > > > > > >
[jira] [Created] (ARROW-8534) [C++][CSV] Issue building CSV component under GCC 6.1.0
Ross Wolfson created ARROW-8534:
---
Summary: [C++][CSV] Issue building CSV component under GCC 6.1.0
Key: ARROW-8534
URL: https://issues.apache.org/jira/browse/ARROW-8534
Project: Apache Arrow
Issue Type: Bug
Components: C++
Affects Versions: 0.17.0
Environment: Centos 7 x86_64
Reporter: Ross Wolfson
Hi,
In the current version (0.17.0), it seems that CSV reader.cc fails to compile
when using GCC 6.1.0. This builds when using older or newer GCC versions (we
tested with 4.8.5, 8.2.0 and 9.3.0).
{{[root@1d4fcfc2580e arrow_src]# /ourgcc/gcc-6.1.0/bin/g++ -c
cpp/src/arrow/csv/reader.cc -I cpp/src}}
{{cpp/src/arrow/csv/reader.cc: In constructor
'arrow::csv::SerialBlockReader::SerialBlockReader(std::unique_ptr,
arrow::Iterator >,
std::shared_ptr)':}}
{{cpp/src/arrow/csv/reader.cc:178:22: error: use of deleted function
'std::unique_ptr<_Tp, _Dp>::unique_ptr(const std::unique_ptr<_Tp, _Dp>&) [with
_Tp = arrow::Chunker; _Dp = std::default_delete]'}}
{{ using BlockReader::BlockReader;}}
{{ ^~~}}
{{In file included from /ourgcc/gcc-6.1.0/include/c++/6.1.0/memory:81:0,}}
{{ from cpp/src/arrow/csv/reader.h:20,}}
{{ from cpp/src/arrow/csv/reader.cc:18:}}
{{/ourgcc/gcc-6.1.0/include/c++/6.1.0/bits/unique_ptr.h:356:7: note: declared
here}}
{{ unique_ptr(const unique_ptr&) = delete;}}
{{ ^~}}
{{cpp/src/arrow/csv/reader.cc:178:22: error: use of deleted function
'arrow::Iterator >::Iterator(const
arrow::Iterator >&)'}}
{{ using BlockReader::BlockReader;}}
{{ ^~~}}
{{In file included from cpp/src/arrow/csv/reader.cc:43:0:}}
{{cpp/src/arrow/util/iterator.h:63:7: note:
'arrow::Iterator >::Iterator(const
arrow::Iterator >&)' is implicitly deleted
because the default definition would be ill-formed:}}
{{ class Iterator : public util::EqualityComparable> {}}
{{ ^~~~}}
{{cpp/src/arrow/util/iterator.h:63:7: error: use of deleted function
'std::unique_ptr<_Tp, _Dp>::unique_ptr(const std::unique_ptr<_Tp, _Dp>&) [with
_Tp = void; _Dp = void (*)(void*)]'}}
{{In file included from /ourgcc/gcc-6.1.0/include/c++/6.1.0/memory:81:0,}}
{{ from cpp/src/arrow/csv/reader.h:20,}}
{{ from cpp/src/arrow/csv/reader.cc:18:}}
{{/ourgcc/gcc-6.1.0/include/c++/6.1.0/bits/unique_ptr.h:356:7: note: declared
here}}
{{ unique_ptr(const unique_ptr&) = delete;}}
{{ ^~}}
{{cpp/src/arrow/csv/reader.cc: In member function 'virtual
arrow::Result >
arrow::csv::SerialTableReader::Read()':}}
{{cpp/src/arrow/csv/reader.cc:750:88: note: synthesized method
'arrow::csv::SerialBlockReader::SerialBlockReader(std::unique_ptr,
arrow::Iterator >,
std::shared_ptr)' first required here}}
{{ std::move(buffer_iterator_), std::move(first_buffer));}}
{{ ^}}
{{cpp/src/arrow/csv/reader.cc: In constructor
'arrow::csv::ThreadedBlockReader::ThreadedBlockReader(std::unique_ptr,
arrow::Iterator >,
std::shared_ptr)':}}
{{cpp/src/arrow/csv/reader.cc:221:22: error: use of deleted function
'std::unique_ptr<_Tp, _Dp>::unique_ptr(const std::unique_ptr<_Tp, _Dp>&) [with
_Tp = arrow::Chunker; _Dp = std::default_delete]'}}
{{ using BlockReader::BlockReader;}}
{{ ^~~}}
{{In file included from /ourgcc/gcc-6.1.0/include/c++/6.1.0/memory:81:0,}}
{{ from cpp/src/arrow/csv/reader.h:20,}}
{{ from cpp/src/arrow/csv/reader.cc:18:}}
{{/ourgcc/gcc-6.1.0/include/c++/6.1.0/bits/unique_ptr.h:356:7: note: declared
here}}
{{ unique_ptr(const unique_ptr&) = delete;}}
{{ ^~}}
{{cpp/src/arrow/csv/reader.cc:221:22: error: use of deleted function
'arrow::Iterator >::Iterator(const
arrow::Iterator >&)'}}
{{ using BlockReader::BlockReader;}}
{{ ^~~}}
{{cpp/src/arrow/csv/reader.cc: In member function 'virtual
arrow::Result >
arrow::csv::ThreadedTableReader::Read()':}}
{{cpp/src/arrow/csv/reader.cc:815:61: note: synthesized method
'arrow::csv::ThreadedBlockReader::ThreadedBlockReader(std::unique_ptr,
arrow::Iterator >,
std::shared_ptr)' first required here}}
{{ std::move(first_buffer));}}
{{ ^}}
My colleague found a workaround that avoids the build error, however, we are
not clear if this is the "best" fix.
{{--- a/cpp/src/arrow/csv/reader.cc}}
{{+++ b/cpp/src/arrow/csv/reader.cc}}
{{@@ -175,7 +175,12 @@ class BlockReader {}}
{{ // using CSVBlock::consume_bytes.}}
{{ class SerialBlockReader : public BlockReader {}}
{{ public:}}
{{- using BlockReader::BlockReader;}}
{{+ SerialBlockReader(std::unique_ptr chunker,}}
{{+ Iterator> buffer_iterator,}}
{{+ std::shared_ptr first_buffer)}}
{{+ : BlockReader(std::move(chunker),}}
{{+ std::move(buffer_iterator),}}
{{+ std::move(first_buffer)) {}}}{{Result>
Next() {}}
{{ if (buffer_ == nullptr) {}}
{{@@ -218,7 +223,12 @@ class SerialBlockReader : public BlockReader {}}
{{ // An object that reads delimited CSV blocks for threaded use.}}
{{ class ThreadedBlockReader : public BlockReader {}}
{{ public:}}
{{- using BlockReader::BlockReader;}}
[jira] [Created] (ARROW-8533) [Release] Don't commit doctrees in the docs post release script
Krisztian Szucs created ARROW-8533: -- Summary: [Release] Don't commit doctrees in the docs post release script Key: ARROW-8533 URL: https://issues.apache.org/jira/browse/ARROW-8533 Project: Apache Arrow Issue Type: Improvement Components: Developer Tools Reporter: Krisztian Szucs A .gitignore file would be enough to prevent committing pickled binaries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
Hi, I've uploaded C# packages. I'll also work on MSYS2: The current status is (`x` means done): 1. [x] rebase 2. [x] upload source 3. [x] upload binaries 4. [x] update website 5. [x] upload ruby gems 6. [x] upload js packages 8. [x] upload C# packages 9. [Andy] upload rust crates 10. [ ] update conda recipes 11. [x] upload wheels to pypi 12. [Neal] update homebrew packages 13. [x] update maven artifacts 14. [kou] update msys2 15. [Neal] update R packages 16. [Krisztian] update docs Thanks, -- kou In "Re: [VOTE] Release Apache Arrow 0.17.0 - RC0" on Mon, 20 Apr 2020 23:20:48 +0200, Krisztián Szűcs wrote: > On Mon, Apr 20, 2020 at 11:17 PM Andy Grove wrote: >> >> Ok, I can look into this after work today (in about 3 hours). > Great, thanks! > > The current status is (`x` means done): > > 1. [x] rebase > 2. [x] upload source > 3. [x] upload binaries > 4. [x] update website > 5. [x] upload ruby gems > 6. [x] upload js packages > 8. [ ] upload C# crates > 9. [Andy] upload rust crates > 10. [ ] update conda recipes > 11. [x] upload wheels to pypi > 12. [Neal] update homebrew packages > 13. [x] update maven artifacts > 14. [ ] update msys2 > 15. [Neal] update R packages > 16. [Krisztian] update docs >> >> On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs >> wrote: >> >> > Thanks Andy! I tried to upload the rust packages but arrow-flight, >> > but a version pin is missing from the package tree: >> > >> > error: all dependencies must have a version specified when publishing. >> > dependency `arrow-flight` does not specify a version >> > >> > Please upload the packages! >> > >> > Also added Uwe and Kou to the package owners. >> > >> > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove wrote: >> > > >> > > You should have an invite for the arrow-flight crate. Please check >> > > https://crates.io/me/pending-invites >> > > >> > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < >> > szucs.kriszt...@gmail.com> >> > > wrote: >> > > >> > > > 1. [x] rebase >> > > > 2. [x] upload source >> > > > 3. [x] upload binaries >> > > > 4. [x] update website >> > > > 5. [x] upload ruby gems >> > > > 6. [x] upload js packages >> > > > 8. [ ] upload C# crates >> > > > 9. [ ] upload rust crates >> > > > 10. [ ] update conda recipes >> > > > 11. [-] upload wheels to pypi >> > > > 12. [ ] update homebrew packages >> > > > 13. [-] update maven artifacts >> > > > 14. [ ] update msys2 >> > > > 15. [ ] update R packages >> > > > >> > > > @Andy Grove please grant me access to the arrow-flight crate >> > > > @Neal Richardson please update the R package >> > > > >> > > > Currently I'm handling maven and pypi artifacts. >> > > > >> > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs >> > > > wrote: >> > > > > >> > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. >> > > > > >> > > > > I'm starting the post-release tasks. I'm going to send updates to >> > this >> > > > > thread to coordinate if anyone wants to help. >> > > > > >> > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc >> > wrote: >> > > > > > >> > > > > > +1 (non-binding) >> > > > > > >> > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. >> > > > > > >> > > > > > Rok >> > > > > > >> > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < >> > > > szucs.kriszt...@gmail.com> >> > > > > > wrote: >> > > > > > >> > > > > > > So far we're having 7 +1 binding votes. I'm planning to close the >> > > > > > > vote later today, I anyone else would like to verify the release >> > > > > > > please do it today. >> > > > > > > >> > > > > > > Thanks, Krisztian >> > > > > > > >> > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney < >> > wesmck...@gmail.com> >> > > > wrote: >> > > > > > > > >> > > > > > > > +1 (binding) >> > > > > > > > >> > > > > > > > I was able to run the verification of source and wheels on >> > Windows >> > > > > > > > after addressing the issues I ran into. The failure reported in >> > > > > > > > ARROW-8510 is worked around by using the UNITY build so we'll >> > need >> > > > to >> > > > > > > > investigate that separately >> > > > > > > > >> > > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney < >> > wesmck...@gmail.com> >> > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > Right, sorry about that. >> > > > > > > > > >> > > > > > > > > So the difference it turns out is the UNITY build. The >> > compiler >> > > > error >> > > > > > > > > in ARROW-8510 does not occur for some reason when >> > > > > > > > > CMAKE_UNITY_BUILD=ON. >> > > > > > > > > >> > > > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou < >> > > > anto...@python.org> >> > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. >> > > > > > > > > > >> > > > > > > > > > Regards >> > > > > > > > > > >> > > > > > > > > > Antoine. >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
On Mon, Apr 20, 2020 at 11:17 PM Andy Grove wrote: > > Ok, I can look into this after work today (in about 3 hours). Great, thanks! The current status is (`x` means done): 1. [x] rebase 2. [x] upload source 3. [x] upload binaries 4. [x] update website 5. [x] upload ruby gems 6. [x] upload js packages 8. [ ] upload C# crates 9. [Andy] upload rust crates 10. [ ] update conda recipes 11. [x] upload wheels to pypi 12. [Neal] update homebrew packages 13. [x] update maven artifacts 14. [ ] update msys2 15. [Neal] update R packages 16. [Krisztian] update docs > > On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs > wrote: > > > Thanks Andy! I tried to upload the rust packages but arrow-flight, > > but a version pin is missing from the package tree: > > > > error: all dependencies must have a version specified when publishing. > > dependency `arrow-flight` does not specify a version > > > > Please upload the packages! > > > > Also added Uwe and Kou to the package owners. > > > > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove wrote: > > > > > > You should have an invite for the arrow-flight crate. Please check > > > https://crates.io/me/pending-invites > > > > > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < > > szucs.kriszt...@gmail.com> > > > wrote: > > > > > > > 1. [x] rebase > > > > 2. [x] upload source > > > > 3. [x] upload binaries > > > > 4. [x] update website > > > > 5. [x] upload ruby gems > > > > 6. [x] upload js packages > > > > 8. [ ] upload C# crates > > > > 9. [ ] upload rust crates > > > > 10. [ ] update conda recipes > > > > 11. [-] upload wheels to pypi > > > > 12. [ ] update homebrew packages > > > > 13. [-] update maven artifacts > > > > 14. [ ] update msys2 > > > > 15. [ ] update R packages > > > > > > > > @Andy Grove please grant me access to the arrow-flight crate > > > > @Neal Richardson please update the R package > > > > > > > > Currently I'm handling maven and pypi artifacts. > > > > > > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs > > > > wrote: > > > > > > > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. > > > > > > > > > > I'm starting the post-release tasks. I'm going to send updates to > > this > > > > > thread to coordinate if anyone wants to help. > > > > > > > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc > > wrote: > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. > > > > > > > > > > > > Rok > > > > > > > > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < > > > > szucs.kriszt...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > So far we're having 7 +1 binding votes. I'm planning to close the > > > > > > > vote later today, I anyone else would like to verify the release > > > > > > > please do it today. > > > > > > > > > > > > > > Thanks, Krisztian > > > > > > > > > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney < > > wesmck...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > I was able to run the verification of source and wheels on > > Windows > > > > > > > > after addressing the issues I ran into. The failure reported in > > > > > > > > ARROW-8510 is worked around by using the UNITY build so we'll > > need > > > > to > > > > > > > > investigate that separately > > > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney < > > wesmck...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Right, sorry about that. > > > > > > > > > > > > > > > > > > So the difference it turns out is the UNITY build. The > > compiler > > > > error > > > > > > > > > in ARROW-8510 does not occur for some reason when > > > > > > > > > CMAKE_UNITY_BUILD=ON. > > > > > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou < > > > > anto...@python.org> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > > > > > > > > The release verification script (which uses the VS 2017 > > > > compiler > > > > > > > mode) > > > > > > > > > > > fails for me on Windows > > > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > > > > > > > > > > > > > > > It looks like we are building with VS 2019 in CI. We may > > > > want to be > > > > > > > > > > > more conservative and build with VS 2015 or VS 2017 in > > CI. > > > > This is > > > > > > > > > > > peculiar though because the wheels are using VS 2015. So > > it's > > > > > > > possible > > > > > > > > > > > there's a compiler bug with VS 2017 that doesn't present > > on > > > > VS > > > > > > > 2015. > > > > > > > > > > > > > > > > > > > > > > I don't think this should
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
Ok, I can look into this after work today (in about 3 hours). On Mon, Apr 20, 2020, 2:47 PM Krisztián Szűcs wrote: > Thanks Andy! I tried to upload the rust packages but arrow-flight, > but a version pin is missing from the package tree: > > error: all dependencies must have a version specified when publishing. > dependency `arrow-flight` does not specify a version > > Please upload the packages! > > Also added Uwe and Kou to the package owners. > > On Mon, Apr 20, 2020 at 10:24 PM Andy Grove wrote: > > > > You should have an invite for the arrow-flight crate. Please check > > https://crates.io/me/pending-invites > > > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs < > szucs.kriszt...@gmail.com> > > wrote: > > > > > 1. [x] rebase > > > 2. [x] upload source > > > 3. [x] upload binaries > > > 4. [x] update website > > > 5. [x] upload ruby gems > > > 6. [x] upload js packages > > > 8. [ ] upload C# crates > > > 9. [ ] upload rust crates > > > 10. [ ] update conda recipes > > > 11. [-] upload wheels to pypi > > > 12. [ ] update homebrew packages > > > 13. [-] update maven artifacts > > > 14. [ ] update msys2 > > > 15. [ ] update R packages > > > > > > @Andy Grove please grant me access to the arrow-flight crate > > > @Neal Richardson please update the R package > > > > > > Currently I'm handling maven and pypi artifacts. > > > > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs > > > wrote: > > > > > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. > > > > > > > > I'm starting the post-release tasks. I'm going to send updates to > this > > > > thread to coordinate if anyone wants to help. > > > > > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc > wrote: > > > > > > > > > > +1 (non-binding) > > > > > > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. > > > > > > > > > > Rok > > > > > > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < > > > szucs.kriszt...@gmail.com> > > > > > wrote: > > > > > > > > > > > So far we're having 7 +1 binding votes. I'm planning to close the > > > > > > vote later today, I anyone else would like to verify the release > > > > > > please do it today. > > > > > > > > > > > > Thanks, Krisztian > > > > > > > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney < > wesmck...@gmail.com> > > > wrote: > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > I was able to run the verification of source and wheels on > Windows > > > > > > > after addressing the issues I ran into. The failure reported in > > > > > > > ARROW-8510 is worked around by using the UNITY build so we'll > need > > > to > > > > > > > investigate that separately > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney < > wesmck...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > Right, sorry about that. > > > > > > > > > > > > > > > > So the difference it turns out is the UNITY build. The > compiler > > > error > > > > > > > > in ARROW-8510 does not occur for some reason when > > > > > > > > CMAKE_UNITY_BUILD=ON. > > > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou < > > > anto...@python.org> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > > > > > > > The release verification script (which uses the VS 2017 > > > compiler > > > > > > mode) > > > > > > > > > > fails for me on Windows > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > > > > > > > > > > > > > It looks like we are building with VS 2019 in CI. We may > > > want to be > > > > > > > > > > more conservative and build with VS 2015 or VS 2017 in > CI. > > > This is > > > > > > > > > > peculiar though because the wheels are using VS 2015. So > it's > > > > > > possible > > > > > > > > > > there's a compiler bug with VS 2017 that doesn't present > on > > > VS > > > > > > 2015. > > > > > > > > > > > > > > > > > > > > I don't think this should block the release however. I'll > > > vote > > > > > > after I > > > > > > > > > > check some more things > > > > > > > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:32 PM Neal Richardson > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >> +1 (binding) > > > > > > > > > >> > > > > > > > > > >> I ran verification on Ubuntu, macOS, and Windows using > > > crossbow. > > > > > > Thanks to > > > > > > > > > >> all who helped debug. A few things are still failing: > see > > > > > > > > > >> > > > https://github.com/apache/arrow/pull/6961#issuecomment-615940240 > > > > > > for > > > > > > > > > >> details. Since Python folks are unconcerned by the wheel > > > issues I > > > > > > > > > >> encountered, I won't object. > > > > > > > > > >> > > > >
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
Thanks Andy! I tried to upload the rust packages but arrow-flight, but a version pin is missing from the package tree: error: all dependencies must have a version specified when publishing. dependency `arrow-flight` does not specify a version Please upload the packages! Also added Uwe and Kou to the package owners. On Mon, Apr 20, 2020 at 10:24 PM Andy Grove wrote: > > You should have an invite for the arrow-flight crate. Please check > https://crates.io/me/pending-invites > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs > wrote: > > > 1. [x] rebase > > 2. [x] upload source > > 3. [x] upload binaries > > 4. [x] update website > > 5. [x] upload ruby gems > > 6. [x] upload js packages > > 8. [ ] upload C# crates > > 9. [ ] upload rust crates > > 10. [ ] update conda recipes > > 11. [-] upload wheels to pypi > > 12. [ ] update homebrew packages > > 13. [-] update maven artifacts > > 14. [ ] update msys2 > > 15. [ ] update R packages > > > > @Andy Grove please grant me access to the arrow-flight crate > > @Neal Richardson please update the R package > > > > Currently I'm handling maven and pypi artifacts. > > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs > > wrote: > > > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. > > > > > > I'm starting the post-release tasks. I'm going to send updates to this > > > thread to coordinate if anyone wants to help. > > > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc wrote: > > > > > > > > +1 (non-binding) > > > > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. > > > > > > > > Rok > > > > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < > > szucs.kriszt...@gmail.com> > > > > wrote: > > > > > > > > > So far we're having 7 +1 binding votes. I'm planning to close the > > > > > vote later today, I anyone else would like to verify the release > > > > > please do it today. > > > > > > > > > > Thanks, Krisztian > > > > > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney > > wrote: > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > I was able to run the verification of source and wheels on Windows > > > > > > after addressing the issues I ran into. The failure reported in > > > > > > ARROW-8510 is worked around by using the UNITY build so we'll need > > to > > > > > > investigate that separately > > > > > > > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney > > > > > wrote: > > > > > > > > > > > > > > Right, sorry about that. > > > > > > > > > > > > > > So the difference it turns out is the UNITY build. The compiler > > error > > > > > > > in ARROW-8510 does not occur for some reason when > > > > > > > CMAKE_UNITY_BUILD=ON. > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou < > > anto...@python.org> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > > > > > > The release verification script (which uses the VS 2017 > > compiler > > > > > mode) > > > > > > > > > fails for me on Windows > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > > > > > > > > > > > It looks like we are building with VS 2019 in CI. We may > > want to be > > > > > > > > > more conservative and build with VS 2015 or VS 2017 in CI. > > This is > > > > > > > > > peculiar though because the wheels are using VS 2015. So it's > > > > > possible > > > > > > > > > there's a compiler bug with VS 2017 that doesn't present on > > VS > > > > > 2015. > > > > > > > > > > > > > > > > > > I don't think this should block the release however. I'll > > vote > > > > > after I > > > > > > > > > check some more things > > > > > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:32 PM Neal Richardson > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >> +1 (binding) > > > > > > > > >> > > > > > > > > >> I ran verification on Ubuntu, macOS, and Windows using > > crossbow. > > > > > Thanks to > > > > > > > > >> all who helped debug. A few things are still failing: see > > > > > > > > >> > > https://github.com/apache/arrow/pull/6961#issuecomment-615940240 > > > > > for > > > > > > > > >> details. Since Python folks are unconcerned by the wheel > > issues I > > > > > > > > >> encountered, I won't object. > > > > > > > > >> > > > > > > > > >> In addition to the usual verifications, I worked with > > Jeroen to > > > > > build the R > > > > > > > > >> Windows binaries for the 0.17 rc0 and confirmed that the R > > > > > package passes > > > > > > > > >> all checks with them. > > > > > > > > >> > > > > > > > > >> Neal > > > > > > > > >> > > > > > > > > >> On Sat, Apr 18, 2020 at 12:13 AM Sutou Kouhei < > > k...@clear-code.com> > > > > > wrote: > > > > > > > > >> > > > > > > > > >>> +1 (binding) > > > > > > >
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
I'm working on R and will also do Homebrew. On Mon, Apr 20, 2020 at 1:24 PM Andy Grove wrote: > You should have an invite for the arrow-flight crate. Please check > https://crates.io/me/pending-invites > > On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs > > wrote: > > > 1. [x] rebase > > 2. [x] upload source > > 3. [x] upload binaries > > 4. [x] update website > > 5. [x] upload ruby gems > > 6. [x] upload js packages > > 8. [ ] upload C# crates > > 9. [ ] upload rust crates > > 10. [ ] update conda recipes > > 11. [-] upload wheels to pypi > > 12. [ ] update homebrew packages > > 13. [-] update maven artifacts > > 14. [ ] update msys2 > > 15. [ ] update R packages > > > > @Andy Grove please grant me access to the arrow-flight crate > > @Neal Richardson please update the R package > > > > Currently I'm handling maven and pypi artifacts. > > > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs > > wrote: > > > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. > > > > > > I'm starting the post-release tasks. I'm going to send updates to this > > > thread to coordinate if anyone wants to help. > > > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc > wrote: > > > > > > > > +1 (non-binding) > > > > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. > > > > > > > > Rok > > > > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < > > szucs.kriszt...@gmail.com> > > > > wrote: > > > > > > > > > So far we're having 7 +1 binding votes. I'm planning to close the > > > > > vote later today, I anyone else would like to verify the release > > > > > please do it today. > > > > > > > > > > Thanks, Krisztian > > > > > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney > > > wrote: > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > I was able to run the verification of source and wheels on > Windows > > > > > > after addressing the issues I ran into. The failure reported in > > > > > > ARROW-8510 is worked around by using the UNITY build so we'll > need > > to > > > > > > investigate that separately > > > > > > > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney < > wesmck...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > Right, sorry about that. > > > > > > > > > > > > > > So the difference it turns out is the UNITY build. The compiler > > error > > > > > > > in ARROW-8510 does not occur for some reason when > > > > > > > CMAKE_UNITY_BUILD=ON. > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou < > > anto...@python.org> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > > > > > > The release verification script (which uses the VS 2017 > > compiler > > > > > mode) > > > > > > > > > fails for me on Windows > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > > > > > > > > > > > It looks like we are building with VS 2019 in CI. We may > > want to be > > > > > > > > > more conservative and build with VS 2015 or VS 2017 in CI. > > This is > > > > > > > > > peculiar though because the wheels are using VS 2015. So > it's > > > > > possible > > > > > > > > > there's a compiler bug with VS 2017 that doesn't present on > > VS > > > > > 2015. > > > > > > > > > > > > > > > > > > I don't think this should block the release however. I'll > > vote > > > > > after I > > > > > > > > > check some more things > > > > > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:32 PM Neal Richardson > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >> +1 (binding) > > > > > > > > >> > > > > > > > > >> I ran verification on Ubuntu, macOS, and Windows using > > crossbow. > > > > > Thanks to > > > > > > > > >> all who helped debug. A few things are still failing: see > > > > > > > > >> > > https://github.com/apache/arrow/pull/6961#issuecomment-615940240 > > > > > for > > > > > > > > >> details. Since Python folks are unconcerned by the wheel > > issues I > > > > > > > > >> encountered, I won't object. > > > > > > > > >> > > > > > > > > >> In addition to the usual verifications, I worked with > > Jeroen to > > > > > build the R > > > > > > > > >> Windows binaries for the 0.17 rc0 and confirmed that the R > > > > > package passes > > > > > > > > >> all checks with them. > > > > > > > > >> > > > > > > > > >> Neal > > > > > > > > >> > > > > > > > > >> On Sat, Apr 18, 2020 at 12:13 AM Sutou Kouhei < > > k...@clear-code.com> > > > > > wrote: > > > > > > > > >> > > > > > > > > >>> +1 (binding) > > > > > > > > >>> > > > > > > > > >>> I ran the followings on Debian GNU/Linux sid: > > > > > > > > >>> > > > > > > > > >>> * TEST_CSHARP=0 \ > > > > > > > > >>> JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ > > > > > > > > >>>
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
You should have an invite for the arrow-flight crate. Please check https://crates.io/me/pending-invites On Mon, Apr 20, 2020 at 2:10 PM Krisztián Szűcs wrote: > 1. [x] rebase > 2. [x] upload source > 3. [x] upload binaries > 4. [x] update website > 5. [x] upload ruby gems > 6. [x] upload js packages > 8. [ ] upload C# crates > 9. [ ] upload rust crates > 10. [ ] update conda recipes > 11. [-] upload wheels to pypi > 12. [ ] update homebrew packages > 13. [-] update maven artifacts > 14. [ ] update msys2 > 15. [ ] update R packages > > @Andy Grove please grant me access to the arrow-flight crate > @Neal Richardson please update the R package > > Currently I'm handling maven and pypi artifacts. > > On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs > wrote: > > > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. > > > > I'm starting the post-release tasks. I'm going to send updates to this > > thread to coordinate if anyone wants to help. > > > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc wrote: > > > > > > +1 (non-binding) > > > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. > > > > > > Rok > > > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs < > szucs.kriszt...@gmail.com> > > > wrote: > > > > > > > So far we're having 7 +1 binding votes. I'm planning to close the > > > > vote later today, I anyone else would like to verify the release > > > > please do it today. > > > > > > > > Thanks, Krisztian > > > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney > wrote: > > > > > > > > > > +1 (binding) > > > > > > > > > > I was able to run the verification of source and wheels on Windows > > > > > after addressing the issues I ran into. The failure reported in > > > > > ARROW-8510 is worked around by using the UNITY build so we'll need > to > > > > > investigate that separately > > > > > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney > > > > wrote: > > > > > > > > > > > > Right, sorry about that. > > > > > > > > > > > > So the difference it turns out is the UNITY build. The compiler > error > > > > > > in ARROW-8510 does not occur for some reason when > > > > > > CMAKE_UNITY_BUILD=ON. > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou < > anto...@python.org> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > > > > > The release verification script (which uses the VS 2017 > compiler > > > > mode) > > > > > > > > fails for me on Windows > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > > > > > > > > > It looks like we are building with VS 2019 in CI. We may > want to be > > > > > > > > more conservative and build with VS 2015 or VS 2017 in CI. > This is > > > > > > > > peculiar though because the wheels are using VS 2015. So it's > > > > possible > > > > > > > > there's a compiler bug with VS 2017 that doesn't present on > VS > > > > 2015. > > > > > > > > > > > > > > > > I don't think this should block the release however. I'll > vote > > > > after I > > > > > > > > check some more things > > > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:32 PM Neal Richardson > > > > > > > > wrote: > > > > > > > >> > > > > > > > >> +1 (binding) > > > > > > > >> > > > > > > > >> I ran verification on Ubuntu, macOS, and Windows using > crossbow. > > > > Thanks to > > > > > > > >> all who helped debug. A few things are still failing: see > > > > > > > >> > https://github.com/apache/arrow/pull/6961#issuecomment-615940240 > > > > for > > > > > > > >> details. Since Python folks are unconcerned by the wheel > issues I > > > > > > > >> encountered, I won't object. > > > > > > > >> > > > > > > > >> In addition to the usual verifications, I worked with > Jeroen to > > > > build the R > > > > > > > >> Windows binaries for the 0.17 rc0 and confirmed that the R > > > > package passes > > > > > > > >> all checks with them. > > > > > > > >> > > > > > > > >> Neal > > > > > > > >> > > > > > > > >> On Sat, Apr 18, 2020 at 12:13 AM Sutou Kouhei < > k...@clear-code.com> > > > > wrote: > > > > > > > >> > > > > > > > >>> +1 (binding) > > > > > > > >>> > > > > > > > >>> I ran the followings on Debian GNU/Linux sid: > > > > > > > >>> > > > > > > > >>> * TEST_CSHARP=0 \ > > > > > > > >>> JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ > > > > > > > >>> CUDA_TOOLKIT_ROOT=/usr \ > > > > > > > >>> ARROW_CMAKE_OPTIONS=-DgRPC_SOURCE=BUNDLED \ > > > > > > > >>> dev/release/verify-release-candidate.sh source > 0.17.0 0 > > > > > > > >>> * dev/release/verify-release-candidate.sh binaries > 0.17.0 0 > > > > > > > >>> * dev/release/verify-release-candidate.sh wheels 0.17.0 0 > > > > > > > >>> > > > > > > > >>> with: > > > > > > > >>> > > > > > > > >>> * gcc (Debian
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
1. [x] rebase 2. [x] upload source 3. [x] upload binaries 4. [x] update website 5. [x] upload ruby gems 6. [x] upload js packages 8. [ ] upload C# crates 9. [ ] upload rust crates 10. [ ] update conda recipes 11. [-] upload wheels to pypi 12. [ ] update homebrew packages 13. [-] update maven artifacts 14. [ ] update msys2 15. [ ] update R packages @Andy Grove please grant me access to the arrow-flight crate @Neal Richardson please update the R package Currently I'm handling maven and pypi artifacts. On Mon, Apr 20, 2020 at 8:27 PM Krisztián Szűcs wrote: > > The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. > > I'm starting the post-release tasks. I'm going to send updates to this > thread to coordinate if anyone wants to help. > > On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc wrote: > > > > +1 (non-binding) > > > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. > > > > Rok > > > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs > > wrote: > > > > > So far we're having 7 +1 binding votes. I'm planning to close the > > > vote later today, I anyone else would like to verify the release > > > please do it today. > > > > > > Thanks, Krisztian > > > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney wrote: > > > > > > > > +1 (binding) > > > > > > > > I was able to run the verification of source and wheels on Windows > > > > after addressing the issues I ran into. The failure reported in > > > > ARROW-8510 is worked around by using the UNITY build so we'll need to > > > > investigate that separately > > > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney > > > wrote: > > > > > > > > > > Right, sorry about that. > > > > > > > > > > So the difference it turns out is the UNITY build. The compiler error > > > > > in ARROW-8510 does not occur for some reason when > > > > > CMAKE_UNITY_BUILD=ON. > > > > > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou > > > wrote: > > > > > > > > > > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > > > > The release verification script (which uses the VS 2017 compiler > > > mode) > > > > > > > fails for me on Windows > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > > > > > > > It looks like we are building with VS 2019 in CI. We may want to > > > > > > > be > > > > > > > more conservative and build with VS 2015 or VS 2017 in CI. This is > > > > > > > peculiar though because the wheels are using VS 2015. So it's > > > possible > > > > > > > there's a compiler bug with VS 2017 that doesn't present on VS > > > 2015. > > > > > > > > > > > > > > I don't think this should block the release however. I'll vote > > > after I > > > > > > > check some more things > > > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:32 PM Neal Richardson > > > > > > > wrote: > > > > > > >> > > > > > > >> +1 (binding) > > > > > > >> > > > > > > >> I ran verification on Ubuntu, macOS, and Windows using crossbow. > > > Thanks to > > > > > > >> all who helped debug. A few things are still failing: see > > > > > > >> https://github.com/apache/arrow/pull/6961#issuecomment-615940240 > > > for > > > > > > >> details. Since Python folks are unconcerned by the wheel issues I > > > > > > >> encountered, I won't object. > > > > > > >> > > > > > > >> In addition to the usual verifications, I worked with Jeroen to > > > build the R > > > > > > >> Windows binaries for the 0.17 rc0 and confirmed that the R > > > package passes > > > > > > >> all checks with them. > > > > > > >> > > > > > > >> Neal > > > > > > >> > > > > > > >> On Sat, Apr 18, 2020 at 12:13 AM Sutou Kouhei > > > > > > >> > > > wrote: > > > > > > >> > > > > > > >>> +1 (binding) > > > > > > >>> > > > > > > >>> I ran the followings on Debian GNU/Linux sid: > > > > > > >>> > > > > > > >>> * TEST_CSHARP=0 \ > > > > > > >>> JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ > > > > > > >>> CUDA_TOOLKIT_ROOT=/usr \ > > > > > > >>> ARROW_CMAKE_OPTIONS=-DgRPC_SOURCE=BUNDLED \ > > > > > > >>> dev/release/verify-release-candidate.sh source 0.17.0 0 > > > > > > >>> * dev/release/verify-release-candidate.sh binaries 0.17.0 0 > > > > > > >>> * dev/release/verify-release-candidate.sh wheels 0.17.0 0 > > > > > > >>> > > > > > > >>> with: > > > > > > >>> > > > > > > >>> * gcc (Debian 9.3.0-10) 9.3.0 > > > > > > >>> * openjdk version "1.8.0_252" > > > > > > >>> * Node.JS v12.1.0 > > > > > > >>> * nvidia-cuda-dev 10.1.168-8 > > > > > > >>> > > > > > > >>> Notes: > > > > > > >>> > > > > > > >>> * C# sourcelink is failed as usual. > > > > > > >>> https://issues.apache.org/jira/browse/ARROW-8505 > > > > > > >>> > > > > > > >>> > > > > > > >>> Thanks, > > > > > > >>> -- > > > > > > >>> kou > > > > > > >>> > > > > > > >>> In > > ykocv1cymwkvhuk...@mail.gmail.com>
[jira] [Created] (ARROW-8532) [C++][CSV] Add support for sentinel values.
Ravil Bikbulatov created ARROW-8532: --- Summary: [C++][CSV] Add support for sentinel values. Key: ARROW-8532 URL: https://issues.apache.org/jira/browse/ARROW-8532 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Ravil Bikbulatov Some systems still use sentinel values to store nulls. It would be good if read_csv would place sentinel values and user wouldn't need to convet null bitmaps to sentinel values. Adding this support doesn't contradict Arrow specification as null values are undefined. Also it wouldn't add any overhead to read_csv. Since Arrow is general purpose framework I think we can relieve users from pain of converting bitmats to sentinel values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
The VOTE carries with 7 binding +1 votes and 1 non-binding +1 vote. I'm starting the post-release tasks. I'm going to send updates to this thread to coordinate if anyone wants to help. On Mon, Apr 20, 2020 at 3:25 PM Rok Mihevc wrote: > > +1 (non-binding) > > Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. > > Rok > > On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs > wrote: > > > So far we're having 7 +1 binding votes. I'm planning to close the > > vote later today, I anyone else would like to verify the release > > please do it today. > > > > Thanks, Krisztian > > > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney wrote: > > > > > > +1 (binding) > > > > > > I was able to run the verification of source and wheels on Windows > > > after addressing the issues I ran into. The failure reported in > > > ARROW-8510 is worked around by using the UNITY build so we'll need to > > > investigate that separately > > > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney > > wrote: > > > > > > > > Right, sorry about that. > > > > > > > > So the difference it turns out is the UNITY build. The compiler error > > > > in ARROW-8510 does not occur for some reason when > > > > CMAKE_UNITY_BUILD=ON. > > > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou > > wrote: > > > > > > > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > > > The release verification script (which uses the VS 2017 compiler > > mode) > > > > > > fails for me on Windows > > > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > > > > > It looks like we are building with VS 2019 in CI. We may want to be > > > > > > more conservative and build with VS 2015 or VS 2017 in CI. This is > > > > > > peculiar though because the wheels are using VS 2015. So it's > > possible > > > > > > there's a compiler bug with VS 2017 that doesn't present on VS > > 2015. > > > > > > > > > > > > I don't think this should block the release however. I'll vote > > after I > > > > > > check some more things > > > > > > > > > > > > On Sat, Apr 18, 2020 at 3:32 PM Neal Richardson > > > > > > wrote: > > > > > >> > > > > > >> +1 (binding) > > > > > >> > > > > > >> I ran verification on Ubuntu, macOS, and Windows using crossbow. > > Thanks to > > > > > >> all who helped debug. A few things are still failing: see > > > > > >> https://github.com/apache/arrow/pull/6961#issuecomment-615940240 > > for > > > > > >> details. Since Python folks are unconcerned by the wheel issues I > > > > > >> encountered, I won't object. > > > > > >> > > > > > >> In addition to the usual verifications, I worked with Jeroen to > > build the R > > > > > >> Windows binaries for the 0.17 rc0 and confirmed that the R > > package passes > > > > > >> all checks with them. > > > > > >> > > > > > >> Neal > > > > > >> > > > > > >> On Sat, Apr 18, 2020 at 12:13 AM Sutou Kouhei > > wrote: > > > > > >> > > > > > >>> +1 (binding) > > > > > >>> > > > > > >>> I ran the followings on Debian GNU/Linux sid: > > > > > >>> > > > > > >>> * TEST_CSHARP=0 \ > > > > > >>> JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ > > > > > >>> CUDA_TOOLKIT_ROOT=/usr \ > > > > > >>> ARROW_CMAKE_OPTIONS=-DgRPC_SOURCE=BUNDLED \ > > > > > >>> dev/release/verify-release-candidate.sh source 0.17.0 0 > > > > > >>> * dev/release/verify-release-candidate.sh binaries 0.17.0 0 > > > > > >>> * dev/release/verify-release-candidate.sh wheels 0.17.0 0 > > > > > >>> > > > > > >>> with: > > > > > >>> > > > > > >>> * gcc (Debian 9.3.0-10) 9.3.0 > > > > > >>> * openjdk version "1.8.0_252" > > > > > >>> * Node.JS v12.1.0 > > > > > >>> * nvidia-cuda-dev 10.1.168-8 > > > > > >>> > > > > > >>> Notes: > > > > > >>> > > > > > >>> * C# sourcelink is failed as usual. > > > > > >>> https://issues.apache.org/jira/browse/ARROW-8505 > > > > > >>> > > > > > >>> > > > > > >>> Thanks, > > > > > >>> -- > > > > > >>> kou > > > > > >>> > > > > > >>> In > ykocv1cymwkvhuk...@mail.gmail.com> > > > > > >>> "[VOTE] Release Apache Arrow 0.17.0 - RC0" on Fri, 17 Apr 2020 > > 02:26:34 > > > > > >>> +0200, > > > > > >>> Krisztián Szűcs wrote: > > > > > >>> > > > > > Hi, > > > > > > > > > > I would like to propose the following release candidate (RC0) > > of Apache > > > > > Arrow version 0.17.0. This is a release consisting of 582 > > > > > resolved JIRA issues[1]. > > > > > > > > > > This release candidate is based on commit: > > > > > 3cbcb7b62c2f2d02851bff837758637eb592a64b [2] > > > > > > > > > > The source release rc0 is hosted at [3]. > > > > > The binary artifacts are hosted at [4][5][6][7]. > > > > > The changelog is located at [8]. > > > > > > > > > > Please download, verify checksums and signatures, run the unit > > tests, > > > > > and vote
[C++] Big-endian support
Hello, Recently some issues have been opened for big-endian support (i.e. support for big-endian *hosts*), and a couple patches submitted, thanks to Kazuaki Ishizaki. See e.g.: https://issues.apache.org/jira/browse/ARROW-8457 https://issues.apache.org/jira/browse/ARROW-8467 https://issues.apache.org/jira/browse/ARROW-8486 https://issues.apache.org/jira/browse/ARROW-8506 https://issues.apache.org/jira/browse/PARQUET-1845 Achieving big-endian support support accross the C++ Arrow and Parquet codebases is likely to be a very significant effort, potentially requiring cooperation between multiple developers. An additional problem is that, without any Continuous Integration set up, it will be impossible to ensure progress and be notified of regressions. If other people are seriously interested in the desired outcome, they should probably team up with Kazuaki Ishizaki and discuss a practical plan to avoid drowning in the difficulties. Regards Antoine.
[NIGHTLY] Arrow Build Report for Job nightly-2020-04-20-1
Arrow Build Report for Job nightly-2020-04-20-1 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1 Failed Tasks: - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-travis-gandiva-jar-osx - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-travis-homebrew-cpp - wheel-manylinux2014-cp35m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-wheel-manylinux2014-cp35m - wheel-manylinux2014-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-wheel-manylinux2014-cp36m - wheel-osx-cp35m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-travis-wheel-osx-cp35m - wheel-osx-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-travis-wheel-osx-cp36m - wheel-osx-cp37m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-travis-wheel-osx-cp37m - wheel-osx-cp38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-travis-wheel-osx-cp38 Succeeded Tasks: - centos-6-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-github-centos-6-amd64 - centos-7-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-github-centos-7-amd64 - centos-8-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-github-centos-8-amd64 - conda-linux-gcc-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-linux-gcc-py36 - conda-linux-gcc-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-linux-gcc-py37 - conda-linux-gcc-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-linux-gcc-py38 - conda-osx-clang-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-osx-clang-py36 - conda-osx-clang-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-osx-clang-py37 - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-osx-clang-py38 - conda-win-vs2015-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-win-vs2015-py36 - conda-win-vs2015-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-win-vs2015-py37 - conda-win-vs2015-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-conda-win-vs2015-py38 - debian-buster-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-github-debian-buster-amd64 - debian-stretch-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-github-debian-stretch-amd64 - gandiva-jar-xenial: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-travis-gandiva-jar-xenial - homebrew-r-autobrew: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-travis-homebrew-r-autobrew - test-conda-cpp-valgrind: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-github-test-conda-cpp-valgrind - test-conda-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-github-test-conda-cpp - test-conda-python-3.6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-azure-test-conda-python-3.6 - test-conda-python-3.7-dask-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-circle-test-conda-python-3.7-dask-latest - test-conda-python-3.7-hdfs-2.9.2: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-circle-test-conda-python-3.7-hdfs-2.9.2 - test-conda-python-3.7-kartothek-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-circle-test-conda-python-3.7-kartothek-latest - test-conda-python-3.7-kartothek-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-circle-test-conda-python-3.7-kartothek-master - test-conda-python-3.7-pandas-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-circle-test-conda-python-3.7-pandas-latest - test-conda-python-3.7-pandas-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-circle-test-conda-python-3.7-pandas-master - test-conda-python-3.7-spark-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-1-circle-test-conda-python-3.7-spark-master -
[jira] [Created] (ARROW-8531) [C++] Deprecate ARROW_USE_SIMD CMake option
Wes McKinney created ARROW-8531:
---
Summary: [C++] Deprecate ARROW_USE_SIMD CMake option
Key: ARROW-8531
URL: https://issues.apache.org/jira/browse/ARROW-8531
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Wes McKinney
Fix For: 1.0.0
This is superseded by the {{ARROW_SIMD_LEVEL}} option
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-8530) [Ruby] Return dictionary value by DictionaryArray#[]
Yosuke Shiro created ARROW-8530: --- Summary: [Ruby] Return dictionary value by DictionaryArray#[] Key: ARROW-8530 URL: https://issues.apache.org/jira/browse/ARROW-8530 Project: Apache Arrow Issue Type: New Feature Components: Ruby Reporter: Yosuke Shiro Assignee: Yosuke Shiro -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
+1 (non-binding) Tested on Ubuntu 18.04. C++ and Python builds / tests went fine. Rok On Mon, Apr 20, 2020 at 11:57 AM Krisztián Szűcs wrote: > So far we're having 7 +1 binding votes. I'm planning to close the > vote later today, I anyone else would like to verify the release > please do it today. > > Thanks, Krisztian > > On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney wrote: > > > > +1 (binding) > > > > I was able to run the verification of source and wheels on Windows > > after addressing the issues I ran into. The failure reported in > > ARROW-8510 is worked around by using the UNITY build so we'll need to > > investigate that separately > > > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney > wrote: > > > > > > Right, sorry about that. > > > > > > So the difference it turns out is the UNITY build. The compiler error > > > in ARROW-8510 does not occur for some reason when > > > CMAKE_UNITY_BUILD=ON. > > > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou > wrote: > > > > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > > The release verification script (which uses the VS 2017 compiler > mode) > > > > > fails for me on Windows > > > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > > > It looks like we are building with VS 2019 in CI. We may want to be > > > > > more conservative and build with VS 2015 or VS 2017 in CI. This is > > > > > peculiar though because the wheels are using VS 2015. So it's > possible > > > > > there's a compiler bug with VS 2017 that doesn't present on VS > 2015. > > > > > > > > > > I don't think this should block the release however. I'll vote > after I > > > > > check some more things > > > > > > > > > > On Sat, Apr 18, 2020 at 3:32 PM Neal Richardson > > > > > wrote: > > > > >> > > > > >> +1 (binding) > > > > >> > > > > >> I ran verification on Ubuntu, macOS, and Windows using crossbow. > Thanks to > > > > >> all who helped debug. A few things are still failing: see > > > > >> https://github.com/apache/arrow/pull/6961#issuecomment-615940240 > for > > > > >> details. Since Python folks are unconcerned by the wheel issues I > > > > >> encountered, I won't object. > > > > >> > > > > >> In addition to the usual verifications, I worked with Jeroen to > build the R > > > > >> Windows binaries for the 0.17 rc0 and confirmed that the R > package passes > > > > >> all checks with them. > > > > >> > > > > >> Neal > > > > >> > > > > >> On Sat, Apr 18, 2020 at 12:13 AM Sutou Kouhei > wrote: > > > > >> > > > > >>> +1 (binding) > > > > >>> > > > > >>> I ran the followings on Debian GNU/Linux sid: > > > > >>> > > > > >>> * TEST_CSHARP=0 \ > > > > >>> JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ > > > > >>> CUDA_TOOLKIT_ROOT=/usr \ > > > > >>> ARROW_CMAKE_OPTIONS=-DgRPC_SOURCE=BUNDLED \ > > > > >>> dev/release/verify-release-candidate.sh source 0.17.0 0 > > > > >>> * dev/release/verify-release-candidate.sh binaries 0.17.0 0 > > > > >>> * dev/release/verify-release-candidate.sh wheels 0.17.0 0 > > > > >>> > > > > >>> with: > > > > >>> > > > > >>> * gcc (Debian 9.3.0-10) 9.3.0 > > > > >>> * openjdk version "1.8.0_252" > > > > >>> * Node.JS v12.1.0 > > > > >>> * nvidia-cuda-dev 10.1.168-8 > > > > >>> > > > > >>> Notes: > > > > >>> > > > > >>> * C# sourcelink is failed as usual. > > > > >>> https://issues.apache.org/jira/browse/ARROW-8505 > > > > >>> > > > > >>> > > > > >>> Thanks, > > > > >>> -- > > > > >>> kou > > > > >>> > > > > >>> In ykocv1cymwkvhuk...@mail.gmail.com> > > > > >>> "[VOTE] Release Apache Arrow 0.17.0 - RC0" on Fri, 17 Apr 2020 > 02:26:34 > > > > >>> +0200, > > > > >>> Krisztián Szűcs wrote: > > > > >>> > > > > Hi, > > > > > > > > I would like to propose the following release candidate (RC0) > of Apache > > > > Arrow version 0.17.0. This is a release consisting of 582 > > > > resolved JIRA issues[1]. > > > > > > > > This release candidate is based on commit: > > > > 3cbcb7b62c2f2d02851bff837758637eb592a64b [2] > > > > > > > > The source release rc0 is hosted at [3]. > > > > The binary artifacts are hosted at [4][5][6][7]. > > > > The changelog is located at [8]. > > > > > > > > Please download, verify checksums and signatures, run the unit > tests, > > > > and vote on the release. See [9] for how to validate a release > candidate. > > > > > > > > The vote will be open for at least 72 hours. > > > > > > > > [ ] +1 Release this as Apache Arrow 0.17.0 > > > > [ ] +0 > > > > [ ] -1 Do not release this as Apache Arrow 0.17.0 because... > > > > > > > > [1]: > > > > >>> > https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.17.0 > > > > [2]: > >
Re: 0.17 release blog post: help needed
I have added some Java items. Best, Liya Fan On Mon, Apr 20, 2020 at 10:49 AM Kenta Murata wrote: > I've edited Ruby and C GLib parts. > Kou and Shiro will check them later. > > 2020年4月20日(月) 11:09 Wes McKinney : > > > > I made a pass through the changelog and added a bunch of TODOs related > > to C++. In general, as a reminder, in these blog posts since the > > releases are growing large we should try to present as compact a high > > level summary as possible to convey some of the highlights of our > > labors (so likely not needed to write out any JIRA numbers, people can > > look at the changelog for that). I'll spend some more time on the blog > > post after others have had a chance to take a pass through > > > > On Sat, Apr 18, 2020 at 12:13 PM Neal Richardson > > wrote: > > > > > > Hi all, > > > Since it looks like we're close to releasing 0.17, we need to fill in > the > > > details for our blog post announcement. I've started a document here: > > > > https://docs.google.com/document/d/16UKZtvL49o8nCDN8JU3Ut6y76Y9d8-4qXv5vFv7aNvs/edit#heading=h.kqqacbm2lpv8 > > > > > > Please fill in the details for the parts of the project you're close > to. > > > I'll handle wrapping this up in the usual boilerplate when we're done. > > > > > > Thanks, > > > Neal > > > > -- > Regards, > Kenta Murata >
[jira] [Created] (ARROW-8529) [C++] Fix usage of NextCounts() in GetBatchWithDict[Spaced]
Antoine Pitrou created ARROW-8529: - Summary: [C++] Fix usage of NextCounts() in GetBatchWithDict[Spaced] Key: ARROW-8529 URL: https://issues.apache.org/jira/browse/ARROW-8529 Project: Apache Arrow Issue Type: Bug Components: C++ Reporter: Antoine Pitrou Assignee: Antoine Pitrou See discussion in ARROW-8486 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-8528) [CI][NIGHTLY:gandiva-jar-osx] gandiva osx build is failing
Prudhvi Porandla created ARROW-8528: --- Summary: [CI][NIGHTLY:gandiva-jar-osx] gandiva osx build is failing Key: ARROW-8528 URL: https://issues.apache.org/jira/browse/ARROW-8528 Project: Apache Arrow Issue Type: Bug Reporter: Prudhvi Porandla Assignee: Prudhvi Porandla -- This message was sent by Atlassian Jira (v8.3.4#803005)
[Java] Memory Allocation Tips
Hi, Does the Arrow community have any tips / recommendations / best practices on how to manage Arrow memory in Java? Is there a way to rely on the GC exclusively (i.e. is there support for heap-only allocation)? Best, Razvan
[jira] [Created] (ARROW-8527) [C++][CSV] Add support for ReadOptions::skip_rows >= block_size
Ravil Bikbulatov created ARROW-8527: --- Summary: [C++][CSV] Add support for ReadOptions::skip_rows >= block_size Key: ARROW-8527 URL: https://issues.apache.org/jira/browse/ARROW-8527 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Ravil Bikbulatov Current implementation throws error in reader.cc:286 when skip_rows > header. However, in some workloads skip_rows used for not only skipping header but for just skipping first n-rows. In this case block-size constraint is greatly interferes. I think this constraint could be removed without performance reduction. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release Apache Arrow 0.17.0 - RC0
So far we're having 7 +1 binding votes. I'm planning to close the vote later today, I anyone else would like to verify the release please do it today. Thanks, Krisztian On Sun, Apr 19, 2020 at 10:59 PM Wes McKinney wrote: > > +1 (binding) > > I was able to run the verification of source and wheels on Windows > after addressing the issues I ran into. The failure reported in > ARROW-8510 is worked around by using the UNITY build so we'll need to > investigate that separately > > On Sat, Apr 18, 2020 at 4:01 PM Wes McKinney wrote: > > > > Right, sorry about that. > > > > So the difference it turns out is the UNITY build. The compiler error > > in ARROW-8510 does not occur for some reason when > > CMAKE_UNITY_BUILD=ON. > > > > On Sat, Apr 18, 2020 at 3:36 PM Antoine Pitrou wrote: > > > > > > > > > We build with both VS2015 and VS2017 on AppVeyor. > > > > > > Regards > > > > > > Antoine. > > > > > > > > > Le 18/04/2020 à 22:34, Wes McKinney a écrit : > > > > The release verification script (which uses the VS 2017 compiler mode) > > > > fails for me on Windows > > > > > > > > https://issues.apache.org/jira/browse/ARROW-8510 > > > > > > > > It looks like we are building with VS 2019 in CI. We may want to be > > > > more conservative and build with VS 2015 or VS 2017 in CI. This is > > > > peculiar though because the wheels are using VS 2015. So it's possible > > > > there's a compiler bug with VS 2017 that doesn't present on VS 2015. > > > > > > > > I don't think this should block the release however. I'll vote after I > > > > check some more things > > > > > > > > On Sat, Apr 18, 2020 at 3:32 PM Neal Richardson > > > > wrote: > > > >> > > > >> +1 (binding) > > > >> > > > >> I ran verification on Ubuntu, macOS, and Windows using crossbow. > > > >> Thanks to > > > >> all who helped debug. A few things are still failing: see > > > >> https://github.com/apache/arrow/pull/6961#issuecomment-615940240 for > > > >> details. Since Python folks are unconcerned by the wheel issues I > > > >> encountered, I won't object. > > > >> > > > >> In addition to the usual verifications, I worked with Jeroen to build > > > >> the R > > > >> Windows binaries for the 0.17 rc0 and confirmed that the R package > > > >> passes > > > >> all checks with them. > > > >> > > > >> Neal > > > >> > > > >> On Sat, Apr 18, 2020 at 12:13 AM Sutou Kouhei > > > >> wrote: > > > >> > > > >>> +1 (binding) > > > >>> > > > >>> I ran the followings on Debian GNU/Linux sid: > > > >>> > > > >>> * TEST_CSHARP=0 \ > > > >>> JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 \ > > > >>> CUDA_TOOLKIT_ROOT=/usr \ > > > >>> ARROW_CMAKE_OPTIONS=-DgRPC_SOURCE=BUNDLED \ > > > >>> dev/release/verify-release-candidate.sh source 0.17.0 0 > > > >>> * dev/release/verify-release-candidate.sh binaries 0.17.0 0 > > > >>> * dev/release/verify-release-candidate.sh wheels 0.17.0 0 > > > >>> > > > >>> with: > > > >>> > > > >>> * gcc (Debian 9.3.0-10) 9.3.0 > > > >>> * openjdk version "1.8.0_252" > > > >>> * Node.JS v12.1.0 > > > >>> * nvidia-cuda-dev 10.1.168-8 > > > >>> > > > >>> Notes: > > > >>> > > > >>> * C# sourcelink is failed as usual. > > > >>> https://issues.apache.org/jira/browse/ARROW-8505 > > > >>> > > > >>> > > > >>> Thanks, > > > >>> -- > > > >>> kou > > > >>> > > > >>> In > > > >>> > > > >>> "[VOTE] Release Apache Arrow 0.17.0 - RC0" on Fri, 17 Apr 2020 > > > >>> 02:26:34 > > > >>> +0200, > > > >>> Krisztián Szűcs wrote: > > > >>> > > > Hi, > > > > > > I would like to propose the following release candidate (RC0) of > > > Apache > > > Arrow version 0.17.0. This is a release consisting of 582 > > > resolved JIRA issues[1]. > > > > > > This release candidate is based on commit: > > > 3cbcb7b62c2f2d02851bff837758637eb592a64b [2] > > > > > > The source release rc0 is hosted at [3]. > > > The binary artifacts are hosted at [4][5][6][7]. > > > The changelog is located at [8]. > > > > > > Please download, verify checksums and signatures, run the unit tests, > > > and vote on the release. See [9] for how to validate a release > > > candidate. > > > > > > The vote will be open for at least 72 hours. > > > > > > [ ] +1 Release this as Apache Arrow 0.17.0 > > > [ ] +0 > > > [ ] -1 Do not release this as Apache Arrow 0.17.0 because... > > > > > > [1]: > > > >>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20status%20in%20%28Resolved%2C%20Closed%29%20AND%20fixVersion%20%3D%200.17.0 > > > [2]: > > > >>> https://github.com/apache/arrow/tree/3cbcb7b62c2f2d02851bff837758637eb592a64b > > > [3]: > > > >>> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-0.17.0-rc0 > > > [4]: https://bintray.com/apache/arrow/centos-rc/0.17.0-rc0 > > > [5]: https://bintray.com/apache/arrow/debian-rc/0.17.0-rc0 > > > [6]:
[jira] [Created] (ARROW-8526) [Python] Fix non-deterministic row order failure in dataset tests
Krisztian Szucs created ARROW-8526: -- Summary: [Python] Fix non-deterministic row order failure in dataset tests Key: ARROW-8526 URL: https://issues.apache.org/jira/browse/ARROW-8526 Project: Apache Arrow Issue Type: Improvement Components: Python Reporter: Krisztian Szucs Assignee: Joris Van den Bossche Fix For: 1.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-8525) [Crossbow] Add support for passing parameters to jobs from CLI
Krisztian Szucs created ARROW-8525: -- Summary: [Crossbow] Add support for passing parameters to jobs from CLI Key: ARROW-8525 URL: https://issues.apache.org/jira/browse/ARROW-8525 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration Reporter: Krisztian Szucs Use case https://github.com/apache/arrow/pull/6961#discussion_r410987966 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [NIGHTLY] Arrow Build Report for Job nightly-2020-04-18-1
By simply removing cached docker images (~13GB), haskell (~8.5GB), swap and apt cache I managed [1] to increase the available free space to 31GB from 8.3GB (out of 84GB). GHA ships with a lot of preinstalled software [2] which we don't need in docker based builds. [1] https://github.com/apache/arrow/pull/6988 [2] https://github.com/actions/virtual-environments/blob/master/images/linux/Ubuntu1804-README.md On Sat, Apr 18, 2020 at 8:27 PM Wes McKinney wrote: > > With continued "No space left on device" issues on GHA, what does that > indicate as far as long-term prospects for GitHub Actions? The amount > of binary artifacts that are builds produce and the amount of third > party code they depend on is going to continue to increase. > > On Sat, Apr 18, 2020 at 12:13 PM Crossbow wrote: > > > > > > Arrow Build Report for Job nightly-2020-04-18-1 > > > > All tasks: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1 > > > > Failed Tasks: > > - centos-8-amd64: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-centos-8-amd64 > > - debian-buster-amd64: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-debian-buster-amd64 > > - homebrew-cpp: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-travis-homebrew-cpp > > - test-conda-cpp-valgrind: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-test-conda-cpp-valgrind > > - test-conda-cpp: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-test-conda-cpp > > - test-r-rhub-ubuntu-gcc-release: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-test-r-rhub-ubuntu-gcc-release > > - test-ubuntu-18.04-cpp-static: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-test-ubuntu-18.04-cpp-static > > - ubuntu-bionic-amd64: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-ubuntu-bionic-amd64 > > - ubuntu-eoan-amd64: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-ubuntu-eoan-amd64 > > - ubuntu-focal-amd64: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-ubuntu-focal-amd64 > > > > Succeeded Tasks: > > - centos-6-amd64: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-centos-6-amd64 > > - centos-7-amd64: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-centos-7-amd64 > > - conda-linux-gcc-py36: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-linux-gcc-py36 > > - conda-linux-gcc-py37: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-linux-gcc-py37 > > - conda-linux-gcc-py38: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-linux-gcc-py38 > > - conda-osx-clang-py36: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-osx-clang-py36 > > - conda-osx-clang-py37: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-osx-clang-py37 > > - conda-osx-clang-py38: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-osx-clang-py38 > > - conda-win-vs2015-py36: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-win-vs2015-py36 > > - conda-win-vs2015-py37: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-win-vs2015-py37 > > - conda-win-vs2015-py38: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-conda-win-vs2015-py38 > > - debian-stretch-amd64: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-github-debian-stretch-amd64 > > - gandiva-jar-osx: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-travis-gandiva-jar-osx > > - gandiva-jar-xenial: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-travis-gandiva-jar-xenial > > - homebrew-r-autobrew: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-travis-homebrew-r-autobrew > > - test-conda-python-3.6: > > URL: > > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-18-1-azure-test-conda-python-3.6 > > - test-conda-python-3.7-dask-latest: > > URL: > >
[jira] [Created] (ARROW-8524) [CI] Free up space on github actions
Krisztian Szucs created ARROW-8524: -- Summary: [CI] Free up space on github actions Key: ARROW-8524 URL: https://issues.apache.org/jira/browse/ARROW-8524 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration Reporter: Krisztian Szucs Assignee: Krisztian Szucs Fix For: 1.0.0 We run into out of space errors in multiple builds. Github actions ship with a lot of preinstalled software which are essentially not required to run the docker-compose based builds, so try to clean up some space. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[NIGHTLY] Arrow Build Report for Job nightly-2020-04-20-0
Arrow Build Report for Job nightly-2020-04-20-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0 Failed Tasks: - centos-8-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-centos-8-amd64 - debian-buster-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-debian-buster-amd64 - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-travis-gandiva-jar-osx - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-travis-homebrew-cpp - test-conda-cpp-valgrind: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-test-conda-cpp-valgrind - test-conda-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-test-conda-cpp - test-conda-r-3.6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-circle-test-conda-r-3.6 - test-ubuntu-18.04-cpp-static: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-test-ubuntu-18.04-cpp-static - ubuntu-bionic-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-ubuntu-bionic-amd64 - ubuntu-eoan-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-ubuntu-eoan-amd64 - ubuntu-focal-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-ubuntu-focal-amd64 - wheel-osx-cp35m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-travis-wheel-osx-cp35m - wheel-osx-cp36m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-travis-wheel-osx-cp36m - wheel-osx-cp37m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-travis-wheel-osx-cp37m - wheel-osx-cp38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-travis-wheel-osx-cp38 Succeeded Tasks: - centos-6-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-centos-6-amd64 - centos-7-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-centos-7-amd64 - conda-linux-gcc-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-linux-gcc-py36 - conda-linux-gcc-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-linux-gcc-py37 - conda-linux-gcc-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-linux-gcc-py38 - conda-osx-clang-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-osx-clang-py36 - conda-osx-clang-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-osx-clang-py37 - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-osx-clang-py38 - conda-win-vs2015-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-win-vs2015-py36 - conda-win-vs2015-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-win-vs2015-py37 - conda-win-vs2015-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-conda-win-vs2015-py38 - debian-stretch-amd64: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-github-debian-stretch-amd64 - gandiva-jar-xenial: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-travis-gandiva-jar-xenial - homebrew-r-autobrew: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-travis-homebrew-r-autobrew - test-conda-python-3.6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-azure-test-conda-python-3.6 - test-conda-python-3.7-dask-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-circle-test-conda-python-3.7-dask-latest - test-conda-python-3.7-hdfs-2.9.2: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-circle-test-conda-python-3.7-hdfs-2.9.2 - test-conda-python-3.7-kartothek-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-circle-test-conda-python-3.7-kartothek-latest - test-conda-python-3.7-kartothek-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-04-20-0-circle-test-conda-python-3.7-kartothek-master - test-conda-python-3.7-pandas-latest: URL:
