[jira] [Created] (ARROW-5002) [C++] Implement GroupBy
Philipp Moritz created ARROW-5002: - Summary: [C++] Implement GroupBy Key: ARROW-5002 URL: https://issues.apache.org/jira/browse/ARROW-5002 Project: Apache Arrow Issue Type: Improvement Reporter: Philipp Moritz Dear all, I wonder what the best way forward is for implementing GroupBy kernels. Initially this was part of https://issues.apache.org/jira/browse/ARROW-4124 but is not contained in the current implementation as far as I can tell. It seems that the part of group by that just returns indices could be conveniently implemented with the HashKernel. That seems useful in any case. Is that indeed the best way forward/should this be done? GroupBy + Aggregate could then either be implemented with that + the Take kernel + aggregation involving more memory copies than necessary though or as part of the aggregate kernel. Probably the latter is preferred, any thoughts on that? Am I missing any other JIRAs related to this? Best, Philipp. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1
Oh, sorry for my partial publishing... Krisztián and Paul, thanks for following-up this. Thanks, -- kou In "Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1" on Sun, 24 Mar 2019 22:42:58 +0100, Krisztián Szűcs wrote: > All of the packages are out now: > https://www.npmjs.com/settings/apache-arrow/packages > > On Sun, Mar 24, 2019 at 10:37 PM Krisztián Szűcs > wrote: > >> Thanks Paul! I'm trying it. >> >> On Sun, Mar 24, 2019 at 10:26 PM Paul Taylor wrote: >> >>> Yes, the solution here is to publish all the packages again, and ignore >>> errors from ones that have already been published via: >>> >>> npx lerna exec --no-bail -- npm publish >>> >>> Best, >>> Paul >>> >>> > On Mar 24, 2019, at 2:10 PM, Krisztián Szűcs >>> wrote: >>> > >>> > Hi Kou, >>> > >>> > Paul has already added me, and I was trying to publish the >>> > packages, but it fails with: >>> > 19 verbose stack Error: 403 Forbidden - PUT >>> > https://registry.npmjs.org/@apache-arrow%2fes2015-umd - You cannot >>> publish >>> > over the previously published versions: 0.4.1. >>> > >>> > It seems like the script has updated three of the packages: >>> > https://www.npmjs.com/settings/apache-arrow/packages >>> > >>> > And now it fails to publish again. >>> > Any suggestions? >>> > >>> > On Sun, Mar 24, 2019 at 10:06 PM Kouhei Sutou >>> wrote: >>> > >>> >> Hi, >>> >> >>> >> I've published 0.4.1: >>> >> https://www.npmjs.com/package/apache-arrow/v/0.4.1 >>> >> >>> >> (It seems that "npx lerna exec -- npm publish" in >>> >> npm-release.sh doesn't work with 2FA enabled account. I >>> >> couldn't input one time password from the standard input. I >>> >> passed one time password by --otp option: npm publish --otp OTP) >>> >> >>> >> Krisztian, could you tell me your user name at mpmjs? I'll >>> >> add you to maintainers. >>> >> >>> >> >>> >> Thanks, >>> >> -- >>> >> kou >>> >> >>> >> In >> > >>> >> "Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1" on Sun, 24 Mar 2019 >>> >> 19:27:25 +0100, >>> >> Krisztián Szűcs wrote: >>> >> >>> >>> Hi All, >>> >>> >>> >>> The vote carries with 3 binding +1 votes. Thanks to everyone for >>> >>> helping verify the release! >>> >>> >>> >>> I've published the release to the Apache dist system [1], however >>> >>> I don't have rights to push the NPM package [2]. >>> >>> Could someone either publish it to NPM or grant me rights? >>> >>> >>> >>> Thanks, Krisztian >>> >>> >>> >>> [1]: https://dist.apache.org/repos/dist/release/arrow/arrow-js-0.4.1/ >>> >>> [2]: https://www.npmjs.com/package/apache-arrow >>> >>> >>> >>> On Thu, Mar 21, 2019 at 10:19 PM Brian Hulette >>> >> wrote: >>> >>> >>> +1 (non-binding) >>> >>> Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` with Node >>> >> v11.12.0 >>> >>> >>> On Thu, Mar 21, 2019 at 1:54 PM Krisztián Szűcs < >>> >> szucs.kriszt...@gmail.com >>> > >>> wrote: >>> >>> > +1 (binding) >>> > >>> > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` >>> > with Node v11.12.0 on OSX 10.14.3 and it looks good. >>> > >>> > On Thu, Mar 21, 2019 at 8:45 PM Krisztián Szűcs < >>> szucs.kriszt...@gmail.com >>> >> >>> > wrote: >>> > >>> >> Hello all, >>> >> >>> >> I would like to propose the following release candidate (rc1) of >>> >> Apache >>> >> Arrow JavaScript version 0.4.1. This is the second release >>> >> candidate, >>> >> including the fix for node version requirement [3]. >>> >> >>> >> The source release rc1 is hosted at [1]. >>> >> >>> >> This release candidate is based on commit >>> >> e9cf83c48b9740d42b5d18158e61c0962fda59c1 >>> >> >>> >> Please download, verify checksums and signatures, run the unit >>> >> tests, >>> and >>> >> vote >>> >> on the release. The easiest way is to use the JavaScript-specific >>> release >>> >> verification script dev/release/js-verify-release-candidate.sh. >>> >> >>> >> [ ] +1 Release this as Apache Arrow JavaScript 0.4.1 >>> >> [ ] +0 >>> >> [ ] -1 Do not release this as Apache Arrow JavaScript 0.4.1 >>> >> because... >>> >> >>> >> >>> >> How to validate a release signature: >>> >> https://httpd.apache.org/dev/verification.html >>> >> >>> >> [1]: >>> >> >>> >>> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-js-0.4.1-rc1/ >>> >> [2]: >>> >> >>> > >>> >>> >> >>> https://github.com/apache/arrow/tree/e9cf83c48b9740d42b5d18158e61c0962fda59c1 >>> >> [3]: https://github.com/apache/arrow/pull/4006/ >>> >> >>> > >>> >>> >> >>> >>>
Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1
All of the packages are out now: https://www.npmjs.com/settings/apache-arrow/packages On Sun, Mar 24, 2019 at 10:37 PM Krisztián Szűcs wrote: > Thanks Paul! I'm trying it. > > On Sun, Mar 24, 2019 at 10:26 PM Paul Taylor wrote: > >> Yes, the solution here is to publish all the packages again, and ignore >> errors from ones that have already been published via: >> >> npx lerna exec --no-bail -- npm publish >> >> Best, >> Paul >> >> > On Mar 24, 2019, at 2:10 PM, Krisztián Szűcs >> wrote: >> > >> > Hi Kou, >> > >> > Paul has already added me, and I was trying to publish the >> > packages, but it fails with: >> > 19 verbose stack Error: 403 Forbidden - PUT >> > https://registry.npmjs.org/@apache-arrow%2fes2015-umd - You cannot >> publish >> > over the previously published versions: 0.4.1. >> > >> > It seems like the script has updated three of the packages: >> > https://www.npmjs.com/settings/apache-arrow/packages >> > >> > And now it fails to publish again. >> > Any suggestions? >> > >> > On Sun, Mar 24, 2019 at 10:06 PM Kouhei Sutou >> wrote: >> > >> >> Hi, >> >> >> >> I've published 0.4.1: >> >> https://www.npmjs.com/package/apache-arrow/v/0.4.1 >> >> >> >> (It seems that "npx lerna exec -- npm publish" in >> >> npm-release.sh doesn't work with 2FA enabled account. I >> >> couldn't input one time password from the standard input. I >> >> passed one time password by --otp option: npm publish --otp OTP) >> >> >> >> Krisztian, could you tell me your user name at mpmjs? I'll >> >> add you to maintainers. >> >> >> >> >> >> Thanks, >> >> -- >> >> kou >> >> >> >> In > > >> >> "Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1" on Sun, 24 Mar 2019 >> >> 19:27:25 +0100, >> >> Krisztián Szűcs wrote: >> >> >> >>> Hi All, >> >>> >> >>> The vote carries with 3 binding +1 votes. Thanks to everyone for >> >>> helping verify the release! >> >>> >> >>> I've published the release to the Apache dist system [1], however >> >>> I don't have rights to push the NPM package [2]. >> >>> Could someone either publish it to NPM or grant me rights? >> >>> >> >>> Thanks, Krisztian >> >>> >> >>> [1]: https://dist.apache.org/repos/dist/release/arrow/arrow-js-0.4.1/ >> >>> [2]: https://www.npmjs.com/package/apache-arrow >> >>> >> >>> On Thu, Mar 21, 2019 at 10:19 PM Brian Hulette >> >> wrote: >> >>> >> +1 (non-binding) >> >> Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` with Node >> >> v11.12.0 >> >> >> On Thu, Mar 21, 2019 at 1:54 PM Krisztián Szűcs < >> >> szucs.kriszt...@gmail.com >> > >> wrote: >> >> > +1 (binding) >> > >> > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` >> > with Node v11.12.0 on OSX 10.14.3 and it looks good. >> > >> > On Thu, Mar 21, 2019 at 8:45 PM Krisztián Szűcs < >> szucs.kriszt...@gmail.com >> >> >> > wrote: >> > >> >> Hello all, >> >> >> >> I would like to propose the following release candidate (rc1) of >> >> Apache >> >> Arrow JavaScript version 0.4.1. This is the second release >> >> candidate, >> >> including the fix for node version requirement [3]. >> >> >> >> The source release rc1 is hosted at [1]. >> >> >> >> This release candidate is based on commit >> >> e9cf83c48b9740d42b5d18158e61c0962fda59c1 >> >> >> >> Please download, verify checksums and signatures, run the unit >> >> tests, >> and >> >> vote >> >> on the release. The easiest way is to use the JavaScript-specific >> release >> >> verification script dev/release/js-verify-release-candidate.sh. >> >> >> >> [ ] +1 Release this as Apache Arrow JavaScript 0.4.1 >> >> [ ] +0 >> >> [ ] -1 Do not release this as Apache Arrow JavaScript 0.4.1 >> >> because... >> >> >> >> >> >> How to validate a release signature: >> >> https://httpd.apache.org/dev/verification.html >> >> >> >> [1]: >> >> >> >> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-js-0.4.1-rc1/ >> >> [2]: >> >> >> > >> >> >> >> https://github.com/apache/arrow/tree/e9cf83c48b9740d42b5d18158e61c0962fda59c1 >> >> [3]: https://github.com/apache/arrow/pull/4006/ >> >> >> > >> >> >> >> >>
Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1
Thanks Paul! I'm trying it. On Sun, Mar 24, 2019 at 10:26 PM Paul Taylor wrote: > Yes, the solution here is to publish all the packages again, and ignore > errors from ones that have already been published via: > > npx lerna exec --no-bail -- npm publish > > Best, > Paul > > > On Mar 24, 2019, at 2:10 PM, Krisztián Szűcs > wrote: > > > > Hi Kou, > > > > Paul has already added me, and I was trying to publish the > > packages, but it fails with: > > 19 verbose stack Error: 403 Forbidden - PUT > > https://registry.npmjs.org/@apache-arrow%2fes2015-umd - You cannot > publish > > over the previously published versions: 0.4.1. > > > > It seems like the script has updated three of the packages: > > https://www.npmjs.com/settings/apache-arrow/packages > > > > And now it fails to publish again. > > Any suggestions? > > > > On Sun, Mar 24, 2019 at 10:06 PM Kouhei Sutou > wrote: > > > >> Hi, > >> > >> I've published 0.4.1: > >> https://www.npmjs.com/package/apache-arrow/v/0.4.1 > >> > >> (It seems that "npx lerna exec -- npm publish" in > >> npm-release.sh doesn't work with 2FA enabled account. I > >> couldn't input one time password from the standard input. I > >> passed one time password by --otp option: npm publish --otp OTP) > >> > >> Krisztian, could you tell me your user name at mpmjs? I'll > >> add you to maintainers. > >> > >> > >> Thanks, > >> -- > >> kou > >> > >> In > >> "Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1" on Sun, 24 Mar 2019 > >> 19:27:25 +0100, > >> Krisztián Szűcs wrote: > >> > >>> Hi All, > >>> > >>> The vote carries with 3 binding +1 votes. Thanks to everyone for > >>> helping verify the release! > >>> > >>> I've published the release to the Apache dist system [1], however > >>> I don't have rights to push the NPM package [2]. > >>> Could someone either publish it to NPM or grant me rights? > >>> > >>> Thanks, Krisztian > >>> > >>> [1]: https://dist.apache.org/repos/dist/release/arrow/arrow-js-0.4.1/ > >>> [2]: https://www.npmjs.com/package/apache-arrow > >>> > >>> On Thu, Mar 21, 2019 at 10:19 PM Brian Hulette > >> wrote: > >>> > +1 (non-binding) > > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` with Node > >> v11.12.0 > > > On Thu, Mar 21, 2019 at 1:54 PM Krisztián Szűcs < > >> szucs.kriszt...@gmail.com > > > wrote: > > > +1 (binding) > > > > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` > > with Node v11.12.0 on OSX 10.14.3 and it looks good. > > > > On Thu, Mar 21, 2019 at 8:45 PM Krisztián Szűcs < > szucs.kriszt...@gmail.com > >> > > wrote: > > > >> Hello all, > >> > >> I would like to propose the following release candidate (rc1) of > >> Apache > >> Arrow JavaScript version 0.4.1. This is the second release > >> candidate, > >> including the fix for node version requirement [3]. > >> > >> The source release rc1 is hosted at [1]. > >> > >> This release candidate is based on commit > >> e9cf83c48b9740d42b5d18158e61c0962fda59c1 > >> > >> Please download, verify checksums and signatures, run the unit > >> tests, > and > >> vote > >> on the release. The easiest way is to use the JavaScript-specific > release > >> verification script dev/release/js-verify-release-candidate.sh. > >> > >> [ ] +1 Release this as Apache Arrow JavaScript 0.4.1 > >> [ ] +0 > >> [ ] -1 Do not release this as Apache Arrow JavaScript 0.4.1 > >> because... > >> > >> > >> How to validate a release signature: > >> https://httpd.apache.org/dev/verification.html > >> > >> [1]: > >> > > https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-js-0.4.1-rc1/ > >> [2]: > >> > > > > >> > https://github.com/apache/arrow/tree/e9cf83c48b9740d42b5d18158e61c0962fda59c1 > >> [3]: https://github.com/apache/arrow/pull/4006/ > >> > > > > >> > >
Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1
Yes, the solution here is to publish all the packages again, and ignore errors from ones that have already been published via: npx lerna exec --no-bail -- npm publish Best, Paul > On Mar 24, 2019, at 2:10 PM, Krisztián Szűcs > wrote: > > Hi Kou, > > Paul has already added me, and I was trying to publish the > packages, but it fails with: > 19 verbose stack Error: 403 Forbidden - PUT > https://registry.npmjs.org/@apache-arrow%2fes2015-umd - You cannot publish > over the previously published versions: 0.4.1. > > It seems like the script has updated three of the packages: > https://www.npmjs.com/settings/apache-arrow/packages > > And now it fails to publish again. > Any suggestions? > > On Sun, Mar 24, 2019 at 10:06 PM Kouhei Sutou wrote: > >> Hi, >> >> I've published 0.4.1: >> https://www.npmjs.com/package/apache-arrow/v/0.4.1 >> >> (It seems that "npx lerna exec -- npm publish" in >> npm-release.sh doesn't work with 2FA enabled account. I >> couldn't input one time password from the standard input. I >> passed one time password by --otp option: npm publish --otp OTP) >> >> Krisztian, could you tell me your user name at mpmjs? I'll >> add you to maintainers. >> >> >> Thanks, >> -- >> kou >> >> In >> "Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1" on Sun, 24 Mar 2019 >> 19:27:25 +0100, >> Krisztián Szűcs wrote: >> >>> Hi All, >>> >>> The vote carries with 3 binding +1 votes. Thanks to everyone for >>> helping verify the release! >>> >>> I've published the release to the Apache dist system [1], however >>> I don't have rights to push the NPM package [2]. >>> Could someone either publish it to NPM or grant me rights? >>> >>> Thanks, Krisztian >>> >>> [1]: https://dist.apache.org/repos/dist/release/arrow/arrow-js-0.4.1/ >>> [2]: https://www.npmjs.com/package/apache-arrow >>> >>> On Thu, Mar 21, 2019 at 10:19 PM Brian Hulette >> wrote: >>> +1 (non-binding) Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` with Node >> v11.12.0 On Thu, Mar 21, 2019 at 1:54 PM Krisztián Szűcs < >> szucs.kriszt...@gmail.com > wrote: > +1 (binding) > > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` > with Node v11.12.0 on OSX 10.14.3 and it looks good. > > On Thu, Mar 21, 2019 at 8:45 PM Krisztián Szűcs < szucs.kriszt...@gmail.com >> > wrote: > >> Hello all, >> >> I would like to propose the following release candidate (rc1) of >> Apache >> Arrow JavaScript version 0.4.1. This is the second release >> candidate, >> including the fix for node version requirement [3]. >> >> The source release rc1 is hosted at [1]. >> >> This release candidate is based on commit >> e9cf83c48b9740d42b5d18158e61c0962fda59c1 >> >> Please download, verify checksums and signatures, run the unit >> tests, and >> vote >> on the release. The easiest way is to use the JavaScript-specific release >> verification script dev/release/js-verify-release-candidate.sh. >> >> [ ] +1 Release this as Apache Arrow JavaScript 0.4.1 >> [ ] +0 >> [ ] -1 Do not release this as Apache Arrow JavaScript 0.4.1 >> because... >> >> >> How to validate a release signature: >> https://httpd.apache.org/dev/verification.html >> >> [1]: >> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-js-0.4.1-rc1/ >> [2]: >> > >> https://github.com/apache/arrow/tree/e9cf83c48b9740d42b5d18158e61c0962fda59c1 >> [3]: https://github.com/apache/arrow/pull/4006/ >> > >>
Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1
I guess those three packages were updated by Kou, so I won't try to update/publish the packages. But I suppose the rest of the packages should be updated as well: https://www.npmjs.com/settings/apache-arrow/packages On Sun, Mar 24, 2019 at 10:10 PM Krisztián Szűcs wrote: > Hi Kou, > > Paul has already added me, and I was trying to publish the > packages, but it fails with: > 19 verbose stack Error: 403 Forbidden - PUT > https://registry.npmjs.org/@apache-arrow%2fes2015-umd - You cannot > publish over the previously published versions: 0.4.1. > > It seems like the script has updated three of the packages: > https://www.npmjs.com/settings/apache-arrow/packages > > And now it fails to publish again. > Any suggestions? > > On Sun, Mar 24, 2019 at 10:06 PM Kouhei Sutou wrote: > >> Hi, >> >> I've published 0.4.1: >> https://www.npmjs.com/package/apache-arrow/v/0.4.1 >> >> (It seems that "npx lerna exec -- npm publish" in >> npm-release.sh doesn't work with 2FA enabled account. I >> couldn't input one time password from the standard input. I >> passed one time password by --otp option: npm publish --otp OTP) >> >> Krisztian, could you tell me your user name at mpmjs? I'll >> add you to maintainers. >> >> >> Thanks, >> -- >> kou >> >> In >> "Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1" on Sun, 24 Mar 2019 >> 19:27:25 +0100, >> Krisztián Szűcs wrote: >> >> > Hi All, >> > >> > The vote carries with 3 binding +1 votes. Thanks to everyone for >> > helping verify the release! >> > >> > I've published the release to the Apache dist system [1], however >> > I don't have rights to push the NPM package [2]. >> > Could someone either publish it to NPM or grant me rights? >> > >> > Thanks, Krisztian >> > >> > [1]: https://dist.apache.org/repos/dist/release/arrow/arrow-js-0.4.1/ >> > [2]: https://www.npmjs.com/package/apache-arrow >> > >> > On Thu, Mar 21, 2019 at 10:19 PM Brian Hulette >> wrote: >> > >> >> +1 (non-binding) >> >> >> >> Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` with Node >> v11.12.0 >> >> >> >> >> >> On Thu, Mar 21, 2019 at 1:54 PM Krisztián Szűcs < >> szucs.kriszt...@gmail.com >> >> > >> >> wrote: >> >> >> >> > +1 (binding) >> >> > >> >> > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` >> >> > with Node v11.12.0 on OSX 10.14.3 and it looks good. >> >> > >> >> > On Thu, Mar 21, 2019 at 8:45 PM Krisztián Szűcs < >> >> szucs.kriszt...@gmail.com >> >> > > >> >> > wrote: >> >> > >> >> > > Hello all, >> >> > > >> >> > > I would like to propose the following release candidate (rc1) of >> Apache >> >> > > Arrow JavaScript version 0.4.1. This is the second release >> candidate, >> >> > > including the fix for node version requirement [3]. >> >> > > >> >> > > The source release rc1 is hosted at [1]. >> >> > > >> >> > > This release candidate is based on commit >> >> > > e9cf83c48b9740d42b5d18158e61c0962fda59c1 >> >> > > >> >> > > Please download, verify checksums and signatures, run the unit >> tests, >> >> and >> >> > > vote >> >> > > on the release. The easiest way is to use the JavaScript-specific >> >> release >> >> > > verification script dev/release/js-verify-release-candidate.sh. >> >> > > >> >> > > [ ] +1 Release this as Apache Arrow JavaScript 0.4.1 >> >> > > [ ] +0 >> >> > > [ ] -1 Do not release this as Apache Arrow JavaScript 0.4.1 >> because... >> >> > > >> >> > > >> >> > > How to validate a release signature: >> >> > > https://httpd.apache.org/dev/verification.html >> >> > > >> >> > > [1]: >> >> > > >> >> >> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-js-0.4.1-rc1/ >> >> > > [2]: >> >> > > >> >> > >> >> >> https://github.com/apache/arrow/tree/e9cf83c48b9740d42b5d18158e61c0962fda59c1 >> >> > > [3]: https://github.com/apache/arrow/pull/4006/ >> >> > > >> >> > >> >> >> >
Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1
Hi Kou, Paul has already added me, and I was trying to publish the packages, but it fails with: 19 verbose stack Error: 403 Forbidden - PUT https://registry.npmjs.org/@apache-arrow%2fes2015-umd - You cannot publish over the previously published versions: 0.4.1. It seems like the script has updated three of the packages: https://www.npmjs.com/settings/apache-arrow/packages And now it fails to publish again. Any suggestions? On Sun, Mar 24, 2019 at 10:06 PM Kouhei Sutou wrote: > Hi, > > I've published 0.4.1: > https://www.npmjs.com/package/apache-arrow/v/0.4.1 > > (It seems that "npx lerna exec -- npm publish" in > npm-release.sh doesn't work with 2FA enabled account. I > couldn't input one time password from the standard input. I > passed one time password by --otp option: npm publish --otp OTP) > > Krisztian, could you tell me your user name at mpmjs? I'll > add you to maintainers. > > > Thanks, > -- > kou > > In > "Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1" on Sun, 24 Mar 2019 > 19:27:25 +0100, > Krisztián Szűcs wrote: > > > Hi All, > > > > The vote carries with 3 binding +1 votes. Thanks to everyone for > > helping verify the release! > > > > I've published the release to the Apache dist system [1], however > > I don't have rights to push the NPM package [2]. > > Could someone either publish it to NPM or grant me rights? > > > > Thanks, Krisztian > > > > [1]: https://dist.apache.org/repos/dist/release/arrow/arrow-js-0.4.1/ > > [2]: https://www.npmjs.com/package/apache-arrow > > > > On Thu, Mar 21, 2019 at 10:19 PM Brian Hulette > wrote: > > > >> +1 (non-binding) > >> > >> Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` with Node > v11.12.0 > >> > >> > >> On Thu, Mar 21, 2019 at 1:54 PM Krisztián Szűcs < > szucs.kriszt...@gmail.com > >> > > >> wrote: > >> > >> > +1 (binding) > >> > > >> > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` > >> > with Node v11.12.0 on OSX 10.14.3 and it looks good. > >> > > >> > On Thu, Mar 21, 2019 at 8:45 PM Krisztián Szűcs < > >> szucs.kriszt...@gmail.com > >> > > > >> > wrote: > >> > > >> > > Hello all, > >> > > > >> > > I would like to propose the following release candidate (rc1) of > Apache > >> > > Arrow JavaScript version 0.4.1. This is the second release > candidate, > >> > > including the fix for node version requirement [3]. > >> > > > >> > > The source release rc1 is hosted at [1]. > >> > > > >> > > This release candidate is based on commit > >> > > e9cf83c48b9740d42b5d18158e61c0962fda59c1 > >> > > > >> > > Please download, verify checksums and signatures, run the unit > tests, > >> and > >> > > vote > >> > > on the release. The easiest way is to use the JavaScript-specific > >> release > >> > > verification script dev/release/js-verify-release-candidate.sh. > >> > > > >> > > [ ] +1 Release this as Apache Arrow JavaScript 0.4.1 > >> > > [ ] +0 > >> > > [ ] -1 Do not release this as Apache Arrow JavaScript 0.4.1 > because... > >> > > > >> > > > >> > > How to validate a release signature: > >> > > https://httpd.apache.org/dev/verification.html > >> > > > >> > > [1]: > >> > > > >> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-js-0.4.1-rc1/ > >> > > [2]: > >> > > > >> > > >> > https://github.com/apache/arrow/tree/e9cf83c48b9740d42b5d18158e61c0962fda59c1 > >> > > [3]: https://github.com/apache/arrow/pull/4006/ > >> > > > >> > > >> >
Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1
Hi, I've published 0.4.1: https://www.npmjs.com/package/apache-arrow/v/0.4.1 (It seems that "npx lerna exec -- npm publish" in npm-release.sh doesn't work with 2FA enabled account. I couldn't input one time password from the standard input. I passed one time password by --otp option: npm publish --otp OTP) Krisztian, could you tell me your user name at mpmjs? I'll add you to maintainers. Thanks, -- kou In "Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1" on Sun, 24 Mar 2019 19:27:25 +0100, Krisztián Szűcs wrote: > Hi All, > > The vote carries with 3 binding +1 votes. Thanks to everyone for > helping verify the release! > > I've published the release to the Apache dist system [1], however > I don't have rights to push the NPM package [2]. > Could someone either publish it to NPM or grant me rights? > > Thanks, Krisztian > > [1]: https://dist.apache.org/repos/dist/release/arrow/arrow-js-0.4.1/ > [2]: https://www.npmjs.com/package/apache-arrow > > On Thu, Mar 21, 2019 at 10:19 PM Brian Hulette wrote: > >> +1 (non-binding) >> >> Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` with Node v11.12.0 >> >> >> On Thu, Mar 21, 2019 at 1:54 PM Krisztián Szűcs > > >> wrote: >> >> > +1 (binding) >> > >> > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` >> > with Node v11.12.0 on OSX 10.14.3 and it looks good. >> > >> > On Thu, Mar 21, 2019 at 8:45 PM Krisztián Szűcs < >> szucs.kriszt...@gmail.com >> > > >> > wrote: >> > >> > > Hello all, >> > > >> > > I would like to propose the following release candidate (rc1) of Apache >> > > Arrow JavaScript version 0.4.1. This is the second release candidate, >> > > including the fix for node version requirement [3]. >> > > >> > > The source release rc1 is hosted at [1]. >> > > >> > > This release candidate is based on commit >> > > e9cf83c48b9740d42b5d18158e61c0962fda59c1 >> > > >> > > Please download, verify checksums and signatures, run the unit tests, >> and >> > > vote >> > > on the release. The easiest way is to use the JavaScript-specific >> release >> > > verification script dev/release/js-verify-release-candidate.sh. >> > > >> > > [ ] +1 Release this as Apache Arrow JavaScript 0.4.1 >> > > [ ] +0 >> > > [ ] -1 Do not release this as Apache Arrow JavaScript 0.4.1 because... >> > > >> > > >> > > How to validate a release signature: >> > > https://httpd.apache.org/dev/verification.html >> > > >> > > [1]: >> > > >> https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-js-0.4.1-rc1/ >> > > [2]: >> > > >> > >> https://github.com/apache/arrow/tree/e9cf83c48b9740d42b5d18158e61c0962fda59c1 >> > > [3]: https://github.com/apache/arrow/pull/4006/ >> > > >> > >>
Re: Java OutOfMemoryException!
Thanks Razvan. Increasing the RootAllocator limit resolved the problem. Regards, Tanveer Ahmad From: Razvan Chitu Sent: Sunday, March 24, 2019 2:44:57 PM To: dev@arrow.apache.org Cc: u...@arrow.apache.org Subject: Re: Java OutOfMemoryException! Hi Tanveer, The stack trace seems to indicate that you you've breached the limit of the allocator used by the ArrowStreamReader, so that's where I'd look first. The limit is usually set when constructing an allocator (e.g. new RootAllocator(myLimit)) or when getting a child allocator (e.g. rootAllocator.newChildAllocator(...)). Razvan On Sun, Mar 24, 2019 at 12:33 PM Tanveer Ahmad - EWI wrote: > Hi, > > I am de-serializing multiple plasma objects in java at the same time, > everything is working fine but when the data size increases the following > error is being occurred for some threads. Any suggestion where I can > increase/change the memory allocation for these processes (I have more > memory available)? Is it JVM related or Arrow specific? > > Exception in thread "Thread-1" > org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of > size 634729984 due to memory limit. Current allocation: 0 > at > org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:273) > at > org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:249) > at > org.apache.arrow.vector.ipc.message.MessageChannelReader.readMessageBody(MessageChannelReader.java:88) > at > org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:204) > at > org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:116) > > > > Thanks. > > > Regards, > Tanveer Ahmad >
Re: [VOTE] Release Apache Arrow JS 0.4.1 - RC1
Hi All, The vote carries with 3 binding +1 votes. Thanks to everyone for helping verify the release! I've published the release to the Apache dist system [1], however I don't have rights to push the NPM package [2]. Could someone either publish it to NPM or grant me rights? Thanks, Krisztian [1]: https://dist.apache.org/repos/dist/release/arrow/arrow-js-0.4.1/ [2]: https://www.npmjs.com/package/apache-arrow On Thu, Mar 21, 2019 at 10:19 PM Brian Hulette wrote: > +1 (non-binding) > > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` with Node v11.12.0 > > > On Thu, Mar 21, 2019 at 1:54 PM Krisztián Szűcs > > wrote: > > > +1 (binding) > > > > Ran `dev/release/js-verify-release-candidate.sh 0.4.1 1` > > with Node v11.12.0 on OSX 10.14.3 and it looks good. > > > > On Thu, Mar 21, 2019 at 8:45 PM Krisztián Szűcs < > szucs.kriszt...@gmail.com > > > > > wrote: > > > > > Hello all, > > > > > > I would like to propose the following release candidate (rc1) of Apache > > > Arrow JavaScript version 0.4.1. This is the second release candidate, > > > including the fix for node version requirement [3]. > > > > > > The source release rc1 is hosted at [1]. > > > > > > This release candidate is based on commit > > > e9cf83c48b9740d42b5d18158e61c0962fda59c1 > > > > > > Please download, verify checksums and signatures, run the unit tests, > and > > > vote > > > on the release. The easiest way is to use the JavaScript-specific > release > > > verification script dev/release/js-verify-release-candidate.sh. > > > > > > [ ] +1 Release this as Apache Arrow JavaScript 0.4.1 > > > [ ] +0 > > > [ ] -1 Do not release this as Apache Arrow JavaScript 0.4.1 because... > > > > > > > > > How to validate a release signature: > > > https://httpd.apache.org/dev/verification.html > > > > > > [1]: > > > > https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-js-0.4.1-rc1/ > > > [2]: > > > > > > https://github.com/apache/arrow/tree/e9cf83c48b9740d42b5d18158e61c0962fda59c1 > > > [3]: https://github.com/apache/arrow/pull/4006/ > > > > > >
Re: Java OutOfMemoryException!
Hi Tanveer, The stack trace seems to indicate that you you've breached the limit of the allocator used by the ArrowStreamReader, so that's where I'd look first. The limit is usually set when constructing an allocator (e.g. new RootAllocator(myLimit)) or when getting a child allocator (e.g. rootAllocator.newChildAllocator(...)). Razvan On Sun, Mar 24, 2019 at 12:33 PM Tanveer Ahmad - EWI wrote: > Hi, > > I am de-serializing multiple plasma objects in java at the same time, > everything is working fine but when the data size increases the following > error is being occurred for some threads. Any suggestion where I can > increase/change the memory allocation for these processes (I have more > memory available)? Is it JVM related or Arrow specific? > > Exception in thread "Thread-1" > org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of > size 634729984 due to memory limit. Current allocation: 0 > at > org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:273) > at > org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:249) > at > org.apache.arrow.vector.ipc.message.MessageChannelReader.readMessageBody(MessageChannelReader.java:88) > at > org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:204) > at > org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:116) > > > > Thanks. > > > Regards, > Tanveer Ahmad >
Java OutOfMemoryException!
Hi, I am de-serializing multiple plasma objects in java at the same time, everything is working fine but when the data size increases the following error is being occurred for some threads. Any suggestion where I can increase/change the memory allocation for these processes (I have more memory available)? Is it JVM related or Arrow specific? Exception in thread "Thread-1" org.apache.arrow.memory.OutOfMemoryException: Unable to allocate buffer of size 634729984 due to memory limit. Current allocation: 0 at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:273) at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:249) at org.apache.arrow.vector.ipc.message.MessageChannelReader.readMessageBody(MessageChannelReader.java:88) at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:204) at org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:116) Thanks. Regards, Tanveer Ahmad
