Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)
+1 (binding) Ran release verification script on Debian GNU/Linux sid * gcc version 8.2.0 (Debian 8.2.0-1) * Ruby 2.6.0dev * NodeJS 10.8.0 * Note: NodeJS 8.11.2 (LTS) failed tests FlatBuffers 1.9.0 can't build with GCC 8.2.0. So I need to use the master like the following command line: % ARROW_FLATBUFFERS_URL=https://github.com/google/flatbuffers/archive/master.tar.gz LANG=en_US.UTF-8 dev/release/verify-release-candidate.sh 0.10.0 1 Thanks, -- kou In "Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)" on Sun, 5 Aug 2018 22:47:20 -0400, Wes McKinney wrote: > +1 (binding) > > * Ran release verification script on Ubuntu 14.04 (gcc 4.8.5, Python > 3.6, Ruby 2.5.x, NodeJS 10.8) > * Ran release verification script on Windows 10 > > I took the opportunity to document my process on > https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates. > The Windows script needed some work: > > https://github.com/apache/arrow/pull/2373 > > I had a failed Python unit test that I reported in > https://issues.apache.org/jira/browse/ARROW-2987. This is dependent on > particular details of the environment > > On Sun, Aug 5, 2018 at 6:41 PM, Phillip Cloud wrote: >> Just updated the pip wheels section >> >> On Sat, Aug 4, 2018 at 4:52 PM Phillip Cloud wrote: >> >>> I've updated the "Main source release and vote" section. I will update the >>> section about uploading pip wheels etc tomorrow. >>> >>> On Sat, Aug 4, 2018 at 4:37 PM Phillip Cloud wrote: >>> Thanks, the edit button is now available. On Sat, Aug 4, 2018 at 4:35 PM Wes McKinney wrote: > I just made you an admin (all PMCs should be admin on this space). I > turned on "Add" permission for pages for all confluence-users. > > On Sat, Aug 4, 2018 at 4:17 PM, Phillip Cloud wrote: > > I don't appear to have the ability to edit the page (despite the > unlocked > > icon saying everyone can edit). I don't see any "edit" button anywhere. > > Tried in both Chromium and Firefox. Is there some other hidden > permission > > that I need? > > > > On Sat, Aug 4, 2018 at 4:02 PM Wes McKinney > wrote: > > > >> Phillip and Krisztian, > >> > >> Would you be able to update the release management guide on the wiki > >> while your knowledge of the process is freshest? > >> > >> > https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide > >> > >> Thanks! > >> > >> On Fri, Aug 3, 2018 at 7:33 PM, Krisztián Szűcs > >> wrote: > >> > +1 > >> > > >> > ran ./dev/release/verify-release-candidate.sh 0.10.0 1 > >> > (same GISCAN issue Uwe has mentioned in RC0 thread) > >> > > >> > * macOS 10.13.5 > >> > * java 10.0.2 > >> > * python 3.6.5 > >> > * clang-902.0.39.2 > >> > * node v10.6.0 > >> > > >> > On Aug 3 2018, at 5:52 pm, Uwe L. Korn wrote: > >> >> > >> >> +1 (binding) > >> >> * ran ./dev/release/verify-release-candidate.sh on MacOS > >> >> * verified together with Marco that his problems are gone > >> >> > >> >> On Fri, Aug 3, 2018, at 5:45 PM, Li Jin wrote: > >> >> > +1 > >> >> > > >> >> > ran ./dev/release/verify-release-candidate.sh on MacOS > >> >> > * Java 8 > >> >> > * clang-902.0.39.2 > >> >> > > >> >> > On Fri, Aug 3, 2018 at 3:31 PM, Phillip Cloud > > >> wrote: > >> >> > > That should say "2nd" release candidate by the way. Apologies. > >> >> > > On Fri, Aug 3, 2018 at 11:29 AM Phillip Cloud < > cpcl...@gmail.com> > >> wrote: > >> >> > > > +1 (binding) > >> >> > > > ran ./dev/release/verify-release-candidate.sh 0.10.0 1 > >> >> > > > * Arch Linux > >> >> > > > * java8 > >> >> > > > * ruby 2.5.1p57 > >> >> > > > * python3.6 > >> >> > > > * clang 6.0.0 > >> >> > > > * node v10.5.0 > >> >> > > > > >> >> > > > On Fri, Aug 3, 2018 at 10:55 AM Phillip Cloud < > cpcl...@gmail.com> > >> wrote: > >> >> > > > > Hello all, > >> >> > > > > I'd like to propose the 1st release candidate (rc1) of > Apache > >> Arrow > >> >> > > > > version > >> >> > > > > 0.10.0. rc0 was invalidated because of ARROW-2963, the fix > for > >> which is > >> >> > > > > incorporated into rc1. > >> >> > > > > > >> >> > > > > This release candidate is distinct from previous releases > in > >> that it > >> >> > > > > includes binary artifacts such as python wheels, conda > >> packages, and > >> >> > > > > various linux distro packages in addition to the source > release > >> as was > >> >> > > > > previously available. > >> >> > > > > > >> >> > > > > The release verification script in master now verifies the > >> checksums and > >> >> > > > > GPG signatures of both the source release and the binary > >> artifacts. > >> >> > > > > > >> >> > > > > The source releas
[DISCUSS] Re-think CI strategy?
Hello, Our CI jobs are taking longer and longer. The main reason seem not to be that our test suites become more thorough (running tests actually seems to account for a very minor fraction of CI times) but the combined fact that 1) fetching dependencies and building is slow 2) we have many configurations tested on our CI jobs. The slowest job on our Travis-CI configuration routinely takes more 40 minutes (this is for a single job, where everything is essentially serialized). As for AppVeyor, the different jobs in a build there are mostly serialized, which balloons the total build time. If we ever move parquet-cpp into the Arrow repo, this will probably become worse again (though it might be better for parquet-cpp itself, which doesn't seem to have received a lot of care on the CI side: for instance, parquet-cpp AppVeyor builds are abysmally slow). I think we will have to cut down on the number of things we exercise on the per-build CI builds. This can be done along two axes: 1) remove some build steps that we deem non-critical or even unimportant 2) remove some build configurations entirely (for instance I don't understand why we need a "static CRT" build on Windows, or worse, why we need to have NMake-based builds at all) Any thoughts? Regards Antoine.
Re: [DISCUSS] Re-think CI strategy?
Hi, A straightforward way would be to run non-critical CI jobs as nightlies. Nightly package builds work pretty well, see the following link https://github.com/kszucs/crossbow/branches/all?query=nightly the notification logic requires improvement though. We should also run integrations tests regularly (dask, hdfs, spark). If We use a non apache queue repository for crossbow, We could even submit jobs to other CI services (e.g. CircleCI) to increase the build parallelism. Krisztian On Aug 6 2018, at 1:23 pm, Antoine Pitrou wrote: > > > Hello, > Our CI jobs are taking longer and longer. The main reason seem not to > be that our test suites become more thorough (running tests actually > seems to account for a very minor fraction of CI times) but the combined > fact that 1) fetching dependencies and building is slow 2) we have many > configurations tested on our CI jobs. > > The slowest job on our Travis-CI configuration routinely takes more 40 > minutes (this is for a single job, where everything is essentially > serialized). As for AppVeyor, the different jobs in a build there are > mostly serialized, which balloons the total build time. > > If we ever move parquet-cpp into the Arrow repo, this will probably > become worse again (though it might be better for parquet-cpp itself, > which doesn't seem to have received a lot of care on the CI side: for > instance, parquet-cpp AppVeyor builds are abysmally slow). > > I think we will have to cut down on the number of things we exercise on > the per-build CI builds. This can be done along two axes: > 1) remove some build steps that we deem non-critical or even unimportant > 2) remove some build configurations entirely (for instance I don't > understand why we need a "static CRT" build on Windows, or worse, why we > need to have NMake-based builds at all) > > Any thoughts? > Regards > Antoine.
Re: [DISCUSS] Re-think CI strategy?
hi Antoine, I completely agree. Part of why I've been so consistently pressing for nightly build tooling is to be able to shift more exhaustive testing out of per-commit runs into a daily build or an on-demand build to be invoked by the user either manually or by means of a bot. If you search in JIRA for the term "nightly" you can see a lot of issue that I have created for this already. In the case of Appveyor for example, I don't think we need the exhaustive build matrix on each commit, but it would be useful to be able to validate if desired (and in an automated way) that the NMake build works properly. This could be automated with the crossbow build tool. I think we can also improve CI build times by caching certain toolchain artifacts that are taking a long time to build (Thrift, I'm looking at you). We can verify that the toolchain will successfully build automatically via ExternalProject in a nightly. - Wes On Mon, Aug 6, 2018 at 7:23 AM, Antoine Pitrou wrote: > > Hello, > > Our CI jobs are taking longer and longer. The main reason seem not to > be that our test suites become more thorough (running tests actually > seems to account for a very minor fraction of CI times) but the combined > fact that 1) fetching dependencies and building is slow 2) we have many > configurations tested on our CI jobs. > > The slowest job on our Travis-CI configuration routinely takes more 40 > minutes (this is for a single job, where everything is essentially > serialized). As for AppVeyor, the different jobs in a build there are > mostly serialized, which balloons the total build time. > > If we ever move parquet-cpp into the Arrow repo, this will probably > become worse again (though it might be better for parquet-cpp itself, > which doesn't seem to have received a lot of care on the CI side: for > instance, parquet-cpp AppVeyor builds are abysmally slow). > > I think we will have to cut down on the number of things we exercise on > the per-build CI builds. This can be done along two axes: > 1) remove some build steps that we deem non-critical or even unimportant > 2) remove some build configurations entirely (for instance I don't > understand why we need a "static CRT" build on Windows, or worse, why we > need to have NMake-based builds at all) > > Any thoughts? > > Regards > > Antoine.
Re: [DISCUSS] Re-think CI strategy?
Le 06/08/2018 à 13:42, Wes McKinney a écrit : > hi Antoine, > > I completely agree. Part of why I've been so consistently pressing for > nightly build tooling is to be able to shift more exhaustive testing > out of per-commit runs into a daily build or an on-demand build to be > invoked by the user either manually or by means of a bot. If you > search in JIRA for the term "nightly" you can see a lot of issue that > I have created for this already. My worry with nightly jobs is that they doesn't clearly pinpoint the specific changeset where a regression occurred; also, since it happens after merging, there is less incentive to clear any mess introduced by a PR; moreover, it makes trying out a fix more difficult, since you don't get direct feedback on a commit or PR. So in exchange for lowering per-commit CI times, nightly jobs require extra human care to track regressions and make sure they get fixed. > it would be useful to be > able to validate if desired (and in an automated way) that the NMake > build works properly I guess the main question is why we're testing for NMake at all. CMake supports a range of different build tools, we can't exercise all of them. So I'd say on each platform we should exercise at most two build tools: - Ninja, because it's cross-platform and the fastest (which makes it desirable for developers *and* for CI) - the standard platform-specific build tool, i.e. GNU make on Unix and Visual Studio (or "msbuild") on Windows > I think we can also improve CI build times by caching certain > toolchain artifacts that are taking a long time to build (Thrift, I'm > looking at you). The 40+ minutes Travis-CI job already uses the toolchain packages AFAIK. Don't they include thrift? (another thought: when do we want to drop Python 2 compatibility?) Regards Antoine.
Re: [DISCUSS] Re-think CI strategy?
hi, On Mon, Aug 6, 2018 at 7:52 AM, Antoine Pitrou wrote: > > Le 06/08/2018 à 13:42, Wes McKinney a écrit : >> hi Antoine, >> >> I completely agree. Part of why I've been so consistently pressing for >> nightly build tooling is to be able to shift more exhaustive testing >> out of per-commit runs into a daily build or an on-demand build to be >> invoked by the user either manually or by means of a bot. If you >> search in JIRA for the term "nightly" you can see a lot of issue that >> I have created for this already. > > My worry with nightly jobs is that they doesn't clearly pinpoint the > specific changeset where a regression occurred; also, since it happens > after merging, there is less incentive to clear any mess introduced by a > PR; moreover, it makes trying out a fix more difficult, since you don't > get direct feedback on a commit or PR. > > So in exchange for lowering per-commit CI times, nightly jobs require > extra human care to track regressions and make sure they get fixed. The way that projects with much more complex testing deal with this is to have a pre-commit test run. The Apache Impala "verify merge" commit take several hours to run, for example. I don't think we are on our way to that point yet. What I'm suggesting is that we be able to write @arrow-test-bot please run exhaustive There are a lot of patches that clearly don't require exhaustive testing. In any case, we are likely to accumulate more testing than we can run on every commit / patch, but as long as we have a convenient mechanism to run the tests, then it is OK. > >> it would be useful to be >> able to validate if desired (and in an automated way) that the NMake >> build works properly > > I guess the main question is why we're testing for NMake at all. CMake > supports a range of different build tools, we can't exercise all of > them. So I'd say on each platform we should exercise at most two build > tools: > - Ninja, because it's cross-platform and the fastest (which makes it > desirable for developers *and* for CI) > - the standard platform-specific build tool, i.e. GNU make on Unix and > Visual Studio (or "msbuild") on Windows Well, I would be OK with dropping NMake in favor of supporting Ninja. > >> I think we can also improve CI build times by caching certain >> toolchain artifacts that are taking a long time to build (Thrift, I'm >> looking at you). > > The 40+ minutes Travis-CI job already uses the toolchain packages AFAIK. > Don't they include thrift? The most time consuming parts of the job are AFAIK: * The asv benchmarks * Code coverage uploads * Using valgrind * Building multiple versions of Python These can be addressed respectively by: * Nightly/opt-in testing asv (via bot) * Nightly/opt-in coverage * Nightly valgrind * No immediate solution. I would like to drop Python 2 in January 2019 - Wes > > (another thought: when do we want to drop Python 2 compatibility?) > > Regards > > Antoine.
Re: [DISCUSS] Re-think CI strategy?
> The 40+ minutes Travis-CI job already uses the toolchain packages AFAIK. > Don't they include thrift? I was referring to your comment about "parquet-cpp AppVeyor builds are abysmally slow". I think the slowness is in significant part due to the ExternalProject builds, where Thrift is the worst offender.
[jira] [Created] (ARROW-2991) [CI] Cut down number of AppVeyor jobs
Antoine Pitrou created ARROW-2991: - Summary: [CI] Cut down number of AppVeyor jobs Key: ARROW-2991 URL: https://issues.apache.org/jira/browse/ARROW-2991 Project: Apache Arrow Issue Type: Task Components: Continuous Integration Affects Versions: 0.10.0 Reporter: Antoine Pitrou AppVeyor builds all jobs serially so it's important not to have too many of them to avoid builds taking too much time and queuing up. I suggest to remove the following jobs: - the Release build with Ninja and VS2015; we already have both a Release build with Ninja and VS2017, and a Debug build with Ninja and VS2015 - the two NMake builds: we already exercise the Ninja (cross-platform, fastest) and Visual Studio (standard under Windows) build chains [~Max Risuhin] you added some of those jobs, do you have any concerns? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Arrow & plasma - java sample to store complex objects
Hi, Not sure this is the right channel for a "user" oriented question but the slack channel on heroku seams to be down... TL;DR: is there some hidden tutorial/java samples to store complex data objects in arrow and access (put/get) with plasma? I'm currently exploring the unit test from the java part of the source, but it's not really obvious... So, I'm starting with arrow and actually I went in for the plasma object store which should address an issue I currently have: sharing objects between process on multiples servers to serve as initial starting point in computations. The computation parts are already done since the need for distribution just recently emerged, and I'm trying to see if I can port the data object within arrow to distribute them over plasma. So far so good: I can launch the plasma_store and access it through the ObjectStoreLink API. But it's really on the byte array level. Any advice or best practice on how to convert existing data model to arrow compliant one? Should I look into the Arrow Schema example? Thanks for any pointer. Cheers, -- Gérard Dupont
[jira] [Created] (ARROW-2992) [Python] Parquet benchmark failure
Antoine Pitrou created ARROW-2992:
-
Summary: [Python] Parquet benchmark failure
Key: ARROW-2992
URL: https://issues.apache.org/jira/browse/ARROW-2992
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Antoine Pitrou
Assignee: Antoine Pitrou
This is a regression on git master:
{code:python}
Traceback (most recent call last):
File "/home/antoine/asv/asv/benchmark.py", line 867, in
commands[mode](args)
File "/home/antoine/asv/asv/benchmark.py", line 844, in main_run
result = benchmark.do_run()
File "/home/antoine/asv/asv/benchmark.py", line 398, in do_run
return self.run(*self._current_params)
File "/home/antoine/asv/asv/benchmark.py", line 473, in run
samples, number = self.benchmark_timing(timer, repeat, warmup_time,
number=number)
File "/home/antoine/asv/asv/benchmark.py", line 520, in benchmark_timing
timing = timer.timeit(number)
File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.6/timeit.py", line
178, in timeit
timing = self.inner(it, self.timer)
File "", line 6, in inner
File "/home/antoine/asv/asv/benchmark.py", line 464, in
func = lambda: self.func(*param)
File "/home/antoine/arrow/python/benchmarks/parquet.py", line 54, in
time_manifest_creation
pq.ParquetManifest(self.tmpdir, thread_pool=thread_pool)
TypeError: __init__() got an unexpected keyword argument 'thread_pool'
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 (Krisztián, Li), and no other votes, the vote passes. Thanks everyone! I will upload the Java packages as per the release management wiki. Would some folks please volunteer to get the Python packaging, documentation and website updates rolling? -Phillip
Re: [DISCUSS] Re-think CI strategy?
Wes, do you primarily want to drop Python 2 to speed up Travis or to reduce the development overhead? In my experience the development overhead is minimal and well worth it. For Travis, we could consider looking into other options like paying for more concurrency. January 2019 is very soon and Python 2 is still massively popular. On Mon, Aug 6, 2018 at 5:11 AM Wes McKinney wrote: > > The 40+ minutes Travis-CI job already uses the toolchain packages AFAIK. > > Don't they include thrift? > > I was referring to your comment about "parquet-cpp AppVeyor builds are > abysmally slow". I think the slowness is in significant part due to > the ExternalProject builds, where Thrift is the worst offender. >
Re: [DISCUSS] Re-think CI strategy?
Not wanting to answer for Wes, but those are two sides of the same coin: reducing CI overhead and complexity helps increase developer productivity. Reducing CI overhead is not a goal *in itself* (unless there are money issues I don't know about) ;-) The productivity cost of being Python 2-compatible is not very high *currently* (since much of the cost is a sunk cost by now), but these things all add up. So at some point we should really drop Python 2. Whether it's 2019 or 2020, I don't know and I don't get to decide. However, anything later than 2020 is excessively conservative IMHO. Regards Antoine. Le 06/08/2018 à 19:46, Robert Nishihara a écrit : > Wes, do you primarily want to drop Python 2 to speed up Travis or to reduce > the development overhead? In my experience the development overhead is > minimal and well worth it. For Travis, we could consider looking into other > options like paying for more concurrency. > > January 2019 is very soon and Python 2 is still massively popular. > > On Mon, Aug 6, 2018 at 5:11 AM Wes McKinney wrote: > >>> The 40+ minutes Travis-CI job already uses the toolchain packages AFAIK. >>> Don't they include thrift? >> >> I was referring to your comment about "parquet-cpp AppVeyor builds are >> abysmally slow". I think the slowness is in significant part due to >> the ExternalProject builds, where Thrift is the worst offender. >> >
Re: [DISCUSS] Re-think CI strategy?
Also, at this point we're sometimes hitting the 50 minutes time limit on our slowest Travis-CI matrix job, which means we have to restart it... making the build even slower. There's something perhaps suboptimal in the way we build Arrow C++ on Travis: - first we build it for no particular Python version - second we build it for Python 2.7 - third we build it for Python 3.6 Even those C++ files that don't depend on Python get re-compiled thrice (and ccache doesn't save us, probably because the compile flags are different). Regards Antoine. Le 06/08/2018 à 19:57, Antoine Pitrou a écrit : > > Not wanting to answer for Wes, but those are two sides of the same coin: > reducing CI overhead and complexity helps increase developer > productivity. Reducing CI overhead is not a goal *in itself* (unless > there are money issues I don't know about) ;-) > > The productivity cost of being Python 2-compatible is not very high > *currently* (since much of the cost is a sunk cost by now), but these > things all add up. So at some point we should really drop Python 2. > Whether it's 2019 or 2020, I don't know and I don't get to decide. > > However, anything later than 2020 is excessively conservative IMHO. > > Regards > > Antoine. > > > Le 06/08/2018 à 19:46, Robert Nishihara a écrit : >> Wes, do you primarily want to drop Python 2 to speed up Travis or to reduce >> the development overhead? In my experience the development overhead is >> minimal and well worth it. For Travis, we could consider looking into other >> options like paying for more concurrency. >> >> January 2019 is very soon and Python 2 is still massively popular. >> >> On Mon, Aug 6, 2018 at 5:11 AM Wes McKinney wrote: >> The 40+ minutes Travis-CI job already uses the toolchain packages AFAIK. Don't they include thrift? >>> >>> I was referring to your comment about "parquet-cpp AppVeyor builds are >>> abysmally slow". I think the slowness is in significant part due to >>> the ExternalProject builds, where Thrift is the worst offender. >>> >>
Re: [DISCUSS] Re-think CI strategy?
> > Also, at this point we're sometimes hitting the 50 minutes time limit on > our slowest Travis-CI matrix job, which means we have to restart it... > making the build even slower. > Only a short-term fix, but Travis can lengthen the max build time if you email them and ask them to.
[jira] [Created] (ARROW-2993) [JS] Document minimum supported NodeJS version
Wes McKinney created ARROW-2993: --- Summary: [JS] Document minimum supported NodeJS version Key: ARROW-2993 URL: https://issues.apache.org/jira/browse/ARROW-2993 Project: Apache Arrow Issue Type: Improvement Components: JavaScript Reporter: Wes McKinney Fix For: JS-0.4.0 The integration tests fail with NodeJS 8.11.3 LTS, but pass with 10.1 and higher. It would be useful to document the minimum supported NodeJS version -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)
Thanks Kou. I also noted the NodeJS 8.11.x LTS failure. Opened https://issues.apache.org/jira/browse/ARROW-2993 about better documenting this On Mon, Aug 6, 2018 at 4:26 AM, Kouhei Sutou wrote: > +1 (binding) > > Ran release verification script on Debian GNU/Linux sid > > * gcc version 8.2.0 (Debian 8.2.0-1) > * Ruby 2.6.0dev > * NodeJS 10.8.0 > * Note: NodeJS 8.11.2 (LTS) failed tests > > FlatBuffers 1.9.0 can't build with GCC 8.2.0. So I need to > use the master like the following command line: > > % > ARROW_FLATBUFFERS_URL=https://github.com/google/flatbuffers/archive/master.tar.gz > LANG=en_US.UTF-8 dev/release/verify-release-candidate.sh 0.10.0 1 > > > Thanks, > -- > kou > > In > "Re: [VOTE] Release Apache Arrow 0.10.0 (RC1)" on Sun, 5 Aug 2018 22:47:20 > -0400, > Wes McKinney wrote: > >> +1 (binding) >> >> * Ran release verification script on Ubuntu 14.04 (gcc 4.8.5, Python >> 3.6, Ruby 2.5.x, NodeJS 10.8) >> * Ran release verification script on Windows 10 >> >> I took the opportunity to document my process on >> https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates. >> The Windows script needed some work: >> >> https://github.com/apache/arrow/pull/2373 >> >> I had a failed Python unit test that I reported in >> https://issues.apache.org/jira/browse/ARROW-2987. This is dependent on >> particular details of the environment >> >> On Sun, Aug 5, 2018 at 6:41 PM, Phillip Cloud wrote: >>> Just updated the pip wheels section >>> >>> On Sat, Aug 4, 2018 at 4:52 PM Phillip Cloud wrote: >>> I've updated the "Main source release and vote" section. I will update the section about uploading pip wheels etc tomorrow. On Sat, Aug 4, 2018 at 4:37 PM Phillip Cloud wrote: > Thanks, the edit button is now available. > > On Sat, Aug 4, 2018 at 4:35 PM Wes McKinney wrote: > >> I just made you an admin (all PMCs should be admin on this space). I >> turned on "Add" permission for pages for all confluence-users. >> >> On Sat, Aug 4, 2018 at 4:17 PM, Phillip Cloud wrote: >> > I don't appear to have the ability to edit the page (despite the >> unlocked >> > icon saying everyone can edit). I don't see any "edit" button anywhere. >> > Tried in both Chromium and Firefox. Is there some other hidden >> permission >> > that I need? >> > >> > On Sat, Aug 4, 2018 at 4:02 PM Wes McKinney >> wrote: >> > >> >> Phillip and Krisztian, >> >> >> >> Would you be able to update the release management guide on the wiki >> >> while your knowledge of the process is freshest? >> >> >> >> >> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide >> >> >> >> Thanks! >> >> >> >> On Fri, Aug 3, 2018 at 7:33 PM, Krisztián Szűcs >> >> wrote: >> >> > +1 >> >> > >> >> > ran ./dev/release/verify-release-candidate.sh 0.10.0 1 >> >> > (same GISCAN issue Uwe has mentioned in RC0 thread) >> >> > >> >> > * macOS 10.13.5 >> >> > * java 10.0.2 >> >> > * python 3.6.5 >> >> > * clang-902.0.39.2 >> >> > * node v10.6.0 >> >> > >> >> > On Aug 3 2018, at 5:52 pm, Uwe L. Korn wrote: >> >> >> >> >> >> +1 (binding) >> >> >> * ran ./dev/release/verify-release-candidate.sh on MacOS >> >> >> * verified together with Marco that his problems are gone >> >> >> >> >> >> On Fri, Aug 3, 2018, at 5:45 PM, Li Jin wrote: >> >> >> > +1 >> >> >> > >> >> >> > ran ./dev/release/verify-release-candidate.sh on MacOS >> >> >> > * Java 8 >> >> >> > * clang-902.0.39.2 >> >> >> > >> >> >> > On Fri, Aug 3, 2018 at 3:31 PM, Phillip Cloud > > >> >> wrote: >> >> >> > > That should say "2nd" release candidate by the way. Apologies. >> >> >> > > On Fri, Aug 3, 2018 at 11:29 AM Phillip Cloud < >> cpcl...@gmail.com> >> >> wrote: >> >> >> > > > +1 (binding) >> >> >> > > > ran ./dev/release/verify-release-candidate.sh 0.10.0 1 >> >> >> > > > * Arch Linux >> >> >> > > > * java8 >> >> >> > > > * ruby 2.5.1p57 >> >> >> > > > * python3.6 >> >> >> > > > * clang 6.0.0 >> >> >> > > > * node v10.5.0 >> >> >> > > > >> >> >> > > > On Fri, Aug 3, 2018 at 10:55 AM Phillip Cloud < >> cpcl...@gmail.com> >> >> wrote: >> >> >> > > > > Hello all, >> >> >> > > > > I'd like to propose the 1st release candidate (rc1) of >> Apache >> >> Arrow >> >> >> > > > > version >> >> >> > > > > 0.10.0. rc0 was invalidated because of ARROW-2963, the fix >> for >> >> which is >> >> >> > > > > incorporated into rc1. >> >> >> > > > > >> >> >> > > > > This release candidate is distinct from previous releases >> in >> >> that it >> >> >> > > > > includes binary artifacts such as python wheels, conda >> >> packages, and >> >> >> > > > > various linux distro packages in addition to the sourc
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
I'm going to start kicking along the conda-forge packages. If anyone can assist with these, it would be much appreciated. I will update the website and write a blog post summarizing the 0.10.0 release (it's the biggest release we've ever done). I'm going to rebase master as soon as I merge the build for this PR completes: https://travis-ci.org/apache/arrow/builds/412745140 On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud wrote: > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 (Krisztián, > Li), and no other votes, the vote passes. Thanks everyone! > > I will upload the Java packages as per the release management wiki. > > Would some folks please volunteer to get the Python packaging, > documentation and website updates rolling? > > -Phillip
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
Wes, I can help You with the forge packages. I'm creating the PRs. On Aug 6 2018, at 8:52 pm, Wes McKinney wrote: > > I'm going to start kicking along the conda-forge packages. If anyone > can assist with these, it would be much appreciated. > > I will update the website and write a blog post summarizing the 0.10.0 > release (it's the biggest release we've ever done). > > I'm going to rebase master as soon as I merge the build for this PR > completes: https://travis-ci.org/apache/arrow/builds/412745140 > > On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud wrote: > > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 > > (Krisztián, > > Li), and no other votes, the vote passes. Thanks everyone! > > > > I will upload the Java packages as per the release management wiki. > > Would some folks please volunteer to get the Python packaging, > > documentation and website updates rolling? > > > > -Phillip
[jira] [Created] (ARROW-2994) [C++] Only include Python C header directories for Python-related compilation units
Wes McKinney created ARROW-2994: --- Summary: [C++] Only include Python C header directories for Python-related compilation units Key: ARROW-2994 URL: https://issues.apache.org/jira/browse/ARROW-2994 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Wes McKinney Fix For: 0.11.0 This is causing ccache artifacts to not be reusable when switching between Python 2.7 and 3.x See relevant line https://github.com/apache/arrow/blob/master/cpp/CMakeLists.txt#L721 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2995) [CI] Build Python libraries in same run when running C++ unit tests so project does not need to be rebuilt again right away
Wes McKinney created ARROW-2995: --- Summary: [CI] Build Python libraries in same run when running C++ unit tests so project does not need to be rebuilt again right away Key: ARROW-2995 URL: https://issues.apache.org/jira/browse/ARROW-2995 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration, Python Reporter: Wes McKinney Fix For: 0.11.0 See also ARROW-2994 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [DISCUSS] Re-think CI strategy?
@Robert, it looks like NumPy is making LTS releases until Jan 1, 2020 https://docs.scipy.org/doc/numpy-1.14.0/neps/dropping-python2.7-proposal.html Based on this, I think it's fine for us to continue to support Python 2.7 until then. It's only 16 months away; are you all ready for the next decade? We should also discuss if we want to continue to build and test Python 3.5. From download statistics it appears that there are 5-10x as many Python 3.6 users as 3.5. I would prefer to drop 3.5 and begin supporting 3.7 soon. @Antoine, I think we can avoid building the C++ codebase 3 times, but it will require a bit of retooling of the scripts. The reason that ccache isn't working properly is probably because the Python include directory is being included even for compilation units that do not use the Python C API. https://github.com/apache/arrow/blob/master/cpp/CMakeLists.txt#L721. I'm opening a JIRA about fixing this https://issues.apache.org/jira/browse/ARROW-2994 Created https://issues.apache.org/jira/browse/ARROW-2995 about removing the redundant build cycle On Mon, Aug 6, 2018 at 2:19 PM, Robert Nishihara wrote: >> >> Also, at this point we're sometimes hitting the 50 minutes time limit on >> our slowest Travis-CI matrix job, which means we have to restart it... >> making the build even slower. >> > Only a short-term fix, but Travis can lengthen the max build time if you > email them and ask them to.
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
I have just rebased master on tag apache-arrow-0.10.0. I will rebase the PRs that might be affected On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs wrote: > Wes, I can help You with the forge packages. I'm creating the PRs. > > On Aug 6 2018, at 8:52 pm, Wes McKinney wrote: >> >> I'm going to start kicking along the conda-forge packages. If anyone >> can assist with these, it would be much appreciated. >> >> I will update the website and write a blog post summarizing the 0.10.0 >> release (it's the biggest release we've ever done). >> >> I'm going to rebase master as soon as I merge the build for this PR >> completes: https://travis-ci.org/apache/arrow/builds/412745140 >> >> On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud wrote: >> > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 >> > (Krisztián, >> > Li), and no other votes, the vote passes. Thanks everyone! >> > >> > I will upload the Java packages as per the release management wiki. >> > Would some folks please volunteer to get the Python packaging, >> > documentation and website updates rolling? >> > >> > -Phillip
[jira] [Created] (ARROW-2996) [C++] Fix typo in cpp/.clang-tidy
Wes McKinney created ARROW-2996: --- Summary: [C++] Fix typo in cpp/.clang-tidy Key: ARROW-2996 URL: https://issues.apache.org/jira/browse/ARROW-2996 Project: Apache Arrow Issue Type: Improvement Components: C++ Reporter: Wes McKinney Fix For: 0.11.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2997) [Python] Scripts for uploading conda binary release artifacts to anaconda.org under @apache account
Wes McKinney created ARROW-2997: --- Summary: [Python] Scripts for uploading conda binary release artifacts to anaconda.org under @apache account Key: ARROW-2997 URL: https://issues.apache.org/jira/browse/ARROW-2997 Project: Apache Arrow Issue Type: Improvement Components: Python Reporter: Wes McKinney Fix For: 0.11.0 We do not have the ability to post these artifacts in conda-forge, but we could post them under apache https://anaconda.org/apache -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
I was trying to build the documentation but hit a JavaScript error. I am not familiar with JavaScript and TypeScript ... Any suggestion? Error log: /apache-arrow/arrow/js /apache-arrow + npm install npm WARN optional Skipping failed optional dependency /chokidar/fsevents: npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.2.4 npm WARN optional Skipping failed optional dependency /sane/fsevents: npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.2.4 npm WARN optional Skipping failed optional dependency /watchpack/chokidar/fsevents: npm WARN notsup Not compatible with your operating system or architecture: fsevents@1.2.4 npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none was installed. npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@^2.0.0 || ^3.0.0 but none was installed. + npm run doc > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts Using TypeScript 2.7.2 from /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib Error: /apache-arrow/arrow/js/src/vector.ts(161) Property 'childData' does not exist on type 'any[] | Data'. Property 'childData' does not exist on type 'any[]'. npm ERR! Linux 4.9.93-linuxkit-aufs npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" npm ERR! node v8.10.0 npm ERR! npm v3.5.2 npm ERR! code ELIFECYCLE npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts` npm ERR! Exit status 4 npm ERR! npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts'. npm ERR! Make sure you have the latest version of node.js and npm installed. npm ERR! If you do, this is most likely a problem with the apache-arrow package, npm ERR! not with npm itself. npm ERR! Tell the author that this fails on your system: npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts npm ERR! You can get information on how to open an issue for this project with: npm ERR! npm bugs apache-arrow npm ERR! Or if that isn't available, you can get their info via: npm ERR! npm owner ls apache-arrow npm ERR! There is likely additional logging output above. npm ERR! Please include the following file with any support request: npm ERR! /apache-arrow/arrow/js/npm-debug.log On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney wrote: > I have just rebased master on tag apache-arrow-0.10.0. I will rebase > the PRs that might be affected > > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs > wrote: > > Wes, I can help You with the forge packages. I'm creating the PRs. > > > > On Aug 6 2018, at 8:52 pm, Wes McKinney wrote: > >> > >> I'm going to start kicking along the conda-forge packages. If anyone > >> can assist with these, it would be much appreciated. > >> > >> I will update the website and write a blog post summarizing the 0.10.0 > >> release (it's the biggest release we've ever done). > >> > >> I'm going to rebase master as soon as I merge the build for this PR > >> completes: https://travis-ci.org/apache/arrow/builds/412745140 > >> > >> On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud > wrote: > >> > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 > (Krisztián, > >> > Li), and no other votes, the vote passes. Thanks everyone! > >> > > >> > I will upload the Java packages as per the release management wiki. > >> > Would some folks please volunteer to get the Python packaging, > >> > documentation and website updates rolling? > >> > > >> > -Phillip >
[jira] [Created] (ARROW-2998) [C++] Add variants of AllocateBuffer, AllocateResizeableBuffer that return unique_ptr
Wes McKinney created ARROW-2998:
---
Summary: [C++] Add variants of AllocateBuffer,
AllocateResizeableBuffer that return unique_ptr
Key: ARROW-2998
URL: https://issues.apache.org/jira/browse/ARROW-2998
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Wes McKinney
In some cases, users may which to use {{unique_ptr}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
You hit https://issues.apache.org/jira/browse/ARROW-2925. We can skip updating the JS documentation for now @Paul, can you take a look? On Mon, Aug 6, 2018 at 3:54 PM, Li Jin wrote: > I was trying to build the documentation but hit a JavaScript error. I am > not familiar with JavaScript and TypeScript ... Any suggestion? > > Error log: > > /apache-arrow/arrow/js /apache-arrow > > + npm install > > npm WARN optional Skipping failed optional dependency /chokidar/fsevents: > > npm WARN notsup Not compatible with your operating system or architecture: > fsevents@1.2.4 > > npm WARN optional Skipping failed optional dependency /sane/fsevents: > > npm WARN notsup Not compatible with your operating system or architecture: > fsevents@1.2.4 > > npm WARN optional Skipping failed optional dependency > /watchpack/chokidar/fsevents: > > npm WARN notsup Not compatible with your operating system or architecture: > fsevents@1.2.4 > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none was > installed. > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@^2.0.0 || > ^3.0.0 but none was installed. > > + npm run doc > > >> apache-arrow@0.3.0 doc /apache-arrow/arrow/js > >> shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts > > > > Using TypeScript 2.7.2 from > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib > > Error: /apache-arrow/arrow/js/src/vector.ts(161) > > Property 'childData' does not exist on type 'any[] | Data'. > > Property 'childData' does not exist on type 'any[]'. > > > npm ERR! Linux 4.9.93-linuxkit-aufs > > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" > > npm ERR! node v8.10.0 > > npm ERR! npm v3.5.2 > > npm ERR! code ELIFECYCLE > > npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode file > --out doc src/Arrow.ts` > > npm ERR! Exit status 4 > > npm ERR! > > npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf ./doc && > typedoc --mode file --out doc src/Arrow.ts'. > > npm ERR! Make sure you have the latest version of node.js and npm installed. > > npm ERR! If you do, this is most likely a problem with the apache-arrow > package, > > npm ERR! not with npm itself. > > npm ERR! Tell the author that this fails on your system: > > npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts > > npm ERR! You can get information on how to open an issue for this project > with: > > npm ERR! npm bugs apache-arrow > > npm ERR! Or if that isn't available, you can get their info via: > > npm ERR! npm owner ls apache-arrow > > npm ERR! There is likely additional logging output above. > > > npm ERR! Please include the following file with any support request: > > npm ERR! /apache-arrow/arrow/js/npm-debug.log > > > On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney wrote: > >> I have just rebased master on tag apache-arrow-0.10.0. I will rebase >> the PRs that might be affected >> >> On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs >> wrote: >> > Wes, I can help You with the forge packages. I'm creating the PRs. >> > >> > On Aug 6 2018, at 8:52 pm, Wes McKinney wrote: >> >> >> >> I'm going to start kicking along the conda-forge packages. If anyone >> >> can assist with these, it would be much appreciated. >> >> >> >> I will update the website and write a blog post summarizing the 0.10.0 >> >> release (it's the biggest release we've ever done). >> >> >> >> I'm going to rebase master as soon as I merge the build for this PR >> >> completes: https://travis-ci.org/apache/arrow/builds/412745140 >> >> >> >> On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud >> wrote: >> >> > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 >> (Krisztián, >> >> > Li), and no other votes, the vote passes. Thanks everyone! >> >> > >> >> > I will upload the Java packages as per the release management wiki. >> >> > Would some folks please volunteer to get the Python packaging, >> >> > documentation and website updates rolling? >> >> > >> >> > -Phillip >>
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
Looks like typedoc is using the wrong version of typescript. I can take a look at updating it later this afternoon if it’s blocking the release. On Mon, Aug 6, 2018 at 12:55 PM Li Jin wrote: > I was trying to build the documentation but hit a JavaScript error. I am > not familiar with JavaScript and TypeScript ... Any suggestion? > > Error log: > > /apache-arrow/arrow/js /apache-arrow > > + npm install > > npm WARN optional Skipping failed optional dependency /chokidar/fsevents: > > npm WARN notsup Not compatible with your operating system or architecture: > fsevents@1.2.4 > > npm WARN optional Skipping failed optional dependency /sane/fsevents: > > npm WARN notsup Not compatible with your operating system or architecture: > fsevents@1.2.4 > > npm WARN optional Skipping failed optional dependency > /watchpack/chokidar/fsevents: > > npm WARN notsup Not compatible with your operating system or architecture: > fsevents@1.2.4 > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none was > installed. > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@^2.0.0 > || > ^3.0.0 but none was installed. > > + npm run doc > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > > > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts > > > > Using TypeScript 2.7.2 from > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib > > Error: /apache-arrow/arrow/js/src/vector.ts(161) > > Property 'childData' does not exist on type 'any[] | Data'. > > Property 'childData' does not exist on type 'any[]'. > > > npm ERR! Linux 4.9.93-linuxkit-aufs > > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" > > npm ERR! node v8.10.0 > > npm ERR! npm v3.5.2 > > npm ERR! code ELIFECYCLE > > npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode file > --out doc src/Arrow.ts` > > npm ERR! Exit status 4 > > npm ERR! > > npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf ./doc && > typedoc --mode file --out doc src/Arrow.ts'. > > npm ERR! Make sure you have the latest version of node.js and npm > installed. > > npm ERR! If you do, this is most likely a problem with the apache-arrow > package, > > npm ERR! not with npm itself. > > npm ERR! Tell the author that this fails on your system: > > npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts > > npm ERR! You can get information on how to open an issue for this project > with: > > npm ERR! npm bugs apache-arrow > > npm ERR! Or if that isn't available, you can get their info via: > > npm ERR! npm owner ls apache-arrow > > npm ERR! There is likely additional logging output above. > > > npm ERR! Please include the following file with any support request: > > npm ERR! /apache-arrow/arrow/js/npm-debug.log > > > On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney wrote: > > > I have just rebased master on tag apache-arrow-0.10.0. I will rebase > > the PRs that might be affected > > > > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs > > wrote: > > > Wes, I can help You with the forge packages. I'm creating the PRs. > > > > > > On Aug 6 2018, at 8:52 pm, Wes McKinney wrote: > > >> > > >> I'm going to start kicking along the conda-forge packages. If anyone > > >> can assist with these, it would be much appreciated. > > >> > > >> I will update the website and write a blog post summarizing the 0.10.0 > > >> release (it's the biggest release we've ever done). > > >> > > >> I'm going to rebase master as soon as I merge the build for this PR > > >> completes: https://travis-ci.org/apache/arrow/builds/412745140 > > >> > > >> On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud > > wrote: > > >> > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 > > (Krisztián, > > >> > Li), and no other votes, the vote passes. Thanks everyone! > > >> > > > >> > I will upload the Java packages as per the release management wiki. > > >> > Would some folks please volunteer to get the Python packaging, > > >> > documentation and website updates rolling? > > >> > > > >> > -Phillip > > >
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
We can comment out the JS doc build for now, so it's not a blocker, but would be good to fix On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor wrote: > Looks like typedoc is using the wrong version of typescript. I can take a > look at updating it later this afternoon if it’s blocking the release. > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin wrote: > >> I was trying to build the documentation but hit a JavaScript error. I am >> not familiar with JavaScript and TypeScript ... Any suggestion? >> >> Error log: >> >> /apache-arrow/arrow/js /apache-arrow >> >> + npm install >> >> npm WARN optional Skipping failed optional dependency /chokidar/fsevents: >> >> npm WARN notsup Not compatible with your operating system or architecture: >> fsevents@1.2.4 >> >> npm WARN optional Skipping failed optional dependency /sane/fsevents: >> >> npm WARN notsup Not compatible with your operating system or architecture: >> fsevents@1.2.4 >> >> npm WARN optional Skipping failed optional dependency >> /watchpack/chokidar/fsevents: >> >> npm WARN notsup Not compatible with your operating system or architecture: >> fsevents@1.2.4 >> >> npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none was >> installed. >> >> npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@^2.0.0 >> || >> ^3.0.0 but none was installed. >> >> + npm run doc >> >> >> > apache-arrow@0.3.0 doc /apache-arrow/arrow/js >> >> > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts >> >> >> >> Using TypeScript 2.7.2 from >> /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib >> >> Error: /apache-arrow/arrow/js/src/vector.ts(161) >> >> Property 'childData' does not exist on type 'any[] | Data'. >> >> Property 'childData' does not exist on type 'any[]'. >> >> >> npm ERR! Linux 4.9.93-linuxkit-aufs >> >> npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" >> >> npm ERR! node v8.10.0 >> >> npm ERR! npm v3.5.2 >> >> npm ERR! code ELIFECYCLE >> >> npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode file >> --out doc src/Arrow.ts` >> >> npm ERR! Exit status 4 >> >> npm ERR! >> >> npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf ./doc && >> typedoc --mode file --out doc src/Arrow.ts'. >> >> npm ERR! Make sure you have the latest version of node.js and npm >> installed. >> >> npm ERR! If you do, this is most likely a problem with the apache-arrow >> package, >> >> npm ERR! not with npm itself. >> >> npm ERR! Tell the author that this fails on your system: >> >> npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts >> >> npm ERR! You can get information on how to open an issue for this project >> with: >> >> npm ERR! npm bugs apache-arrow >> >> npm ERR! Or if that isn't available, you can get their info via: >> >> npm ERR! npm owner ls apache-arrow >> >> npm ERR! There is likely additional logging output above. >> >> >> npm ERR! Please include the following file with any support request: >> >> npm ERR! /apache-arrow/arrow/js/npm-debug.log >> >> >> On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney wrote: >> >> > I have just rebased master on tag apache-arrow-0.10.0. I will rebase >> > the PRs that might be affected >> > >> > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs >> > wrote: >> > > Wes, I can help You with the forge packages. I'm creating the PRs. >> > > >> > > On Aug 6 2018, at 8:52 pm, Wes McKinney wrote: >> > >> >> > >> I'm going to start kicking along the conda-forge packages. If anyone >> > >> can assist with these, it would be much appreciated. >> > >> >> > >> I will update the website and write a blog post summarizing the 0.10.0 >> > >> release (it's the biggest release we've ever done). >> > >> >> > >> I'm going to rebase master as soon as I merge the build for this PR >> > >> completes: https://travis-ci.org/apache/arrow/builds/412745140 >> > >> >> > >> On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud >> > wrote: >> > >> > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 >> > (Krisztián, >> > >> > Li), and no other votes, the vote passes. Thanks everyone! >> > >> > >> > >> > I will upload the Java packages as per the release management wiki. >> > >> > Would some folks please volunteer to get the Python packaging, >> > >> > documentation and website updates rolling? >> > >> > >> > >> > -Phillip >> > >>
[jira] [Created] (ARROW-2999) [Python] Do not run ASV benchmarks in every Travis CI build to improve runtimes
Wes McKinney created ARROW-2999: --- Summary: [Python] Do not run ASV benchmarks in every Travis CI build to improve runtimes Key: ARROW-2999 URL: https://issues.apache.org/jira/browse/ARROW-2999 Project: Apache Arrow Issue Type: Improvement Components: Python Reporter: Wes McKinney Fix For: 0.11.0 I suggest we devise another way to make sure the benchmarks do not bitrot. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-3000) [Python] Do not build unit tests other than python-test in travis_script_python.sh
Wes McKinney created ARROW-3000:
---
Summary: [Python] Do not build unit tests other than python-test
in travis_script_python.sh
Key: ARROW-3000
URL: https://issues.apache.org/jira/browse/ARROW-3000
Project: Apache Arrow
Issue Type: Improvement
Components: Python
Reporter: Wes McKinney
Fix For: 0.11.0
This is inflating CI runtime. This is a bit of work as we do not currently have
granular control over which unit tests get built. We may have to add some logic
to tag tests with extra metadata that can be used for conditional compilation
(beyond the all-or-nothing {{NO_TESTS}} CMake variable being used now)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
The release artifacts are uploaded to SVN. On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney wrote: > We can comment out the JS doc build for now, so it's not a blocker, > but would be good to fix > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor wrote: > > Looks like typedoc is using the wrong version of typescript. I can take a > > look at updating it later this afternoon if it’s blocking the release. > > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin wrote: > > > >> I was trying to build the documentation but hit a JavaScript error. I am > >> not familiar with JavaScript and TypeScript ... Any suggestion? > >> > >> Error log: > >> > >> /apache-arrow/arrow/js /apache-arrow > >> > >> + npm install > >> > >> npm WARN optional Skipping failed optional dependency > /chokidar/fsevents: > >> > >> npm WARN notsup Not compatible with your operating system or > architecture: > >> fsevents@1.2.4 > >> > >> npm WARN optional Skipping failed optional dependency /sane/fsevents: > >> > >> npm WARN notsup Not compatible with your operating system or > architecture: > >> fsevents@1.2.4 > >> > >> npm WARN optional Skipping failed optional dependency > >> /watchpack/chokidar/fsevents: > >> > >> npm WARN notsup Not compatible with your operating system or > architecture: > >> fsevents@1.2.4 > >> > >> npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none was > >> installed. > >> > >> npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@ > ^2.0.0 > >> || > >> ^3.0.0 but none was installed. > >> > >> + npm run doc > >> > >> > >> > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > >> > >> > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts > >> > >> > >> > >> Using TypeScript 2.7.2 from > >> /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib > >> > >> Error: /apache-arrow/arrow/js/src/vector.ts(161) > >> > >> Property 'childData' does not exist on type 'any[] | Data'. > >> > >> Property 'childData' does not exist on type 'any[]'. > >> > >> > >> npm ERR! Linux 4.9.93-linuxkit-aufs > >> > >> npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" > >> > >> npm ERR! node v8.10.0 > >> > >> npm ERR! npm v3.5.2 > >> > >> npm ERR! code ELIFECYCLE > >> > >> npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode > file > >> --out doc src/Arrow.ts` > >> > >> npm ERR! Exit status 4 > >> > >> npm ERR! > >> > >> npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf ./doc > && > >> typedoc --mode file --out doc src/Arrow.ts'. > >> > >> npm ERR! Make sure you have the latest version of node.js and npm > >> installed. > >> > >> npm ERR! If you do, this is most likely a problem with the apache-arrow > >> package, > >> > >> npm ERR! not with npm itself. > >> > >> npm ERR! Tell the author that this fails on your system: > >> > >> npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc > src/Arrow.ts > >> > >> npm ERR! You can get information on how to open an issue for this > project > >> with: > >> > >> npm ERR! npm bugs apache-arrow > >> > >> npm ERR! Or if that isn't available, you can get their info via: > >> > >> npm ERR! npm owner ls apache-arrow > >> > >> npm ERR! There is likely additional logging output above. > >> > >> > >> npm ERR! Please include the following file with any support request: > >> > >> npm ERR! /apache-arrow/arrow/js/npm-debug.log > >> > >> > >> On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney > wrote: > >> > >> > I have just rebased master on tag apache-arrow-0.10.0. I will rebase > >> > the PRs that might be affected > >> > > >> > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs > >> > wrote: > >> > > Wes, I can help You with the forge packages. I'm creating the PRs. > >> > > > >> > > On Aug 6 2018, at 8:52 pm, Wes McKinney > wrote: > >> > >> > >> > >> I'm going to start kicking along the conda-forge packages. If > anyone > >> > >> can assist with these, it would be much appreciated. > >> > >> > >> > >> I will update the website and write a blog post summarizing the > 0.10.0 > >> > >> release (it's the biggest release we've ever done). > >> > >> > >> > >> I'm going to rebase master as soon as I merge the build for this PR > >> > >> completes: https://travis-ci.org/apache/arrow/builds/412745140 > >> > >> > >> > >> On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud > >> > wrote: > >> > >> > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 > >> > (Krisztián, > >> > >> > Li), and no other votes, the vote passes. Thanks everyone! > >> > >> > > >> > >> > I will upload the Java packages as per the release management > wiki. > >> > >> > Would some folks please volunteer to get the Python packaging, > >> > >> > documentation and website updates rolling? > >> > >> > > >> > >> > -Phillip > >> > > >> >
[jira] [Created] (ARROW-3001) [Packaging] Don't modify PATH during rust release verification
Krisztian Szucs created ARROW-3001: -- Summary: [Packaging] Don't modify PATH during rust release verification Key: ARROW-3001 URL: https://issues.apache.org/jira/browse/ARROW-3001 Project: Apache Arrow Issue Type: Task Reporter: Krisztian Szucs Assignee: Krisztian Szucs Sadly rustup has a side effect, it modifies .profile or .zprofile. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-3002) Inconsistent DataType Hashing
Sam Oluwalana created ARROW-3002:
Summary: Inconsistent DataType Hashing
Key: ARROW-3002
URL: https://issues.apache.org/jira/browse/ARROW-3002
Project: Apache Arrow
Issue Type: Bug
Reporter: Sam Oluwalana
{code:python}
>>> x = pa.field('record', pa.struct([pa.field('x', pa.int32(),
>>> nullable=False)]))
>>> y = pa.field('record', pa.struct([pa.field('x', pa.int32(),
>>> nullable=True)]))
>>> x.__hash__()
-9223372036569171727
>>> y.__hash__()
285604054
>>> x.type
StructType(struct)
>>> x.type.__hash__()
429437081997812647
>>> y.type.__hash__()
429437081997812647
>>> x
pyarrow.Field>
>>> y
pyarrow.Field>
}
{code}
The StructType should take nullable fields into account when generating the
hash.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
Wheels are up https://pypi.org/project/pyarrow/#files On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud wrote: > The release artifacts are uploaded to SVN. > > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney wrote: > >> We can comment out the JS doc build for now, so it's not a blocker, >> but would be good to fix >> >> On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor wrote: >> > Looks like typedoc is using the wrong version of typescript. I can take a >> > look at updating it later this afternoon if it’s blocking the release. >> > >> > On Mon, Aug 6, 2018 at 12:55 PM Li Jin wrote: >> > >> >> I was trying to build the documentation but hit a JavaScript error. I am >> >> not familiar with JavaScript and TypeScript ... Any suggestion? >> >> >> >> Error log: >> >> >> >> /apache-arrow/arrow/js /apache-arrow >> >> >> >> + npm install >> >> >> >> npm WARN optional Skipping failed optional dependency >> /chokidar/fsevents: >> >> >> >> npm WARN notsup Not compatible with your operating system or >> architecture: >> >> fsevents@1.2.4 >> >> >> >> npm WARN optional Skipping failed optional dependency /sane/fsevents: >> >> >> >> npm WARN notsup Not compatible with your operating system or >> architecture: >> >> fsevents@1.2.4 >> >> >> >> npm WARN optional Skipping failed optional dependency >> >> /watchpack/chokidar/fsevents: >> >> >> >> npm WARN notsup Not compatible with your operating system or >> architecture: >> >> fsevents@1.2.4 >> >> >> >> npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none was >> >> installed. >> >> >> >> npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@ >> ^2.0.0 >> >> || >> >> ^3.0.0 but none was installed. >> >> >> >> + npm run doc >> >> >> >> >> >> > apache-arrow@0.3.0 doc /apache-arrow/arrow/js >> >> >> >> > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts >> >> >> >> >> >> >> >> Using TypeScript 2.7.2 from >> >> /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib >> >> >> >> Error: /apache-arrow/arrow/js/src/vector.ts(161) >> >> >> >> Property 'childData' does not exist on type 'any[] | Data'. >> >> >> >> Property 'childData' does not exist on type 'any[]'. >> >> >> >> >> >> npm ERR! Linux 4.9.93-linuxkit-aufs >> >> >> >> npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" >> >> >> >> npm ERR! node v8.10.0 >> >> >> >> npm ERR! npm v3.5.2 >> >> >> >> npm ERR! code ELIFECYCLE >> >> >> >> npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode >> file >> >> --out doc src/Arrow.ts` >> >> >> >> npm ERR! Exit status 4 >> >> >> >> npm ERR! >> >> >> >> npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf ./doc >> && >> >> typedoc --mode file --out doc src/Arrow.ts'. >> >> >> >> npm ERR! Make sure you have the latest version of node.js and npm >> >> installed. >> >> >> >> npm ERR! If you do, this is most likely a problem with the apache-arrow >> >> package, >> >> >> >> npm ERR! not with npm itself. >> >> >> >> npm ERR! Tell the author that this fails on your system: >> >> >> >> npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc >> src/Arrow.ts >> >> >> >> npm ERR! You can get information on how to open an issue for this >> project >> >> with: >> >> >> >> npm ERR! npm bugs apache-arrow >> >> >> >> npm ERR! Or if that isn't available, you can get their info via: >> >> >> >> npm ERR! npm owner ls apache-arrow >> >> >> >> npm ERR! There is likely additional logging output above. >> >> >> >> >> >> npm ERR! Please include the following file with any support request: >> >> >> >> npm ERR! /apache-arrow/arrow/js/npm-debug.log >> >> >> >> >> >> On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney >> wrote: >> >> >> >> > I have just rebased master on tag apache-arrow-0.10.0. I will rebase >> >> > the PRs that might be affected >> >> > >> >> > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs >> >> > wrote: >> >> > > Wes, I can help You with the forge packages. I'm creating the PRs. >> >> > > >> >> > > On Aug 6 2018, at 8:52 pm, Wes McKinney >> wrote: >> >> > >> >> >> > >> I'm going to start kicking along the conda-forge packages. If >> anyone >> >> > >> can assist with these, it would be much appreciated. >> >> > >> >> >> > >> I will update the website and write a blog post summarizing the >> 0.10.0 >> >> > >> release (it's the biggest release we've ever done). >> >> > >> >> >> > >> I'm going to rebase master as soon as I merge the build for this PR >> >> > >> completes: https://travis-ci.org/apache/arrow/builds/412745140 >> >> > >> >> >> > >> On Mon, Aug 6, 2018 at 12:25 PM, Phillip Cloud >> >> > wrote: >> >> > >> > With 4 binding +1 votes (myself, Uwe, Wes, Kou), 2 non-binding +1 >> >> > (Krisztián, >> >> > >> > Li), and no other votes, the vote passes. Thanks everyone! >> >> > >> > >> >> > >> > I will upload the Java packages as per the release management >> wiki. >> >> > >> > Would some folks please volunteer to get the Python packaging, >> >> > >> > documentation and website updates rolling? >>
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
Great! Just tested to be sure, the osx wheels (all three python versions) are working. On Aug 6 2018, at 11:16 pm, Wes McKinney wrote: > > Wheels are up https://pypi.org/project/pyarrow/#files > On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud wrote: > > The release artifacts are uploaded to SVN. > > > > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney wrote: > > > We can comment out the JS doc build for now, so it's not a blocker, > > > but would be good to fix > > > > > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor wrote: > > > > Looks like typedoc is using the wrong version of typescript. I can take > > > > a > > > > look at updating it later this afternoon if it’s blocking the release. > > > > > > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin wrote: > > > > > I was trying to build the documentation but hit a JavaScript error. I > > > > > am > > > > > not familiar with JavaScript and TypeScript ... Any suggestion? > > > > > > > > > > Error log: > > > > > /apache-arrow/arrow/js /apache-arrow > > > > > + npm install > > > > > npm WARN optional Skipping failed optional dependency > > > /chokidar/fsevents: > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > architecture: > > > > > fsevents@1.2.4 > > > > > > > > > > npm WARN optional Skipping failed optional dependency /sane/fsevents: > > > > > npm WARN notsup Not compatible with your operating system or > > > architecture: > > > > > fsevents@1.2.4 > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > /watchpack/chokidar/fsevents: > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > architecture: > > > > > fsevents@1.2.4 > > > > > > > > > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none was > > > > > installed. > > > > > > > > > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@ > > > ^2.0.0 > > > > > || > > > > > ^3.0.0 but none was installed. > > > > > > > > > > + npm run doc > > > > > > > > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > > > > > > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts > > > > > > > > > > > > > > > Using TypeScript 2.7.2 from > > > > > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib > > > > > > > > > > Error: /apache-arrow/arrow/js/src/vector.ts(161) > > > > > Property 'childData' does not exist on type 'any[] | Data'. > > > > > Property 'childData' does not exist on type 'any[]'. > > > > > > > > > > npm ERR! Linux 4.9.93-linuxkit-aufs > > > > > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" > > > > > npm ERR! node v8.10.0 > > > > > npm ERR! npm v3.5.2 > > > > > npm ERR! code ELIFECYCLE > > > > > npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode > > > file > > > > > --out doc src/Arrow.ts` > > > > > > > > > > npm ERR! Exit status 4 > > > > > npm ERR! > > > > > npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf ./doc > > > && > > > > > typedoc --mode file --out doc src/Arrow.ts'. > > > > > > > > > > npm ERR! Make sure you have the latest version of node.js and npm > > > > > installed. > > > > > > > > > > npm ERR! If you do, this is most likely a problem with the > > > > > apache-arrow > > > > > package, > > > > > > > > > > npm ERR! not with npm itself. > > > > > npm ERR! Tell the author that this fails on your system: > > > > > npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc > > > src/Arrow.ts > > > > > > > > > > npm ERR! You can get information on how to open an issue for this > > > project > > > > > with: > > > > > > > > > > npm ERR! npm bugs apache-arrow > > > > > npm ERR! Or if that isn't available, you can get their info via: > > > > > npm ERR! npm owner ls apache-arrow > > > > > npm ERR! There is likely additional logging output above. > > > > > > > > > > npm ERR! Please include the following file with any support request: > > > > > npm ERR! /apache-arrow/arrow/js/npm-debug.log > > > > > > > > > > On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney > > > wrote: > > > > > > > > > > > I have just rebased master on tag apache-arrow-0.10.0. I will rebase > > > > > > the PRs that might be affected > > > > > > > > > > > > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs > > > > > > wrote: > > > > > > > Wes, I can help You with the forge packages. I'm creating the PRs. > > > > > > > > > > > > > > On Aug 6 2018, at 8:52 pm, Wes McKinney > > > wrote: > > > > > > > > > > > > > > > > I'm going to start kicking along the conda-forge packages. If > > > anyone > > > > > > > > can assist with these, it would be much appreciated. > > > > > > > > > > > > > > > > I will update the website and write a blog post summarizing the > > > 0.10.0 > > > > > > > > release (it's the biggest release we've ever done). > > > > > > > > > > > > > > > > I'm going to rebase master as soon as I merge the build for > > > > > > > > this PR > > > > > > > > completes: https://travis-ci.org/apache/arrow/builds/412745140 > > > > > > > > > >
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
Excellent. Thanks for testing that out Krisztián. On Mon, Aug 6, 2018 at 5:32 PM Krisztián Szűcs wrote: > Great! Just tested to be sure, the osx wheels (all three python versions) > are working. > > On Aug 6 2018, at 11:16 pm, Wes McKinney wrote: > > > > Wheels are up https://pypi.org/project/pyarrow/#files > > On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud wrote: > > > The release artifacts are uploaded to SVN. > > > > > > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney > wrote: > > > > We can comment out the JS doc build for now, so it's not a blocker, > > > > but would be good to fix > > > > > > > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor > wrote: > > > > > Looks like typedoc is using the wrong version of typescript. I can > take a > > > > > look at updating it later this afternoon if it’s blocking the > release. > > > > > > > > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin > wrote: > > > > > > I was trying to build the documentation but hit a JavaScript > error. I am > > > > > > not familiar with JavaScript and TypeScript ... Any suggestion? > > > > > > > > > > > > Error log: > > > > > > /apache-arrow/arrow/js /apache-arrow > > > > > > + npm install > > > > > > npm WARN optional Skipping failed optional dependency > > > > /chokidar/fsevents: > > > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > > architecture: > > > > > > fsevents@1.2.4 > > > > > > > > > > > > npm WARN optional Skipping failed optional dependency > /sane/fsevents: > > > > > > npm WARN notsup Not compatible with your operating system or > > > > architecture: > > > > > > fsevents@1.2.4 > > > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > > /watchpack/chokidar/fsevents: > > > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > > architecture: > > > > > > fsevents@1.2.4 > > > > > > > > > > > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but > none was > > > > > > installed. > > > > > > > > > > > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of > webpack@ > > > > ^2.0.0 > > > > > > || > > > > > > ^3.0.0 but none was installed. > > > > > > > > > > > > + npm run doc > > > > > > > > > > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > > > > > > > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts > > > > > > > > > > > > > > > > > > Using TypeScript 2.7.2 from > > > > > > > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib > > > > > > > > > > > > Error: /apache-arrow/arrow/js/src/vector.ts(161) > > > > > > Property 'childData' does not exist on type 'any[] | Data'. > > > > > > Property 'childData' does not exist on type 'any[]'. > > > > > > > > > > > > npm ERR! Linux 4.9.93-linuxkit-aufs > > > > > > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" > > > > > > npm ERR! node v8.10.0 > > > > > > npm ERR! npm v3.5.2 > > > > > > npm ERR! code ELIFECYCLE > > > > > > npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc > --mode > > > > file > > > > > > --out doc src/Arrow.ts` > > > > > > > > > > > > npm ERR! Exit status 4 > > > > > > npm ERR! > > > > > > npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm > -rf ./doc > > > > && > > > > > > typedoc --mode file --out doc src/Arrow.ts'. > > > > > > > > > > > > npm ERR! Make sure you have the latest version of node.js and npm > > > > > > installed. > > > > > > > > > > > > npm ERR! If you do, this is most likely a problem with the > apache-arrow > > > > > > package, > > > > > > > > > > > > npm ERR! not with npm itself. > > > > > > npm ERR! Tell the author that this fails on your system: > > > > > > npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc > > > > src/Arrow.ts > > > > > > > > > > > > npm ERR! You can get information on how to open an issue for this > > > > project > > > > > > with: > > > > > > > > > > > > npm ERR! npm bugs apache-arrow > > > > > > npm ERR! Or if that isn't available, you can get their info via: > > > > > > npm ERR! npm owner ls apache-arrow > > > > > > npm ERR! There is likely additional logging output above. > > > > > > > > > > > > npm ERR! Please include the following file with any support > request: > > > > > > npm ERR! /apache-arrow/arrow/js/npm-debug.log > > > > > > > > > > > > On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney < > wesmck...@gmail.com> > > > > wrote: > > > > > > > > > > > > > I have just rebased master on tag apache-arrow-0.10.0. I will > rebase > > > > > > > the PRs that might be affected > > > > > > > > > > > > > > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs > > > > > > > wrote: > > > > > > > > Wes, I can help You with the forge packages. I'm creating > the PRs. > > > > > > > > > > > > > > > > On Aug 6 2018, at 8:52 pm, Wes McKinney > > > > > wrote: > > > > > > > > > > > > > > > > > > I'm going to start kicking along the conda-forge packages. > If > > > > anyone > > > > > > > > > can assist with these, it would be much appreciated. > > > > > > > > > > >
[jira] [Created] (ARROW-3003) Unable Java doc in dev/gen_apidocs/create_documents.sh
Li Jin created ARROW-3003: - Summary: Unable Java doc in dev/gen_apidocs/create_documents.sh Key: ARROW-3003 URL: https://issues.apache.org/jira/browse/ARROW-3003 Project: Apache Arrow Issue Type: Task Reporter: Li Jin This is currently disabled but I have verified it works. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
I've generated the docs but unfortunately is stuck on a weird git issue. For some reason git doesn't want to add a few files after "git add ." (10 python files): modified: docs/python/parquet.html modified: docs/python/plasma.html modified: docs/python/search.html modified: docs/python/searchindex.js Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git checkout -- ..." to discard changes in working directory) modified: docs/python/_sources/generated/pyarrow.Array.rst.txt modified: docs/python/_sources/generated/pyarrow.Column.rst.txt modified: docs/python/_sources/generated/pyarrow.Field.rst.txt modified: docs/python/_sources/generated/pyarrow.Schema.rst.txt modified: docs/python/_sources/generated/pyarrow.lib.Array.rst.txt modified: docs/python/generated/pyarrow.Array.html modified: docs/python/generated/pyarrow.Column.html modified: docs/python/generated/pyarrow.Field.html modified: docs/python/generated/pyarrow.Schema.html modified: docs/python/generated/pyarrow.lib.Array.html I have tried a few things but no luck, including: * adding these files manually (git add docs/python/...) * stage only python docs * commit the staged change first and try to stage the rest of files * check git-ignore (there is none) * git add --force ... Unfortunately none of these seems to work. Anyone has a clue what might be going on? A separate issue is that I noticed Java doc is comment out from dev/gen_apidocs/create_documents.sh (probably due to Java8 lint-checks). I have uncommented it and it works. (Opened ARROW-3003) On Mon, Aug 6, 2018 at 9:33 PM, Phillip Cloud wrote: > Excellent. Thanks for testing that out Krisztián. > > On Mon, Aug 6, 2018 at 5:32 PM Krisztián Szűcs > wrote: > > > Great! Just tested to be sure, the osx wheels (all three python versions) > > are working. > > > > On Aug 6 2018, at 11:16 pm, Wes McKinney wrote: > > > > > > Wheels are up https://pypi.org/project/pyarrow/#files > > > On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud > wrote: > > > > The release artifacts are uploaded to SVN. > > > > > > > > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney > > wrote: > > > > > We can comment out the JS doc build for now, so it's not a blocker, > > > > > but would be good to fix > > > > > > > > > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor > > wrote: > > > > > > Looks like typedoc is using the wrong version of typescript. I > can > > take a > > > > > > look at updating it later this afternoon if it’s blocking the > > release. > > > > > > > > > > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin > > wrote: > > > > > > > I was trying to build the documentation but hit a JavaScript > > error. I am > > > > > > > not familiar with JavaScript and TypeScript ... Any suggestion? > > > > > > > > > > > > > > Error log: > > > > > > > /apache-arrow/arrow/js /apache-arrow > > > > > > > + npm install > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > /chokidar/fsevents: > > > > > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > > > architecture: > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > /sane/fsevents: > > > > > > > npm WARN notsup Not compatible with your operating system or > > > > > architecture: > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > > > /watchpack/chokidar/fsevents: > > > > > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > > > architecture: > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but > > none was > > > > > > > installed. > > > > > > > > > > > > > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of > > webpack@ > > > > > ^2.0.0 > > > > > > > || > > > > > > > ^3.0.0 but none was installed. > > > > > > > > > > > > > > + npm run doc > > > > > > > > > > > > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > > > > > > > > shx rm -rf ./doc && typedoc --mode file --out doc > src/Arrow.ts > > > > > > > > > > > > > > > > > > > > > Using TypeScript 2.7.2 from > > > > > > > > > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib > > > > > > > > > > > > > > Error: /apache-arrow/arrow/js/src/vector.ts(161) > > > > > > > Property 'childData' does not exist on type 'any[] | Data'. > > > > > > > Property 'childData' does not exist on type 'any[]'. > > > > > > > > > > > > > > npm ERR! Linux 4.9.93-linuxkit-aufs > > > > > > > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" > > > > > > > npm ERR! node v8.10.0 > > > > > > > npm ERR! npm v3.5.2 > > > > > > > npm ERR! code ELIFECYCLE > > > > > > > npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc > > --mode > > > > > file > > > > > > > --out doc src/Arrow.ts` > > > > > > > > > > > > > > npm ERR! Exit status 4 > > > > > > > n
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
Is it being checked out as a submodule? On Mon, Aug 6, 2018 at 6:07 PM Li Jin wrote: > I've generated the docs but unfortunately is stuck on a weird git issue. > For some reason git doesn't want to add a few files after "git add ." (10 > python files): > > modified: docs/python/parquet.html > > modified: docs/python/plasma.html > > modified: docs/python/search.html > > modified: docs/python/searchindex.js > > > Changes not staged for commit: > > (use "git add ..." to update what will be committed) > > (use "git checkout -- ..." to discard changes in working directory) > > > modified: docs/python/_sources/generated/pyarrow.Array.rst.txt > > modified: docs/python/_sources/generated/pyarrow.Column.rst.txt > > modified: docs/python/_sources/generated/pyarrow.Field.rst.txt > > modified: docs/python/_sources/generated/pyarrow.Schema.rst.txt > > modified: docs/python/_sources/generated/pyarrow.lib.Array.rst.txt > > modified: docs/python/generated/pyarrow.Array.html > > modified: docs/python/generated/pyarrow.Column.html > > modified: docs/python/generated/pyarrow.Field.html > > modified: docs/python/generated/pyarrow.Schema.html > > modified: docs/python/generated/pyarrow.lib.Array.html > > I have tried a few things but no luck, including: > * adding these files manually (git add docs/python/...) > * stage only python docs > * commit the staged change first and try to stage the rest of files > * check git-ignore (there is none) > * git add --force > ... > > Unfortunately none of these seems to work. Anyone has a clue what might be > going on? > > A separate issue is that I noticed Java doc is comment out from > dev/gen_apidocs/create_documents.sh > (probably due to Java8 lint-checks). I have uncommented it and it works. > (Opened ARROW-3003) > > On Mon, Aug 6, 2018 at 9:33 PM, Phillip Cloud wrote: > > > Excellent. Thanks for testing that out Krisztián. > > > > On Mon, Aug 6, 2018 at 5:32 PM Krisztián Szűcs < > szucs.kriszt...@gmail.com> > > wrote: > > > > > Great! Just tested to be sure, the osx wheels (all three python > versions) > > > are working. > > > > > > On Aug 6 2018, at 11:16 pm, Wes McKinney wrote: > > > > > > > > Wheels are up https://pypi.org/project/pyarrow/#files > > > > On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud > > wrote: > > > > > The release artifacts are uploaded to SVN. > > > > > > > > > > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney > > > wrote: > > > > > > We can comment out the JS doc build for now, so it's not a > blocker, > > > > > > but would be good to fix > > > > > > > > > > > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor > > > wrote: > > > > > > > Looks like typedoc is using the wrong version of typescript. I > > can > > > take a > > > > > > > look at updating it later this afternoon if it’s blocking the > > > release. > > > > > > > > > > > > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin > > > wrote: > > > > > > > > I was trying to build the documentation but hit a JavaScript > > > error. I am > > > > > > > > not familiar with JavaScript and TypeScript ... Any > suggestion? > > > > > > > > > > > > > > > > Error log: > > > > > > > > /apache-arrow/arrow/js /apache-arrow > > > > > > > > + npm install > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > > /chokidar/fsevents: > > > > > > > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > > > > architecture: > > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > /sane/fsevents: > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > > > > architecture: > > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > > > > /watchpack/chokidar/fsevents: > > > > > > > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > > > > architecture: > > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 > but > > > none was > > > > > > > > installed. > > > > > > > > > > > > > > > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of > > > webpack@ > > > > > > ^2.0.0 > > > > > > > > || > > > > > > > > ^3.0.0 but none was installed. > > > > > > > > > > > > > > > > + npm run doc > > > > > > > > > > > > > > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > > > > > > > > > shx rm -rf ./doc && typedoc --mode file --out doc > > src/Arrow.ts > > > > > > > > > > > > > > > > > > > > > > > > Using TypeScript 2.7.2 from > > > > > > > > > > > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib > > > > > > > > > > > > > > > > Error: /apache-arrow/arrow/js/src/vector.ts(161) > > > > > > > > Property 'childData' does not exist on type 'any[] | > Data'. > > > > > > > > Property 'childData' does not exist on type 'any[]'. > > > > > > > > > > > > > > > > npm ERR! Linux 4.9.93-linuxkit-aufs
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
I don’t think so. I manually checked it out under arrow/site. On Mon, Aug 6, 2018 at 6:20 PM Phillip Cloud wrote: > Is it being checked out as a submodule? > > On Mon, Aug 6, 2018 at 6:07 PM Li Jin wrote: > > > I've generated the docs but unfortunately is stuck on a weird git issue. > > For some reason git doesn't want to add a few files after "git add ." (10 > > python files): > > > > modified: docs/python/parquet.html > > > > modified: docs/python/plasma.html > > > > modified: docs/python/search.html > > > > modified: docs/python/searchindex.js > > > > > > Changes not staged for commit: > > > > (use "git add ..." to update what will be committed) > > > > (use "git checkout -- ..." to discard changes in working > directory) > > > > > > modified: docs/python/_sources/generated/pyarrow.Array.rst.txt > > > > modified: docs/python/_sources/generated/pyarrow.Column.rst.txt > > > > modified: docs/python/_sources/generated/pyarrow.Field.rst.txt > > > > modified: docs/python/_sources/generated/pyarrow.Schema.rst.txt > > > > modified: docs/python/_sources/generated/pyarrow.lib.Array.rst.txt > > > > modified: docs/python/generated/pyarrow.Array.html > > > > modified: docs/python/generated/pyarrow.Column.html > > > > modified: docs/python/generated/pyarrow.Field.html > > > > modified: docs/python/generated/pyarrow.Schema.html > > > > modified: docs/python/generated/pyarrow.lib.Array.html > > > > I have tried a few things but no luck, including: > > * adding these files manually (git add docs/python/...) > > * stage only python docs > > * commit the staged change first and try to stage the rest of files > > * check git-ignore (there is none) > > * git add --force > > ... > > > > Unfortunately none of these seems to work. Anyone has a clue what might > be > > going on? > > > > A separate issue is that I noticed Java doc is comment out from > > dev/gen_apidocs/create_documents.sh > > (probably due to Java8 lint-checks). I have uncommented it and it works. > > (Opened ARROW-3003) > > > > On Mon, Aug 6, 2018 at 9:33 PM, Phillip Cloud wrote: > > > > > Excellent. Thanks for testing that out Krisztián. > > > > > > On Mon, Aug 6, 2018 at 5:32 PM Krisztián Szűcs < > > szucs.kriszt...@gmail.com> > > > wrote: > > > > > > > Great! Just tested to be sure, the osx wheels (all three python > > versions) > > > > are working. > > > > > > > > On Aug 6 2018, at 11:16 pm, Wes McKinney > wrote: > > > > > > > > > > Wheels are up https://pypi.org/project/pyarrow/#files > > > > > On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud > > > wrote: > > > > > > The release artifacts are uploaded to SVN. > > > > > > > > > > > > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney > > > > > wrote: > > > > > > > We can comment out the JS doc build for now, so it's not a > > blocker, > > > > > > > but would be good to fix > > > > > > > > > > > > > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor < > ptay...@apache.org> > > > > wrote: > > > > > > > > Looks like typedoc is using the wrong version of typescript. > I > > > can > > > > take a > > > > > > > > look at updating it later this afternoon if it’s blocking the > > > > release. > > > > > > > > > > > > > > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin < > ice.xell...@gmail.com> > > > > wrote: > > > > > > > > > I was trying to build the documentation but hit a > JavaScript > > > > error. I am > > > > > > > > > not familiar with JavaScript and TypeScript ... Any > > suggestion? > > > > > > > > > > > > > > > > > > Error log: > > > > > > > > > /apache-arrow/arrow/js /apache-arrow > > > > > > > > > + npm install > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > > > /chokidar/fsevents: > > > > > > > > > > > > > > > > > > npm WARN notsup Not compatible with your operating system > or > > > > > > > architecture: > > > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > /sane/fsevents: > > > > > > > > > npm WARN notsup Not compatible with your operating system > or > > > > > > > architecture: > > > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > > > > > /watchpack/chokidar/fsevents: > > > > > > > > > > > > > > > > > > npm WARN notsup Not compatible with your operating system > or > > > > > > > architecture: > > > > > > > > > fsevents@1.2.4 > > > > > > > > > > > > > > > > > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 > > but > > > > none was > > > > > > > > > installed. > > > > > > > > > > > > > > > > > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of > > > > webpack@ > > > > > > > ^2.0.0 > > > > > > > > > || > > > > > > > > > ^3.0.0 but none was installed. > > > > > > > > > > > > > > > > > > + npm run doc > > > > > > > > > > > > > > > > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > > > > > > > > > > shx rm -rf ./doc && typedoc --mode file --out doc > > > src/Arrow.ts > > >
Re: [DISCUSS] Contribution of Gandiva to Apache Arrow
hi Ravindra, Thanks for the update. I will start a vote to approve the donation of the codebase soon if there are no objections. The merge will be subject to completing the standard ASF IP Clearance and having a Software Grant filed by Dremio. I noticed some things in the Gandiva codebase vs. the Arrow codebase that will need to be changed, such as having a separate include/ directory (we don't do that; very Google-style in this regard). Let us know when you're ready to start some code review to prepare the initial integration to help minimize code churn. best Wes On Mon, Aug 6, 2018 at 1:48 AM, Pindikura Ravindra wrote: > Sorry for the delay, Wes. > > We plan to finish off some feature development work next week (eg. Filter > support), and then, start work on merging gandiva into arrow (third week of > august). Since we have been using only the public arrow APIs in the code > and in tests, we do not foresee any major hiccups on the code side. We > think the challenges will be in build/packaging, especially related to LLVM. > > I'll post on this thread again with specifics once we begin the work to > integrate. > > thanks & regards, > ravindra. > > > On Sat, Aug 4, 2018 at 9:08 PM Wes McKinney wrote: > >> It's been about 2 weeks since the discussion started. What are the >> next steps? It would be great to hear from one or more of the Gandiva >> developers directly here. >> >> best >> Wes >> >> On Wed, Jul 25, 2018 at 5:11 PM, Li Jin wrote: >> > Although I am not familiar with LLVM details but I think at the high >> level >> > such component fits in Arrow project. Hopefully I can learn more and help >> > as well. >> > >> > +1 >> > >> > On Wed, Jul 25, 2018 at 8:45 PM, Philipp Moritz >> wrote: >> > >> >> +1 on merging it and also agreed with Uwe that we will need to deal with >> >> LLVM version conflicts. In addition it would be good to come up with a >> plan >> >> on how it can be useful for other DataFrame open source projects. Having >> >> end-to-end applications that let people profit from this code will help >> >> adoption/development a lot I think. >> >> >> >> On Wed, Jul 25, 2018 at 6:56 AM, Uwe L. Korn wrote: >> >> >> >> > Having it in Arrow also will enable us to better promote it on the >> Python >> >> > side with the pyarrow package. It will be a great addition but we will >> >> need >> >> > to figure out then the problem of dealing with in-memory LLVM >> versions so >> >> > that we don't conflict with packages like Numba. This is a problem on >> the >> >> > Python packaging side we should address early but won't be a blocker >> in >> >> > getting Gandiva into the Arrow repositories and infrastructure. >> >> > >> >> > Uwe >> >> > >> >> > On Tue, Jul 24, 2018, at 9:39 PM, Phillip Cloud wrote: >> >> > > While I'm not a level 83 LLVM wizard like Antoine :) I have a small >> >> > amount >> >> > > of experience with it and would also be happy to review/merge >> patches. >> >> > > >> >> > > Having Gandiva in arrow will simplify packaging and building the >> >> library, >> >> > > which IME has always been annoying with large cross-platform and >> >> > > cross-language projects. >> >> > > >> >> > > I think this would be a great addition to Arrow. >> >> > > >> >> > > On Tue, Jul 24, 2018 at 8:46 AM Uwe L. Korn >> wrote: >> >> > > >> >> > > > Small update: >> >> > > > > * CentOS 6 and later (we use the RedHat devtoolset-2 compiler >> >> > toolchain) >> >> > > > >> >> > > > We still build the Python wheels on CentOS 5. This should be no >> >> > problem as >> >> > > > llvmlite can also be build on that OS. >> >> > > > >> >> > > > It looks like there will be a new Linux Python wheel standard soon >> >> > where >> >> > > > we will upgrade to CentOS 6. >> >> > > > >> >> > > > Cheers, >> >> > > > >> >> > > > Uwe >> >> > > > >> >> > >> >> >>
Re: Arrow & plasma - java sample to store complex objects
hi Gerard, This is the right place to ask questions. The Slack channel was closed (see prior discussions on the mailing list); few Java developers were on Slack anyway so it wouldn't have been a good place to get help. Using Java with Plasma is very bleeding edge territory. I don't know if anyone has an example yet of using Plasma with end-to-end Arrow columnar read and write. I would say it's definitely the domain of developers working on the Java codebase to build out support tooling for these workflows. We'd be glad to have you involved. For general Arrow workflows, I would recommend looking at the Arrow conversion paths in the Spark SQL codebase. There we have record batches being streamed to Python and then results received back on the JVM side. - Wes On Mon, Aug 6, 2018 at 10:19 AM, Gérard Dupont wrote: > Hi, > Not sure this is the right channel for a "user" oriented question but the > slack channel on heroku seams to be down... > > TL;DR: is there some hidden tutorial/java samples to store complex data > objects in arrow and access (put/get) with plasma? I'm currently exploring > the unit test from the java part of the source, but it's not really > obvious... > > So, I'm starting with arrow and actually I went in for the plasma object > store which should address an issue I currently have: sharing objects > between process on multiples servers to serve as initial starting point in > computations. > > The computation parts are already done since the need for distribution just > recently emerged, and I'm trying to see if I can port the data object > within arrow to distribute them over plasma. So far so good: I can launch > the plasma_store and access it through the ObjectStoreLink API. But it's > really on the byte array level. Any advice or best practice on how to > convert existing data model to arrow compliant one? Should I look into the > Arrow Schema example? > > Thanks for any pointer. > > Cheers, > -- > Gérard Dupont
[jira] [Created] (ARROW-3004) [Documentation] dev/gen_apidocs.sh builds docs for master rather than a pinned commit
Wes McKinney created ARROW-3004: --- Summary: [Documentation] dev/gen_apidocs.sh builds docs for master rather than a pinned commit Key: ARROW-3004 URL: https://issues.apache.org/jira/browse/ARROW-3004 Project: Apache Arrow Issue Type: Bug Components: Documentation Reporter: Wes McKinney Fix For: 0.11.0 This is inflexible because there may be fixed to the documentation build scripts -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
Conda packages (arrow-cpp, parquet-cpp, pyarrow) are also available now on conda-forge. Tested pyarrow on OSX + PY27/35/36, all of them are working. On Aug 6 2018, at 11:16 pm, Wes McKinney wrote: > > Wheels are up https://pypi.org/project/pyarrow/#files > On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud wrote: > > The release artifacts are uploaded to SVN. > > > > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney wrote: > > > We can comment out the JS doc build for now, so it's not a blocker, > > > but would be good to fix > > > > > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor wrote: > > > > Looks like typedoc is using the wrong version of typescript. I can take > > > > a > > > > look at updating it later this afternoon if it’s blocking the release. > > > > > > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin wrote: > > > > > I was trying to build the documentation but hit a JavaScript error. I > > > > > am > > > > > not familiar with JavaScript and TypeScript ... Any suggestion? > > > > > > > > > > Error log: > > > > > /apache-arrow/arrow/js /apache-arrow > > > > > + npm install > > > > > npm WARN optional Skipping failed optional dependency > > > /chokidar/fsevents: > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > architecture: > > > > > fsevents@1.2.4 > > > > > > > > > > npm WARN optional Skipping failed optional dependency /sane/fsevents: > > > > > npm WARN notsup Not compatible with your operating system or > > > architecture: > > > > > fsevents@1.2.4 > > > > > > > > > > npm WARN optional Skipping failed optional dependency > > > > > /watchpack/chokidar/fsevents: > > > > > > > > > > npm WARN notsup Not compatible with your operating system or > > > architecture: > > > > > fsevents@1.2.4 > > > > > > > > > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none was > > > > > installed. > > > > > > > > > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@ > > > ^2.0.0 > > > > > || > > > > > ^3.0.0 but none was installed. > > > > > > > > > > + npm run doc > > > > > > > > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js > > > > > > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts > > > > > > > > > > > > > > > Using TypeScript 2.7.2 from > > > > > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib > > > > > > > > > > Error: /apache-arrow/arrow/js/src/vector.ts(161) > > > > > Property 'childData' does not exist on type 'any[] | Data'. > > > > > Property 'childData' does not exist on type 'any[]'. > > > > > > > > > > npm ERR! Linux 4.9.93-linuxkit-aufs > > > > > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" > > > > > npm ERR! node v8.10.0 > > > > > npm ERR! npm v3.5.2 > > > > > npm ERR! code ELIFECYCLE > > > > > npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode > > > file > > > > > --out doc src/Arrow.ts` > > > > > > > > > > npm ERR! Exit status 4 > > > > > npm ERR! > > > > > npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf ./doc > > > && > > > > > typedoc --mode file --out doc src/Arrow.ts'. > > > > > > > > > > npm ERR! Make sure you have the latest version of node.js and npm > > > > > installed. > > > > > > > > > > npm ERR! If you do, this is most likely a problem with the > > > > > apache-arrow > > > > > package, > > > > > > > > > > npm ERR! not with npm itself. > > > > > npm ERR! Tell the author that this fails on your system: > > > > > npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc > > > src/Arrow.ts > > > > > > > > > > npm ERR! You can get information on how to open an issue for this > > > project > > > > > with: > > > > > > > > > > npm ERR! npm bugs apache-arrow > > > > > npm ERR! Or if that isn't available, you can get their info via: > > > > > npm ERR! npm owner ls apache-arrow > > > > > npm ERR! There is likely additional logging output above. > > > > > > > > > > npm ERR! Please include the following file with any support request: > > > > > npm ERR! /apache-arrow/arrow/js/npm-debug.log > > > > > > > > > > On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney > > > wrote: > > > > > > > > > > > I have just rebased master on tag apache-arrow-0.10.0. I will rebase > > > > > > the PRs that might be affected > > > > > > > > > > > > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs > > > > > > wrote: > > > > > > > Wes, I can help You with the forge packages. I'm creating the PRs. > > > > > > > > > > > > > > On Aug 6 2018, at 8:52 pm, Wes McKinney > > > wrote: > > > > > > > > > > > > > > > > I'm going to start kicking along the conda-forge packages. If > > > anyone > > > > > > > > can assist with these, it would be much appreciated. > > > > > > > > > > > > > > > > I will update the website and write a blog post summarizing the > > > 0.10.0 > > > > > > > > release (it's the biggest release we've ever done). > > > > > > > > > > > > > > > > I'm going to rebase master as soon as I merge the build for > > > > > > > > this PR > > > > > > > > completes: https://t
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
I just pushed the documentation update. I had to comment out the JS doc build but otherwise didn't have any issues On Mon, Aug 6, 2018 at 8:59 PM, Krisztián Szűcs wrote: > Conda packages (arrow-cpp, parquet-cpp, pyarrow) are also available now on > conda-forge. > > Tested pyarrow on OSX + PY27/35/36, all of them are working. > On Aug 6 2018, at 11:16 pm, Wes McKinney wrote: >> >> Wheels are up https://pypi.org/project/pyarrow/#files >> On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud wrote: >> > The release artifacts are uploaded to SVN. >> > >> > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney wrote: >> > > We can comment out the JS doc build for now, so it's not a blocker, >> > > but would be good to fix >> > > >> > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor wrote: >> > > > Looks like typedoc is using the wrong version of typescript. I can >> > > > take a >> > > > look at updating it later this afternoon if it’s blocking the release. >> > > > >> > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin wrote: >> > > > > I was trying to build the documentation but hit a JavaScript error. >> > > > > I am >> > > > > not familiar with JavaScript and TypeScript ... Any suggestion? >> > > > > >> > > > > Error log: >> > > > > /apache-arrow/arrow/js /apache-arrow >> > > > > + npm install >> > > > > npm WARN optional Skipping failed optional dependency >> > > /chokidar/fsevents: >> > > > > >> > > > > npm WARN notsup Not compatible with your operating system or >> > > architecture: >> > > > > fsevents@1.2.4 >> > > > > >> > > > > npm WARN optional Skipping failed optional dependency /sane/fsevents: >> > > > > npm WARN notsup Not compatible with your operating system or >> > > architecture: >> > > > > fsevents@1.2.4 >> > > > > >> > > > > npm WARN optional Skipping failed optional dependency >> > > > > /watchpack/chokidar/fsevents: >> > > > > >> > > > > npm WARN notsup Not compatible with your operating system or >> > > architecture: >> > > > > fsevents@1.2.4 >> > > > > >> > > > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none >> > > > > was >> > > > > installed. >> > > > > >> > > > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@ >> > > ^2.0.0 >> > > > > || >> > > > > ^3.0.0 but none was installed. >> > > > > >> > > > > + npm run doc >> > > > > >> > > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js >> > > > > > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts >> > > > > >> > > > > >> > > > > Using TypeScript 2.7.2 from >> > > > > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib >> > > > > >> > > > > Error: /apache-arrow/arrow/js/src/vector.ts(161) >> > > > > Property 'childData' does not exist on type 'any[] | Data'. >> > > > > Property 'childData' does not exist on type 'any[]'. >> > > > > >> > > > > npm ERR! Linux 4.9.93-linuxkit-aufs >> > > > > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" >> > > > > npm ERR! node v8.10.0 >> > > > > npm ERR! npm v3.5.2 >> > > > > npm ERR! code ELIFECYCLE >> > > > > npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode >> > > file >> > > > > --out doc src/Arrow.ts` >> > > > > >> > > > > npm ERR! Exit status 4 >> > > > > npm ERR! >> > > > > npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf >> > > > > ./doc >> > > && >> > > > > typedoc --mode file --out doc src/Arrow.ts'. >> > > > > >> > > > > npm ERR! Make sure you have the latest version of node.js and npm >> > > > > installed. >> > > > > >> > > > > npm ERR! If you do, this is most likely a problem with the >> > > > > apache-arrow >> > > > > package, >> > > > > >> > > > > npm ERR! not with npm itself. >> > > > > npm ERR! Tell the author that this fails on your system: >> > > > > npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc >> > > src/Arrow.ts >> > > > > >> > > > > npm ERR! You can get information on how to open an issue for this >> > > project >> > > > > with: >> > > > > >> > > > > npm ERR! npm bugs apache-arrow >> > > > > npm ERR! Or if that isn't available, you can get their info via: >> > > > > npm ERR! npm owner ls apache-arrow >> > > > > npm ERR! There is likely additional logging output above. >> > > > > >> > > > > npm ERR! Please include the following file with any support request: >> > > > > npm ERR! /apache-arrow/arrow/js/npm-debug.log >> > > > > >> > > > > On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney >> > > wrote: >> > > > > >> > > > > > I have just rebased master on tag apache-arrow-0.10.0. I will >> > > > > > rebase >> > > > > > the PRs that might be affected >> > > > > > >> > > > > > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs >> > > > > > wrote: >> > > > > > > Wes, I can help You with the forge packages. I'm creating the >> > > > > > > PRs. >> > > > > > > >> > > > > > > On Aug 6 2018, at 8:52 pm, Wes McKinney >> > > wrote: >> > > > > > > > >> > > > > > > > I'm going to start kicking along the conda-forge packages. If >> > > anyone >> > > > > > > > can assist with these, it would
Re: [DISCUSS] Re-think CI strategy?
Thanks Wes. As for Python 3.5, 3.6, and 3.7, I think testing any one of them should be sufficient (I can't recall any errors that happened with one version and not the other). On Mon, Aug 6, 2018 at 12:01 PM Wes McKinney wrote: > @Robert, it looks like NumPy is making LTS releases until Jan 1, 2020 > > > https://docs.scipy.org/doc/numpy-1.14.0/neps/dropping-python2.7-proposal.html > > Based on this, I think it's fine for us to continue to support Python > 2.7 until then. It's only 16 months away; are you all ready for the > next decade? > > We should also discuss if we want to continue to build and test Python > 3.5. From download statistics it appears that there are 5-10x as many > Python 3.6 users as 3.5. I would prefer to drop 3.5 and begin > supporting 3.7 soon. > > @Antoine, I think we can avoid building the C++ codebase 3 times, but > it will require a bit of retooling of the scripts. The reason that > ccache isn't working properly is probably because the Python include > directory is being included even for compilation units that do not use > the Python C API. > https://github.com/apache/arrow/blob/master/cpp/CMakeLists.txt#L721. > I'm opening a JIRA about fixing this > https://issues.apache.org/jira/browse/ARROW-2994 > > Created https://issues.apache.org/jira/browse/ARROW-2995 about > removing the redundant build cycle > > On Mon, Aug 6, 2018 at 2:19 PM, Robert Nishihara > wrote: > >> > >> Also, at this point we're sometimes hitting the 50 minutes time limit on > >> our slowest Travis-CI matrix job, which means we have to restart it... > >> making the build even slower. > >> > > Only a short-term fix, but Travis can lengthen the max build time if you > > email them and ask them to. >
[jira] [Created] (ARROW-3005) [Website] Update website and write blog post for 0.10.0 release announcement
Wes McKinney created ARROW-3005: --- Summary: [Website] Update website and write blog post for 0.10.0 release announcement Key: ARROW-3005 URL: https://issues.apache.org/jira/browse/ARROW-3005 Project: Apache Arrow Issue Type: Improvement Components: Website Reporter: Wes McKinney Assignee: Wes McKinney Fix For: 0.10.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [RESULT][VOTE] Release Apache Arrow 0.10.0 (RC1)
OK, website update PR is up https://github.com/apache/arrow/pull/2387 Everyone OK to announce at 12pm Eastern tomorrow? On Mon, Aug 6, 2018 at 9:03 PM, Wes McKinney wrote: > I just pushed the documentation update. I had to comment out the JS > doc build but otherwise didn't have any issues > > On Mon, Aug 6, 2018 at 8:59 PM, Krisztián Szűcs > wrote: >> Conda packages (arrow-cpp, parquet-cpp, pyarrow) are also available now on >> conda-forge. >> >> Tested pyarrow on OSX + PY27/35/36, all of them are working. >> On Aug 6 2018, at 11:16 pm, Wes McKinney wrote: >>> >>> Wheels are up https://pypi.org/project/pyarrow/#files >>> On Mon, Aug 6, 2018 at 4:38 PM, Phillip Cloud wrote: >>> > The release artifacts are uploaded to SVN. >>> > >>> > On Mon, Aug 6, 2018 at 4:01 PM Wes McKinney wrote: >>> > > We can comment out the JS doc build for now, so it's not a blocker, >>> > > but would be good to fix >>> > > >>> > > On Mon, Aug 6, 2018 at 3:59 PM, Paul Taylor wrote: >>> > > > Looks like typedoc is using the wrong version of typescript. I can >>> > > > take a >>> > > > look at updating it later this afternoon if it’s blocking the release. >>> > > > >>> > > > On Mon, Aug 6, 2018 at 12:55 PM Li Jin wrote: >>> > > > > I was trying to build the documentation but hit a JavaScript error. >>> > > > > I am >>> > > > > not familiar with JavaScript and TypeScript ... Any suggestion? >>> > > > > >>> > > > > Error log: >>> > > > > /apache-arrow/arrow/js /apache-arrow >>> > > > > + npm install >>> > > > > npm WARN optional Skipping failed optional dependency >>> > > /chokidar/fsevents: >>> > > > > >>> > > > > npm WARN notsup Not compatible with your operating system or >>> > > architecture: >>> > > > > fsevents@1.2.4 >>> > > > > >>> > > > > npm WARN optional Skipping failed optional dependency >>> > > > > /sane/fsevents: >>> > > > > npm WARN notsup Not compatible with your operating system or >>> > > architecture: >>> > > > > fsevents@1.2.4 >>> > > > > >>> > > > > npm WARN optional Skipping failed optional dependency >>> > > > > /watchpack/chokidar/fsevents: >>> > > > > >>> > > > > npm WARN notsup Not compatible with your operating system or >>> > > architecture: >>> > > > > fsevents@1.2.4 >>> > > > > >>> > > > > npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none >>> > > > > was >>> > > > > installed. >>> > > > > >>> > > > > npm WARN uglifyjs-webpack-plugin@1.1.6 requires a peer of webpack@ >>> > > ^2.0.0 >>> > > > > || >>> > > > > ^3.0.0 but none was installed. >>> > > > > >>> > > > > + npm run doc >>> > > > > >>> > > > > > apache-arrow@0.3.0 doc /apache-arrow/arrow/js >>> > > > > > shx rm -rf ./doc && typedoc --mode file --out doc src/Arrow.ts >>> > > > > >>> > > > > >>> > > > > Using TypeScript 2.7.2 from >>> > > > > /apache-arrow/arrow/js/node_modules/typedoc/node_modules/typescript/lib >>> > > > > >>> > > > > Error: /apache-arrow/arrow/js/src/vector.ts(161) >>> > > > > Property 'childData' does not exist on type 'any[] | Data'. >>> > > > > Property 'childData' does not exist on type 'any[]'. >>> > > > > >>> > > > > npm ERR! Linux 4.9.93-linuxkit-aufs >>> > > > > npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "doc" >>> > > > > npm ERR! node v8.10.0 >>> > > > > npm ERR! npm v3.5.2 >>> > > > > npm ERR! code ELIFECYCLE >>> > > > > npm ERR! apache-arrow@0.3.0 doc: `shx rm -rf ./doc && typedoc --mode >>> > > file >>> > > > > --out doc src/Arrow.ts` >>> > > > > >>> > > > > npm ERR! Exit status 4 >>> > > > > npm ERR! >>> > > > > npm ERR! Failed at the apache-arrow@0.3.0 doc script 'shx rm -rf >>> > > > > ./doc >>> > > && >>> > > > > typedoc --mode file --out doc src/Arrow.ts'. >>> > > > > >>> > > > > npm ERR! Make sure you have the latest version of node.js and npm >>> > > > > installed. >>> > > > > >>> > > > > npm ERR! If you do, this is most likely a problem with the >>> > > > > apache-arrow >>> > > > > package, >>> > > > > >>> > > > > npm ERR! not with npm itself. >>> > > > > npm ERR! Tell the author that this fails on your system: >>> > > > > npm ERR! shx rm -rf ./doc && typedoc --mode file --out doc >>> > > src/Arrow.ts >>> > > > > >>> > > > > npm ERR! You can get information on how to open an issue for this >>> > > project >>> > > > > with: >>> > > > > >>> > > > > npm ERR! npm bugs apache-arrow >>> > > > > npm ERR! Or if that isn't available, you can get their info via: >>> > > > > npm ERR! npm owner ls apache-arrow >>> > > > > npm ERR! There is likely additional logging output above. >>> > > > > >>> > > > > npm ERR! Please include the following file with any support request: >>> > > > > npm ERR! /apache-arrow/arrow/js/npm-debug.log >>> > > > > >>> > > > > On Mon, Aug 6, 2018 at 7:41 PM, Wes McKinney >>> > > wrote: >>> > > > > >>> > > > > > I have just rebased master on tag apache-arrow-0.10.0. I will >>> > > > > > rebase >>> > > > > > the PRs that might be affected >>> > > > > > >>> > > > > > On Mon, Aug 6, 2018 at 2:55 PM, Krisztián Szűcs >>> > > > > > wrote: >>> > > > > > > Wes, I can
Re: [DISCUSS] Re-think CI strategy?
It would be good to test all Python versions in a cron build, but I agree we may not need to test all Python 3 versions in per-commit builds. Regards Antoine. Le 07/08/2018 à 03:14, Robert Nishihara a écrit : > Thanks Wes. > > As for Python 3.5, 3.6, and 3.7, I think testing any one of them should be > sufficient (I can't recall any errors that happened with one version and > not the other). > > On Mon, Aug 6, 2018 at 12:01 PM Wes McKinney wrote: > >> @Robert, it looks like NumPy is making LTS releases until Jan 1, 2020 >> >> >> https://docs.scipy.org/doc/numpy-1.14.0/neps/dropping-python2.7-proposal.html >> >> Based on this, I think it's fine for us to continue to support Python >> 2.7 until then. It's only 16 months away; are you all ready for the >> next decade? >> >> We should also discuss if we want to continue to build and test Python >> 3.5. From download statistics it appears that there are 5-10x as many >> Python 3.6 users as 3.5. I would prefer to drop 3.5 and begin >> supporting 3.7 soon. >> >> @Antoine, I think we can avoid building the C++ codebase 3 times, but >> it will require a bit of retooling of the scripts. The reason that >> ccache isn't working properly is probably because the Python include >> directory is being included even for compilation units that do not use >> the Python C API. >> https://github.com/apache/arrow/blob/master/cpp/CMakeLists.txt#L721. >> I'm opening a JIRA about fixing this >> https://issues.apache.org/jira/browse/ARROW-2994 >> >> Created https://issues.apache.org/jira/browse/ARROW-2995 about >> removing the redundant build cycle >> >> On Mon, Aug 6, 2018 at 2:19 PM, Robert Nishihara >> wrote: Also, at this point we're sometimes hitting the 50 minutes time limit on our slowest Travis-CI matrix job, which means we have to restart it... making the build even slower. >>> Only a short-term fix, but Travis can lengthen the max build time if you >>> email them and ask them to. >> >
