RE: [Celebrate] Arrow has reached 2000 stargeezers
Congratulations to the entire Arrow team!!! -Original Message- From: Wes McKinney [mailto:wesmck...@gmail.com] Sent: Monday, May 28, 2018 9:20 PM To: dev@arrow.apache.org Subject: Re: [Celebrate] Arrow has reached 2000 stargeezers Congrats all! The journey continues On Mon, May 28, 2018 at 9:43 AM, Krisztián Szűcs wrote: > Which makes Arrow the 33rd most starred Apache repository (out of 1555, > according to github). > > Congratulations!
[jira] [Created] (ARROW-2642) [Python] Fail building parquet binding on Windows
Antoine Pitrou created ARROW-2642: - Summary: [Python] Fail building parquet binding on Windows Key: ARROW-2642 URL: https://issues.apache.org/jira/browse/ARROW-2642 Project: Apache Arrow Issue Type: Bug Components: Python Affects Versions: 0.9.0 Reporter: Antoine Pitrou For some reason I get the following error. I'm not sure why Thrift is needed here: {code} -- Found the Parquet library: C:/Miniconda3/envs/arrow/Library/lib/parquet.lib -- THRIFT_HOME: -- Thrift compiler/libraries NOT found: (THRIFT_INCLUDE_DIR-NOTFOUND, THRIFT_ST ATIC_LIB-NOTFOUND). Looked in system search paths. -- Boost version: 1.66.0 -- Found the following Boost libraries: -- regex Added static library dependency boost_regex: C:/Miniconda3/envs/arrow/Library/li b/libboost_regex.lib Added static library dependency parquet: C:/Miniconda3/envs/arrow/Library/lib/pa rquet_static.lib CMake Error at C:/t/arrow/cpp/cmake_modules/BuildUtils.cmake:88 (message): No static or shared library provided for thrift Call Stack (most recent call first): CMakeLists.txt:376 (ADD_THIRDPARTY_LIB) {code} The {{thrift-cpp}} package from conda-forge is installed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2643) [C++] Travis-CI build failure with cpp toolchain enabled
Antoine Pitrou created ARROW-2643: - Summary: [C++] Travis-CI build failure with cpp toolchain enabled Key: ARROW-2643 URL: https://issues.apache.org/jira/browse/ARROW-2643 Project: Apache Arrow Issue Type: Bug Components: C++, Continuous Integration Affects Versions: 0.9.0 Reporter: Antoine Pitrou This is a new failure, perhaps triggered by a conda-forge package update. See example at https://travis-ci.org/apache/arrow/jobs/385002355#L2235 {code} /usr/bin/ld: /home/travis/build/apache/arrow/cpp-toolchain/lib/libz.a(deflate.o): relocation R_X86_64_32S against `zcalloc' can not be used when making a shared object; recompile with -fPIC {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
JDBC Adapter PR - 1759
Hi Sid, Laurent, Uwe, Any idea when can someone take a look at the PR https://github.com/apache/arrow/pull/1759/. Laurent had given bunch of comments earlier and now we have taken care of most of those. We have also added multiple test cases. It will be great if someone can take a look. Regards, -Atul
Re: Peak memory usage for pyarrow.parquet.read_table
Following up on what I have found with Uwe's advice and poking around the code base. * `columns=` helped but it was because forced me to realize I did not need all of the columns at once every time. No particular column was significantly worse in memory usage. * There seems to be some interaction between `parquet::internal::RecordReader` and `arrow::PoolBuffer` or `arrow::DefaultMemoryPool`. `RecordReader` request an allocation to hold the entire column in memory without compression/encoding even though Arrow supports dictionary encoding (and the column is dictionary encoded). I imagine `RecordReader` requests enough memory to hold the data without encoding/compression for good reason (perhaps more robust assumptions about the underlying memory pool?), but is there a way to request only the memory require for dictionary encoding when it is an option? My (incomplete) understanding comes from the surrounding lines here https://github.com/apache/parquet-cpp/blob/c405bf36506ec584e8009a6d53349277e600467d/src/parquet/arrow/record_reader.cc#L232 . On Wed, Apr 25, 2018 at 2:23 PM Bryant Menn wrote: > Uwe, > > I'll try pinpointing things further with `columns=` and try to reproduce > what I find with data I can share. > > Thanks for the pointer. > > -Bryant > > On Wed, Apr 25, 2018 at 2:10 PM Uwe L. Korn wrote: > >> No, there is no need to pass any options on reading. Sometimes they are >> beneficial depending on what you want to achieve but defaults are ok, too. >> >> I'm not sure if you're able to post an example but it would be nice if >> you could post the resulting Arrow schema from the table. It might be >> related to a specific type. A quick way to debug this on your side would >> also be to specify only a subset of columns to read using the `columns=` >> attribute on read_table. Maybe you can already pinpoint the memory problems >> to a specific column. Having these hints would it make easier for us to >> diagnose what the underlying problem is. >> >> Uwe >> >> On Wed, Apr 25, 2018, at 8:06 PM, Bryant Menn wrote: >> > Uwe, >> > >> > I am not. Should I be? I forgot to mention earlier that the Parquet file >> > came from Spark/PySpark. >> > >> > On Wed, Apr 25, 2018 at 1:32 PM Uwe L. Korn wrote: >> > >> > > Hello Bryant, >> > > >> > > are you using any options on `pyarrow.parquet.read_table` or a >> possible >> > > `to_pandas` afterwards? >> > > >> > > Uwe >> > > >> > > On Wed, Apr 25, 2018, at 7:27 PM, Bryant Menn wrote: >> > > > I tried reading a Parquet file (<200MB, lots of text with snappy) >> using >> > > > read_table and saw the memory usage peak over 8GB before settling >> back >> > > down >> > > > to ~200MB. This surprised me as I was expecting to be able to >> handle a >> > > > Parquet file of this size with much less RAM (doing some processing >> with >> > > > smaller VMs). >> > > > >> > > > I am not sure if this expected, but I thought I might check with >> everyone >> > > > here and learn something new. Poking around it seems to be related >> with >> > > > ParquetReader.read_all? >> > > > >> > > > Thanks in advance, >> > > > Bryant >> > > >> >
[jira] [Created] (ARROW-2644) [Python] parquet binding fails building on AppVeyor
Antoine Pitrou created ARROW-2644: - Summary: [Python] parquet binding fails building on AppVeyor Key: ARROW-2644 URL: https://issues.apache.org/jira/browse/ARROW-2644 Project: Apache Arrow Issue Type: Bug Components: Python Affects Versions: 0.9.0 Reporter: Antoine Pitrou Assignee: Antoine Pitrou This is a new issue (perhaps due to a new Cython version). See e.g. https://ci.appveyor.com/project/pitrou/arrow/build/1.0.509/job/dxdqcdk30kmiy6pd#L4291 Excerpt: {code} -- Running cmake --build for pyarrow C:\Program Files (x86)\CMake\bin\cmake.exe --build . --config release [1/8] cmd.exe /C "cd /D C:\projects\arrow\python\build\temp.win-amd64-3.6\Release && C:\Miniconda36-x64\envs\arrow\python.exe -m cython --cplus --working C:/projects/arrow/python --output-file C:/projects/arrow/python/build/temp.win-amd64-3.6/Release/_parquet.cpp C:/projects/arrow/python/pyarrow/_parquet.pyx" [2/8] cmd.exe /c [3/8] cmd.exe /C "cd /D C:\projects\arrow\python\build\temp.win-amd64-3.6\Release && C:\Miniconda36-x64\envs\arrow\python.exe -m cython --cplus --working C:/projects/arrow/python --output-file C:/projects/arrow/python/build/temp.win-amd64-3.6/Release/lib.cpp C:/projects/arrow/python/pyarrow/lib.pyx" [4/8] cmd.exe /c [5/8] C:\PROGRA~2\MIB055~1\2017\COMMUN~1\VC\Tools\MSVC\1414~1.264\bin\Hostx64\x64\cl.exe /TP -DARROW_EXPORTING -D_CRT_SECURE_NO_WARNINGS -D_parquet_EXPORTS -IC:\Miniconda36-x64\envs\arrow\lib\site-packages\numpy\core\include -IC:\Miniconda36-x64\envs\arrow\include -I..\..\..\src -IC:\Miniconda36-x64\envs\arrow\Library\include /bigobj /W3 /wd4800 /DWIN32 /D_WINDOWS /GR /EHsc /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING /WX /wd4190 /wd4293 /wd4800 /MD /O2 /Ob2 /DNDEBUG /showIncludes /FoCMakeFiles\_parquet.dir\_parquet.cpp.obj /FdCMakeFiles\_parquet.dir\ /FS -c _parquet.cpp FAILED: CMakeFiles/_parquet.dir/_parquet.cpp.obj C:\PROGRA~2\MIB055~1\2017\COMMUN~1\VC\Tools\MSVC\1414~1.264\bin\Hostx64\x64\cl.exe /TP -DARROW_EXPORTING -D_CRT_SECURE_NO_WARNINGS -D_parquet_EXPORTS -IC:\Miniconda36-x64\envs\arrow\lib\site-packages\numpy\core\include -IC:\Miniconda36-x64\envs\arrow\include -I..\..\..\src -IC:\Miniconda36-x64\envs\arrow\Library\include /bigobj /W3 /wd4800 /DWIN32 /D_WINDOWS /GR /EHsc /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING /WX /wd4190 /wd4293 /wd4800 /MD /O2 /Ob2 /DNDEBUG /showIncludes /FoCMakeFiles\_parquet.dir\_parquet.cpp.obj /FdCMakeFiles\_parquet.dir\ /FS -c _parquet.cpp Microsoft (R) C/C++ Optimizing Compiler Version 19.14.26428.1 for x64 Copyright (C) Microsoft Corporation. All rights reserved. _parquet.cpp(6790): error C2220: warning treated as error - no 'object' file generated _parquet.cpp(6790): warning C4244: 'argument': conversion from 'int64_t' to 'long', possible loss of data [6/8] C:\PROGRA~2\MIB055~1\2017\COMMUN~1\VC\Tools\MSVC\1414~1.264\bin\Hostx64\x64\cl.exe /TP -DARROW_EXPORTING -D_CRT_SECURE_NO_WARNINGS -Dlib_EXPORTS -IC:\Miniconda36-x64\envs\arrow\lib\site-packages\numpy\core\include -IC:\Miniconda36-x64\envs\arrow\include -I..\..\..\src -IC:\Miniconda36-x64\envs\arrow\Library\include /bigobj /W3 /wd4800 /DWIN32 /D_WINDOWS /GR /EHsc /D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING /WX /wd4190 /wd4293 /wd4800 /MD /O2 /Ob2 /DNDEBUG /showIncludes /FoCMakeFiles\lib.dir\lib.cpp.obj /FdCMakeFiles\lib.dir\ /FS -c lib.cpp Microsoft (R) C/C++ Optimizing Compiler Version 19.14.26428.1 for x64 Copyright (C) Microsoft Corporation. All rights reserved. ninja: build stopped: subcommand failed. error: command 'C:\\Program Files (x86)\\CMake\\bin\\cmake.exe' failed with exit status 1 (arrow) C:\projects\arrow\python>set lastexitcode=1 {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: JDBC Adapter PR - 1759
Hi Atul, I will take a look today. Thanks, Sidd On Tue, May 29, 2018 at 2:45 AM, Atul Dambalkar wrote: > Hi Sid, Laurent, Uwe, > > Any idea when can someone take a look at the PR https://github.com/apache/ > arrow/pull/1759/. > > Laurent had given bunch of comments earlier and now we have taken care of > most of those. We have also added multiple test cases. It will be great if > someone can take a look. > > Regards, > -Atul > >
[jira] [Created] (ARROW-2645) [Java] ArrowStreamWriter accumulates DictionaryBatch ArrowBlocks
Bryan Cutler created ARROW-2645: --- Summary: [Java] ArrowStreamWriter accumulates DictionaryBatch ArrowBlocks Key: ARROW-2645 URL: https://issues.apache.org/jira/browse/ARROW-2645 Project: Apache Arrow Issue Type: Improvement Components: Java - Vectors Reporter: Bryan Cutler Assignee: Bryan Cutler While reading the base method ensureStarted in ArrowStreamWriter accumulates Dictionary blocks. This is used for ArrowFileWriter but not ArrowStreamWriter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: JDBC Adapter PR - 1759
Same here. On Tue, May 29, 2018 at 9:59 AM, Siddharth Teotia wrote: > Hi Atul, > > I will take a look today. > > Thanks, > Sidd > > On Tue, May 29, 2018 at 2:45 AM, Atul Dambalkar < > atul.dambal...@xoriant.com> > wrote: > > > Hi Sid, Laurent, Uwe, > > > > Any idea when can someone take a look at the PR > https://github.com/apache/ > > arrow/pull/1759/. > > > > Laurent had given bunch of comments earlier and now we have taken care of > > most of those. We have also added multiple test cases. It will be great > if > > someone can take a look. > > > > Regards, > > -Atul > > > > >