Sorry to jump in like this, but I was wondering if parquet-rs can read the file correctly, or the issue also happens there. Alex, could you give it a go and see if file and metadata can be read with parquet-rs (https://github.com/sunchao/parquet-rs, you can run cargo install parquet to install parquet tools).
Cheers, Ivan On Mon, 30 Jul 2018 at 21:49 ALeX Wang <[email protected]> wrote: > Thanks for the quick reply @Wes, > > Too bad this is causing a lot of delays (due to page fault handing) for > light queries (ones that query only few rows/columns), > > Will try to use jemallc and see,,, > > One more question, when i upgrade to 1.4.0 or later code, and use the same > cmake options, and environment, OpenFile result in segfault,,, > > ``` > awake@ev003:/tmp$ cat tmpfile > (gdb) where > #0 0x00007fc542eebc3c in free () from /lib64/libc.so.6 > #1 0x0000000000f13cb1 in arrow::DefaultMemoryPool::Free (this=0x16e71e0 > <arrow::default_memory_pool()::default_memory_pool_>, buffer=0x7fc52f425040 > <Address 0x7fc52f425040 out of bounds>, size=616512) > at > > /opt/parquet-cpp/arrow_ep-prefix/src/arrow_ep/cpp/src/arrow/memory_pool.cc:147 > #2 0x0000000000f117b6 in arrow::PoolBuffer::~PoolBuffer (this=0x34b5fb8, > __in_chrg=<optimized out>) at > /opt/parquet-cpp/arrow_ep-prefix/src/arrow_ep/cpp/src/arrow/buffer.cc:70 > #3 0x0000000000e364b7 in > __gnu_cxx::new_allocator<arrow::PoolBuffer>::destroy<arrow::PoolBuffer> > (this=0x34b5fb0, __p=0x34b5fb8) at > /usr/include/c++/4.8.2/ext/new_allocator.h:124 > #4 0x0000000000e35e10 in > std::allocator_traits<std::allocator<arrow::PoolBuffer> > >::_S_destroy<arrow::PoolBuffer> (__a=..., __p=0x34b5fb8) at > /usr/include/c++/4.8.2/bits/alloc_traits.h:281 > #5 0x0000000000e34ea3 in > std::allocator_traits<std::allocator<arrow::PoolBuffer> > >::destroy<arrow::PoolBuffer> (__a=..., __p=0x34b5fb8) at > /usr/include/c++/4.8.2/bits/alloc_traits.h:405 > #6 0x0000000000e33f01 in std::_Sp_counted_ptr_inplace<arrow::PoolBuffer, > std::allocator<arrow::PoolBuffer>, (__gnu_cxx::_Lock_policy)2>::_M_dispose > (this=0x34b5fa0) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:407 > #7 0x0000000000e27748 in > std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release > (this=0x34b5fa0) at /usr/include/c++/4.8.2/bits/shared_ptr_base.h:144 > #8 0x0000000000e255bb in > std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count > (this=0x7ffea5fffc88, __in_chrg=<optimized out>) at > /usr/include/c++/4.8.2/bits/shared_ptr_base.h:546 > #9 0x0000000000e23eae in std::__shared_ptr<arrow::PoolBuffer, > (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7ffea5fffc80, > __in_chrg=<optimized out>) at > /usr/include/c++/4.8.2/bits/shared_ptr_base.h:781 > #10 0x0000000000e23ec8 in std::shared_ptr<arrow::PoolBuffer>::~shared_ptr > (this=0x7ffea5fffc80, __in_chrg=<optimized out>) at > /usr/include/c++/4.8.2/bits/shared_ptr.h:93 > #11 0x0000000000e875a4 in parquet::SerializedFile::ParseMetaData > (this=0x34b5f60) at /opt/parquet-cpp/src/parquet/file_reader.cc:213 > #12 0x0000000000e858d4 in parquet::ParquetFileReader::Contents::Open > (source=std::unique_ptr<parquet::RandomAccessSource> containing 0x0, > props=..., metadata=std::shared_ptr (empty) 0x0) at > /opt/parquet-cpp/src/parquet/file_reader.cc:247 > ---Type <return> to continue, or q <return> to quit--- > #13 0x0000000000e85a6f in parquet::ParquetFileReader::Open > (source=std::unique_ptr<parquet::RandomAccessSource> containing 0x0, > props=..., metadata=std::shared_ptr (empty) 0x0) at > /opt/parquet-cpp/src/parquet/file_reader.cc:265 > #14 0x0000000000e859ba in parquet::ParquetFileReader::Open > (source=std::shared_ptr (count 2, weak 0) 0x34b5e50, props=..., > metadata=std::shared_ptr (empty) 0x0) at > /opt/parquet-cpp/src/parquet/file_reader.cc:259 > #15 0x0000000000e85df4 in parquet::ParquetFileReader::OpenFile > > (path="/data-slow/data0/test_parquet_file/seg0/0_1530129731023-1530136801030", > memory_map=false, props=..., metadata=std::shared_ptr (empty) 0x0) at > /opt/parquet-cpp/src/parquet/file_reader.cc:287 > ``` > > Is this a known issue? > > Thanks, > Alex Wang, > > > > On Mon, Jul 30, 2018, 11:22 AM Wes McKinney <[email protected]> wrote: > > > hi Alex, > > > > It looks like the mallocs are coming from Thrift > > (parquet/parquet_types.cpp is generated by Thrift). I'm not sure if we > > can do much about this. I'm curious if it's possible to pass a custom > > STL allocator to Thrift so we could use a different allocation > > strategy than the default STL allocator > > > > - Wes > > > > On Mon, Jul 30, 2018 at 1:54 PM, ALeX Wang <[email protected]> wrote: > > > Hi, > > > > > > I'm reading parquet file (generated by Java parquet library). Our > schema > > > has 400 columns (including non-array elements, 1-dimensional array > > > elements). > > > > > > I'm using release 1.3.1, gcc 4.8.5, boost static library 1.53, > > > > > > I compile parquet-cpp with following cmake options, > > > ``` > > > cmake3 -DCMAKE_BUILD_TYPE=Debug -DPARQUET_BUILD_EXAMPLES=OFF > > > -DPARQUET_BUILD_TESTS=OFF -DPARQUET_ARROW_LINKAGE="static" > > > -DPARQUET_BUILD_SHARED=OFF -DPARQUET_BOOST_USE_SHARED=OFF . > > > ``` > > > > > > One thing we noticed is that the cpp library conducts a lot of small > > > mallocs during the open file and the reading metadata phases... shown > > > below: > > > > > > ``` > > > (gdb) where > > > #0 0x00007fdf40594801 in malloc () from /lib64/libc.so.6 > > > #1 0x00007fdf40e52ecd in operator new(unsigned long) () from > > > /lib64/libstdc++.so.6 > > > #2 0x0000000000ea16c0 in > __gnu_cxx::new_allocator<std::string>::allocate > > > (this=0x33e6930, __n=3) at > /usr/include/c++/4.8.2/ext/new_allocator.h:104 > > > #3 0x0000000000e9eabb in std::_Vector_base<std::string, > > > std::allocator<std::string> >::_M_allocate (this=0x33e6930, __n=3) at > > > /usr/include/c++/4.8.2/bits/stl_vector.h:168 > > > #4 0x0000000000ecf512 in std::vector<std::string, > > > std::allocator<std::string> >::_M_default_append (this=0x33e6930, > __n=3) > > at > > > /usr/include/c++/4.8.2/bits/vector.tcc:549 > > > #5 0x0000000000eca887 in std::vector<std::string, > > > std::allocator<std::string> >::resize (this=0x33e6930, __new_size=3) at > > > /usr/include/c++/4.8.2/bits/stl_vector.h:667 > > > #6 0x0000000000ebd589 in parquet::format::ColumnMetaData::read > > > (this=0x33e6908, iprot=0x3337300) at > > > /opt/parquet-cpp/src/parquet/parquet_types.cpp:3845 > > > #7 0x0000000000ebf9ed in parquet::format::ColumnChunk::read > > > (this=0x33e68f0, iprot=0x3337300) at > > > /opt/parquet-cpp/src/parquet/parquet_types.cpp:4246 > > > #8 0x0000000000ec0cd2 in parquet::format::RowGroup::read > > (this=0x33cf7c0, > > > iprot=0x3337300) at /opt/parquet-cpp/src/parquet/parquet_types.cpp:4451 > > > #9 0x0000000000ec4e22 in parquet::format::FileMetaData::read > > > (this=0x3337270, iprot=0x3337300) at > > > /opt/parquet-cpp/src/parquet/parquet_types.cpp:5385 > > > #10 0x0000000000e9364d in > > > parquet::DeserializeThriftMsg<parquet::format::FileMetaData> > > > (buf=0x7fdf2cace040 "\025\002\031\374\313\004H\bsessions\025\374\005", > > > len=0x7ffc8c96ff34, deserialized_msg=0x3337270) at > > > /opt/parquet-cpp/src/parquet/thrift.h:119 > > > #11 0x0000000000e8fda5 in > > > parquet::FileMetaData::FileMetaDataImpl::FileMetaDataImpl > > (this=0x3302fb0, > > > metadata=0x7fdf2cace040 > > "\025\002\031\374\313\004H\bsessions\025\374\005", > > > metadata_len=0x7ffc8c96ff34) at > > /opt/parquet-cpp/src/parquet/metadata.cc:303 > > > #12 0x0000000000e8bf4f in parquet::FileMetaData::FileMetaData > > > (this=0x31a4ca0, metadata=0x7fdf2cace040 > > > "\025\002\031\374\313\004H\bsessions\025\374\005", > > > metadata_len=0x7ffc8c96ff34) at > > /opt/parquet-cpp/src/parquet/metadata.cc:403 > > > #13 0x0000000000e8bee3 in parquet::FileMetaData::Make > > > (metadata=0x7fdf2cace040 > > "\025\002\031\374\313\004H\bsessions\025\374\005", > > > metadata_len=0x7ffc8c96ff34) at > > /opt/parquet-cpp/src/parquet/metadata.cc:398 > > > #14 0x0000000000e87572 in parquet::SerializedFile::ParseMetaData > > > (this=0x3241450) at /opt/parquet-cpp/src/parquet/file_reader.cc:213 > > > #15 0x0000000000e858d4 in parquet::ParquetFileReader::Contents::Open > > > (source=std::unique_ptr<parquet::RandomAccessSource> containing 0x0, > > > props=..., metadata=std::shared_ptr (empty) 0x0) at > > > /opt/parquet-cpp/src/parquet/file_reader.cc:247 > > > #16 0x0000000000e85a6f in parquet::ParquetFileReader::Open > > > (source=std::unique_ptr<parquet::RandomAccessSource> containing 0x0, > > > props=..., metadata=std::shared_ptr (empty) 0x0) at > > > /opt/parquet-cpp/src/parquet/file_reader.cc:265 > > > #17 0x0000000000e859ba in parquet::ParquetFileReader::Open > > > (source=std::shared_ptr (count 2, weak 0) 0x32e2e80, props=..., > > > metadata=std::shared_ptr (empty) 0x0) at > > > /opt/parquet-cpp/src/parquet/file_reader.cc:259 > > > #18 0x0000000000e85df4 in parquet::ParquetFileReader::OpenFile > > > > > > (path="/data-slow/data0/test_parquet_file/seg0/0_1530129731023-1530136801030", > > > memory_map=false, props=..., metadata=std::shared_ptr (empty) 0x0) at > > > /opt/parquet-cpp/src/parquet/file_reader.cc:287 > > > > > > (gdb) info br > > > Num Type Disp Enb Address What > > > 1 breakpoint keep y <MULTIPLE> > > > breakpoint already hit 2679 times > > > ignore next 2321 hits > > > ``` > > > > > > I set the breakpoint to `malloc`, above ^ > > > > > > This seems to be the case regardless of mmap option. > > > > > > Would really appreciate some pointer on how to avoid this. > > > > > > Thanks, > > > Alex Wang, > > > > > > -- > > > Alex Wang, > > > Open vSwitch developer > > >
