[ https://issues.apache.org/jira/browse/ARROW-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16076847#comment-16076847 ]
Jeff Knupp commented on ARROW-1167: ----------------------------------- Ah, OK. I misunderstood. The pandas problem is somewhat similar but that one _is_ caused by the size of the type used to calculate memory (re)allocations. Nevermind! I'll ask this question on the pandas PR. I'll also look into implementing a chunked array in {{Table.from_pandas}}. > Writing pyarrow Table to Parquet core dumps > ------------------------------------------- > > Key: ARROW-1167 > URL: https://issues.apache.org/jira/browse/ARROW-1167 > Project: Apache Arrow > Issue Type: Bug > Reporter: Jeff Knupp > > When writing a pyarrow Table (instantiated from a Pandas dataframe reading in > a ~5GB CSV file) to a parquet file, the interpreter cores with the following > stack trace from gdb: > {code} > #0 __memmove_avx_unaligned () at > ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:181 > #1 0x00007fbaa5c779f1 in parquet::InMemoryOutputStream::Write(unsigned char > const*, long) () from > /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/libparquet.so.1 > #2 0x00007fbaa5c0ce97 in > parquet::PlainEncoder<parquet::DataType<(parquet::Type::type)6> > >::Put(parquet::ByteArray const*, int) () > from > /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/libparquet.so.1 > #3 0x00007fbaa5c18855 in > parquet::TypedColumnWriter<parquet::DataType<(parquet::Type::type)6> > >::WriteMiniBatch(long, short const*, short const*, parquet::ByteArray > const*) () > from > /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/libparquet.so.1 > #4 0x00007fbaa5c189d5 in > parquet::TypedColumnWriter<parquet::DataType<(parquet::Type::type)6> > >::WriteBatch(long, short const*, short const*, parquet::ByteArray const*) () > from > /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/libparquet.so.1 > #5 0x00007fbaa5be0900 in arrow::Status > parquet::arrow::FileWriter::Impl::TypedWriteBatch<parquet::DataType<(parquet::Type::type)6>, > arrow::BinaryType>(parquet::ColumnWriter*, std::shared_ptr<arrow::Array> > const&, long, short const*, short const*) () from > /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/libparquet.so.1 > #6 0x00007fbaa5be171d in > parquet::arrow::FileWriter::Impl::WriteColumnChunk(arrow::Array const&) () > from /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/libparquet.so.1 > #7 0x00007fbaa5be1dad in > parquet::arrow::FileWriter::WriteColumnChunk(arrow::Array const&) () from > /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/libparquet.so.1 > #8 0x00007fbaa5be2047 in parquet::arrow::FileWriter::WriteTable(arrow::Table > const&, long) () from > /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/libparquet.so.1 > #9 0x00007fbaa51e1f53 in > __pyx_pw_7pyarrow_8_parquet_13ParquetWriter_5write_table(_object*, _object*, > _object*) () > from > /home/ubuntu/.local/lib/python3.5/site-packages/pyarrow/_parquet.cpython-35m-x86_64-linux-gnu.so > #10 0x00000000004e9bc7 in PyCFunction_Call () at ../Objects/methodobject.c:98 > #11 0x0000000000529885 in do_call (nk=<optimized out>, na=<optimized out>, > pp_stack=0x7ffe6510a6c0, func=<optimized out>) at ../Python/ceval.c:4933 > #12 call_function (oparg=<optimized out>, pp_stack=0x7ffe6510a6c0) at > ../Python/ceval.c:4732 > #13 PyEval_EvalFrameEx () at ../Python/ceval.c:3236 > #14 0x000000000052d2e3 in _PyEval_EvalCodeWithName () at > ../Python/ceval.c:4018 > #15 0x0000000000528eee in fast_function (nk=<optimized out>, na=<optimized > out>, n=<optimized out>, pp_stack=0x7ffe6510a8d0, func=<optimized out>) at > ../Python/ceval.c:4813 > #16 call_function (oparg=<optimized out>, pp_stack=0x7ffe6510a8d0) at > ../Python/ceval.c:4730 > #17 PyEval_EvalFrameEx () at ../Python/ceval.c:3236 > #18 0x000000000052d2e3 in _PyEval_EvalCodeWithName () at > ../Python/ceval.c:4018 > #19 0x0000000000528eee in fast_function (nk=<optimized out>, na=<optimized > out>, n=<optimized out>, pp_stack=0x7ffe6510aae0, func=<optimized out>) at > ../Python/ceval.c:4813 > #20 call_function (oparg=<optimized out>, pp_stack=0x7ffe6510aae0) at > ../Python/ceval.c:4730 > #21 PyEval_EvalFrameEx () at ../Python/ceval.c:3236 > #22 0x0000000000528814 in fast_function (nk=<optimized out>, na=<optimized > out>, n=<optimized out>, pp_stack=0x7ffe6510ac10, func=<optimized out>) at > ../Python/ceval.c:4803 > #23 call_function (oparg=<optimized out>, pp_stack=0x7ffe6510ac10) at > ../Python/ceval.c:4730 > #24 PyEval_EvalFrameEx () at ../Python/ceval.c:3236 > #25 0x0000000000528814 in fast_function (nk=<optimized out>, na=<optimized > out>, n=<optimized out>, pp_stack=0x7ffe6510ad40, func=<optimized out>) at > ../Python/ceval.c:4803 > #26 call_function (oparg=<optimized out>, pp_stack=0x7ffe6510ad40) at > ../Python/ceval.c:4730 > #27 PyEval_EvalFrameEx () at ../Python/ceval.c:3236 > #28 0x000000000052d2e3 in _PyEval_EvalCodeWithName () at > ../Python/ceval.c:4018 > #29 0x000000000052dfdf in PyEval_EvalCodeEx () at ../Python/ceval.c:4039 > #30 PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, > locals=<optimized out>) at ../Python/ceval.c:777 > #31 0x00000000005fd2c2 in run_mod () at ../Python/pythonrun.c:976 > #32 0x00000000005ff76a in PyRun_FileExFlags () at ../Python/pythonrun.c:929 > #33 0x00000000005ff95c in PyRun_SimpleFileExFlags () at > ../Python/pythonrun.c:396 > #34 0x000000000063e7d6 in run_file (p_cf=0x7ffe6510afb0, filename=0x2161260 > L"scripts/parquet_export.py", fp=0x226fde0) at ../Modules/main.c:318 > #35 Py_Main () at ../Modules/main.c:768 > #36 0x00000000004cfe41 in main () at ../Programs/python.c:65 > #37 0x00007fbadf0db830 in __libc_start_main (main=0x4cfd60 <main>, argc=2, > argv=0x7ffe6510b1c8, init=<optimized out>, fini=<optimized out>, > rtld_fini=<optimized out>, stack_end=0x7ffe6510b1b8) > at ../csu/libc-start.c:291 > #38 0x00000000005d5f29 in _start () > {code} > This is occurring in a pretty vanilla call to `pq.write_table(table, > output)`. Before the crash, I'm able to print out the table's schema and it > looks a little odd (all columns are explicitly specified in > {{pandas.read_csv()}} to be strings... > {code} > _id: string > ref_id: string > ref_no: string > stage: string > stage2_ref_id: string > org_id: string > classification: string > solicitation_no: string > notice_type: string > business_category: string > procurement_mode: string > funding_instrument: string > funding_source: string > approved_budget: string > publish_date: string > closing_date: string > contract_duration: string > calendar_type: string > trade_agreement: string > pre_bid_date: string > pre_bid_venue: string > procuring_entity_org_id: string > procuring_entity_org: string > client_agency_org_id: string > client_agency_org: string > contact_person: string > contact_person_address: string > tender_title: string > description: string > other_info: string > reason: string > created_by: string > creation_date: string > modified_date: string > special_instruction: string > collection_contact: string > tender_status: string > collection_point: string > date_available: string > serialid: string > __index_level_0__: int64 > -- metadata -- > pandas: {"index_columns": ["__index_level_0__"], "columns": [{"pandas_type": > "unicode", "numpy_type": "object", "metadata": null, "name": "_id"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "ref_id"}, {"pandas_type": "unicode", "numpy_type": "object", "metadata": > null, "name": "ref_no"}, {"pandas_type": "unicode", "numpy_type": "object", > "metadata": null, "name": "stage"}, {"pandas_type": "mixed", "numpy_type": > "object", "metadata": null, "name": "stage2_ref_id"}, {"pandas_type": > "unicode", "numpy_type": "object", "metadata": null, "name": "org_id"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "classification"}, {"pandas_type": "mixed", "numpy_type": "object", > "metadata": null, "name": "solicitation_no"}, {"pandas_type": "unicode", > "numpy_type": "object", "metadata": null, "name": "notice_type"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "business_category"}, {"pandas_type": "unicode", "numpy_type": "object", > "metadata": null, "name": "procurement_mode"}, {"pandas_type": "mixed", > "numpy_type": "object", "metadata": null, "name": "funding_instrument"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "funding_source"}, {"pandas_type": "unicode", "numpy_type": "object", > "metadata": null, "name": "approved_budget"}, {"pandas_type": "mixed", > "numpy_type": "object", "metadata": null, "name": "publish_date"}, > {"pandas_type": "mixed", "numpy_type": "object", "metadata": null, "name": > "closing_date"}, {"pandas_type": "unicode", "numpy_type": "object", > "metadata": null, "name": "contract_duration"}, {"pandas_type": "mixed", > "numpy_type": "object", "metadata": null, "name": "calendar_type"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "trade_agreement"}, {"pandas_type": "mixed", "numpy_type": "object", > "metadata": null, "name": "pre_bid_date"}, {"pandas_type": "mixed", > "numpy_type": "object", "metadata": null, "name": "pre_bid_venue"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "procuring_entity_org_id"}, {"pandas_type": "unicode", "numpy_type": > "object", "metadata": null, "name": "procuring_entity_org"}, {"pandas_type": > "unicode", "numpy_type": "object", "metadata": null, "name": > "client_agency_org_id"}, {"pandas_type": "mixed", "numpy_type": "object", > "metadata": null, "name": "client_agency_org"}, {"pandas_type": "mixed", > "numpy_type": "object", "metadata": null, "name": "contact_person"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "contact_person_address"}, {"pandas_type": "mixed", "numpy_type": "object", > "metadata": null, "name": "tender_title"}, {"pandas_type": "mixed", > "numpy_type": "object", "metadata": null, "name": "description"}, > {"pandas_type": "mixed", "numpy_type": "object", "metadata": null, "name": > "other_info"}, {"pandas_type": "mixed", "numpy_type": "object", "metadata": > null, "name": "reason"}, {"pandas_type": "unicode", "numpy_type": "object", > "metadata": null, "name": "created_by"}, {"pandas_type": "unicode", > "numpy_type": "object", "metadata": null, "name": "creation_date"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "modified_date"}, {"pandas_type": "mixed", "numpy_type": "object", > "metadata": null, "name": "special_instruction"}, {"pandas_type": "mixed", > "numpy_type": "object", "metadata": null, "name": "collection_contact"}, > {"pandas_type": "mixed", "numpy_type": "object", "metadata": null, "name": > "tender_status"}, {"pandas_type": "mixed", "numpy_type": "object", > "metadata": null, "name": "collection_point"}, {"pandas_type": "mixed", > "numpy_type": "object", "metadata": null, "name": "date_available"}, > {"pandas_type": "unicode", "numpy_type": "object", "metadata": null, "name": > "serialid"}, {"pandas_type": "int64", "numpy_type": "int64", "metadata": > null, "name": "__index_level_0__"}], "pandas_version": "0.19.2"} > Segmentation fault (core dumped) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)