Re: Writing very large rowgroups to Apache Parquet

2020-07-09 Thread Micah Kornfield
+parquet-dev as this seems more concerned with the non-arrow pieces of parquet Hi Roman, Answers inline. One way to solve that problem would be to use memory mapped files instead > of plain memory buffers. That way, the number of required memory can be > limited by the number of columns times

[jira] [Commented] (PARQUET-1883) int96 support in parquet-avro

2020-07-09 Thread Xinli Shang (Jira)
[ https://issues.apache.org/jira/browse/PARQUET-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154888#comment-17154888 ] Xinli Shang commented on PARQUET-1883: -- [~gszadovszky], Do you still have links for INT96 will be

[jira] [Updated] (PARQUET-1883) int96 support in parquet-avro

2020-07-09 Thread satish (Jira)
[ https://issues.apache.org/jira/browse/PARQUET-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] satish updated PARQUET-1883: Description: Hi It looks like 'timestamp' is being converted to 'int64' primitive type in

[jira] [Updated] (PARQUET-1883) int96 support in parquet-avro

2020-07-09 Thread satish (Jira)
[ https://issues.apache.org/jira/browse/PARQUET-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] satish updated PARQUET-1883: Description: Hi It looks like 'timestamp' is being converted to 'int64' primitive type in

[jira] [Created] (PARQUET-1883) int96 support in parquet-avro

2020-07-09 Thread satish (Jira)
satish created PARQUET-1883: --- Summary: int96 support in parquet-avro Key: PARQUET-1883 URL: https://issues.apache.org/jira/browse/PARQUET-1883 Project: Parquet Issue Type: Bug

[jira] [Commented] (PARQUET-1882) Writing an all-null column and then reading it with buffered_stream aborts the process

2020-07-09 Thread Wes McKinney (Jira)
[ https://issues.apache.org/jira/browse/PARQUET-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154724#comment-17154724 ] Wes McKinney commented on PARQUET-1882: --- Can you provide a reproducible code example? > Writing

[jira] [Updated] (PARQUET-1882) Writing an all-null column and then reading it with buffered_stream aborts the process

2020-07-09 Thread Eric Gorelik (Jira)
[ https://issues.apache.org/jira/browse/PARQUET-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Gorelik updated PARQUET-1882: -- Affects Version/s: (was: cpp-1.5.0) > Writing an all-null column and then reading it

[jira] [Created] (PARQUET-1882) Writing an all-null column and then reading it with buffered_stream aborts the process

2020-07-09 Thread Eric Gorelik (Jira)
Eric Gorelik created PARQUET-1882: - Summary: Writing an all-null column and then reading it with buffered_stream aborts the process Key: PARQUET-1882 URL: https://issues.apache.org/jira/browse/PARQUET-1882