[jira] [Updated] (ARROW-11678) Broken round-trip with ParquetWriter.write_table -> read_table -> ParquetWriter.write_table

2021-02-17 Thread ARF (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ARF updated ARROW-11678:

Description: 
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 

{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 })

with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 
{color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
 {color:#00}}){color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}table{color}{color:#00}){color}

{color:#001080}existing_table{color}{color:#00} = 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#795e26}read_table{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00}){color}

{color:#af00db}with{color}{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}existing_table{color}{color:#00}){color}

 

 

*Error Message:*

ValueError: Table schema does not match schema used to create file:
table:
code: int64
  -- field metadata --
  PARQUET:field_id: '1' vs.
file:
code: uint32

  was:
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 

{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 })

with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 
{color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
 {color:#00}}){color}
 {color:#00}

[jira] [Updated] (ARROW-11678) Broken round-trip with ParquetWriter.write_table -> read_table -> ParquetWriter.write_table

2021-02-17 Thread ARF (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ARF updated ARROW-11678:

Description: 
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 

{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 })

with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 
{color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
 {color:#00}}){color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}table{color}{color:#00}){color}

{color:#001080}existing_table{color}{color:#00} = 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#795e26}read_table{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00}){color}

{color:#af00db}with{color}{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}existing_table{color}{color:#00}){color}

 

 

*Error Message:*

{{ValueError: Table schema does not match schema used to create file:
table:
code: int64
  -- field metadata --
  PARQUET:field_id: '1' vs.
file:
code: uint32}}

  was:
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 

{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 })

with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 
{color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
 {color:#00}}){color}
 {color:#00} 

[jira] [Updated] (ARROW-11678) Broken round-trip with ParquetWriter.write_table -> read_table -> ParquetWriter.write_table

2021-02-17 Thread ARF (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ARF updated ARROW-11678:

Description: 
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 

{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 })

with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 
{color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
 {color:#00}}){color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}table{color}{color:#00}){color}

{color:#001080}existing_table{color}{color:#00} = 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#795e26}read_table{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00}){color}

{color:#af00db}with{color}{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}existing_table{color}{color:#00}){color}

 

 

*Error Message:*

{{
ValueError: Table schema does not match schema used to create file:
table:
code: int64
  -- field metadata --
  PARQUET:field_id: '1' vs.
file:
code: uint32
}}

  was:
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 

{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 })

with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 
{color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
 {color:#00}}){color}

[jira] [Updated] (ARROW-11678) Broken round-trip with ParquetWriter.write_table -> read_table -> ParquetWriter.write_table

2021-02-17 Thread ARF (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ARF updated ARROW-11678:

Description: 
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 

{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 })

with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 
{color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
 {color:#00}}){color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}table{color}{color:#00}){color}

{color:#001080}existing_table{color}{color:#00} = 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#795e26}read_table{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00}){color}

{color:#af00db}with{color}{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}existing_table{color}{color:#00}){color}

 

 

*Error Message:*

{{ValueError: Table schema does not match schema used to create file:}}
 {{table:}}
 {{code: int64}}
 \{{ – field metadata --}}
 \{{ PARQUET:field_id: '1' vs.}}
 {{file:}}
 {{code: uint32}}

  was:
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 


 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 \{color:#00}})
 
 with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 

[jira] [Updated] (ARROW-11678) Broken round-trip with ParquetWriter.write_table -> read_table -> ParquetWriter.write_table

2021-02-17 Thread ARF (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ARF updated ARROW-11678:

Description: 
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:

 


 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
 {color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
 \{color:#00}})
 
 with{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
 {color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
 {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 
{color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
 {color:#00}}){color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}table{color}{color:#00}){color}

{color:#001080}existing_table{color}{color:#00} = 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#795e26}read_table{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00}){color}

{color:#af00db}with{color}{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
 {color:#00}
{color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}existing_table{color}{color:#00}){color}

 

 

*Error Message:*

{{ValueError: Table schema does not match schema used to create file:}}
 {{table:}}
 {{code: int64}}
 \{{ – field metadata --}}
 \{{ PARQUET:field_id: '1' vs.}}
 {{file:}}
 {{code: uint32}}

  was:
Round-tripping with ParquetWriter.write_table() -> pyarrow.parquet.read_table() 
-> ParquetWriter.write_table() is broken:
{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00} 
{color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
{color:#af00db}import{color}{color:#00} 
{color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
 {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pq{color}

{color:#001080}schema{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.schema({{color}
{color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
{color:#00}}){color}

{color:#af00db}with{color}{color:#00} 
{color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
 {color}{color:#001080}schema{color}{color:#00}) 
{color}{color:#af00db}as{color}{color:#00} 
{color}{color:#001080}pqwriter{color}{color:#00}:{color}
{color:#00}{color}{color:#001080}code{color}{color:#00} = 
{color}{color:#098658}111000{color}
{color:#00}{color}{color:#001080}table{color}{color:#00} = 
{color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
{color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
{color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
 

[jira] [Updated] (ARROW-11678) Broken round-trip with ParquetWriter.write_table -> read_table -> ParquetWriter.write_table

2021-02-17 Thread ARF (Jira) 


 [ 
https://issues.apache.org/jira/browse/ARROW-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ARF updated ARROW-11678:

Issue Type: Bug  (was: Improvement)

> Broken round-trip with ParquetWriter.write_table -> read_table -> 
> ParquetWriter.write_table
> ---
>
> Key: ARROW-11678
> URL: https://issues.apache.org/jira/browse/ARROW-11678
> Project: Apache Arrow
>  Issue Type: Bug
>  Components: Python
>Affects Versions: 3.0.0
>Reporter: ARF
>Priority: Critical
>
> Round-tripping with ParquetWriter.write_table() -> 
> pyarrow.parquet.read_table() -> ParquetWriter.write_table() is broken:
> {color:#af00db}import{color}{color:#00} 
> {color}{color:#267f99}pyarrow{color}{color:#00} 
> {color}{color:#af00db}as{color}{color:#00} {color}{color:#267f99}pa{color}
> {color:#af00db}import{color}{color:#00} 
> {color}{color:#267f99}pyarrow{color}{color:#00}.{color}{color:#267f99}parquet{color}{color:#00}
>  {color}{color:#af00db}as{color}{color:#00} 
> {color}{color:#267f99}pq{color}
> {color:#001080}schema{color}{color:#00} = 
> {color}{color:#267f99}pa{color}{color:#00}.schema({{color}
> {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
> {color}{color:#267f99}pa{color}{color:#00}.uint32(),{color}
> {color:#00}}){color}
> {color:#af00db}with{color}{color:#00} 
> {color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
>  {color}{color:#001080}schema{color}{color:#00}) 
> {color}{color:#af00db}as{color}{color:#00} 
> {color}{color:#001080}pqwriter{color}{color:#00}:{color}
> {color:#00}{color}{color:#001080}code{color}{color:#00} = 
> {color}{color:#098658}111000{color}
> {color:#00}{color}{color:#001080}table{color}{color:#00} = 
> {color}{color:#267f99}pa{color}{color:#00}.Table.from_pydict({{color}
> {color:#00}{color}{color:#a31515}'code'{color}{color:#00}: 
> {color}{color:#267f99}pa{color}{color:#00}.nulls({color}{color:#098658}10{color}{color:#00},
>  
> {color}{color:#001080}schema{color}{color:#00}.field({color}{color:#a31515}'code'{color}{color:#00}).type).fill_null({color}{color:#001080}code{color}{color:#00}),{color}
> {color:#00}}){color}
> {color:#00}
> {color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}table{color}{color:#00}){color}
> {color:#001080}existing_table{color}{color:#00} = 
> {color}{color:#267f99}pq{color}{color:#00}.{color}{color:#795e26}read_table{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00}){color}
> {color:#af00db}with{color}{color:#00} 
> {color}{color:#267f99}pq{color}{color:#00}.{color}{color:#267f99}ParquetWriter{color}{color:#00}({color}{color:#a31515}'test_metadata.parquet'{color}{color:#00},
>  {color}{color:#001080}schema{color}{color:#00}) 
> {color}{color:#af00db}as{color}{color:#00} 
> {color}{color:#001080}pqwriter{color}{color:#00}:{color}
> {color:#00}
> {color}{color:#001080}pqwriter{color}{color:#00}.{color}{color:#795e26}write_table{color}{color:#00}({color}{color:#001080}existing_table{color}{color:#00}){color}
> 
> *Error Message:*
> {{ValueError: Table schema does not match schema used to create file:}}
> {{table:}}
> {{code: int64}}
> {{ -- field metadata --}}
> {{ PARQUET:field_id: '1' vs.}}
> {{file:}}
> {{code: uint32}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)