[
https://issues.apache.org/jira/browse/PARQUET-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163339#comment-17163339
]
Øyvind Strømmen commented on PARQUET-1887:
------------------------------------------
Thanks for picking this up so quickly! I tested the fix, and it works for that
test case, but I was able to create another test case that provokes the same
error, only in a slightly different manner. Please see sample code below:
{code:java}
final Schema schema;
try (InputStream avroSchema =
Resources.getResource("persons.json").openStream()) {
schema = new Schema.Parser().parse(avroSchema);
}
Configuration conf = new Configuration();
conf.setBoolean(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE, false);
try (ParquetWriter<GenericRecord> writer =
AvroParquetWriter.<GenericRecord>builder(new
org.apache.hadoop.fs.Path("/tmp/persons.parquet"))
.withSchema(schema)
.withConf(conf)
.build()) {
// To trigger exception, add array with null element.
try {
writer.write(new GenericRecordBuilder(schema).set("address",
Arrays.asList("first", null, "last")).build());
} catch (RuntimeException e) {
e.printStackTrace() // "java.lang.RuntimeException: Null list element for
array"
}
// At this point all future calls to writer.write will fail
try {
writer.write(new GenericRecordBuilder(schema).set("address",
Arrays.asList("foo", "bar")).build());
} catch (Exception e) {
e.printStackTrace() // "java.lang.ArrayIndexOutOfBoundsException: Index 3
out of bounds for length 3"
}
} {code}
Seems like {{currentLevel}} in {{MessageColumnIORecordConsumer}} isn't being
reset properly.
> Exception thrown by AvroParquetWriter#write causes all subsequent calls to it
> to fail
> -------------------------------------------------------------------------------------
>
> Key: PARQUET-1887
> URL: https://issues.apache.org/jira/browse/PARQUET-1887
> Project: Parquet
> Issue Type: Bug
> Components: parquet-avro
> Affects Versions: 1.11.0, 1.8.3
> Reporter: Øyvind Strømmen
> Priority: Major
> Attachments: person1_11_0.parquet, person1_8_3.parquet
>
>
> Please see sample code below:
> {code:java}
> Schema schema = new Schema.Parser().parse("""
> {
> "type": "record",
> "name": "person",
> "fields": [
> {
> "name": "address",
> "type": [
> "null",
> {
> "type": "array",
> "items": "string"
> }
> ],
> "default": null
> }
> ]
> }
> """
> );
> ParquetWriter<GenericRecord> writer =
> AvroParquetWriter.<GenericRecord>builder(new
> org.apache.hadoop.fs.Path("/tmp/person.parquet"))
> .withSchema(schema)
> .build();
> try {
> // To trigger exception, add array with null element.
> writer.write(new GenericRecordBuilder(schema).set("address",
> Arrays.asList("first", null, "last")).build());
> } catch (Exception e) {
> e.printStackTrace(); // "java.lang.NullPointerException: Array contains a
> null element at 1"
> }
> try {
> // At this point all future calls to writer.write will fail
> writer.write(new GenericRecordBuilder(schema).set("address",
> Arrays.asList("foo", "bar")).build());
> } catch (Exception e) {
> e.printStackTrace(); // "org.apache.parquet.io.InvalidRecordException:
> 1(r) > 0 ( schema r)"
> }
> writer.close();
> {code}
> It seems to me this is caused by state not being reset between writes. Is
> this the indented behavior of the writer? And if so, does one have to create
> a new writer whenever a write fails?
> I'm able to reproduce this using both parquet 1.8.3 and 1.11.0, and have
> attached a sample parquet file for each version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
