tpalfy commented on a change in pull request #3679: NIFI-6089 Add Parquet
record reader and writer
URL: https://github.com/apache/nifi/pull/3679#discussion_r319890595
##########
File path:
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/utils/ParquetUtils.java
##########
@@ -96,18 +97,40 @@
.allowableValues(org.apache.parquet.column.ParquetProperties.WriterVersion.values())
.build();
- public static List<AllowableValue> COMPRESSION_TYPES =
getCompressionTypes();
+ public static final PropertyDescriptor AVRO_READ_COMPATIBILITY = new
PropertyDescriptor.Builder()
+ .name("avro-read-compatibility")
+ .displayName("Avro Read Compatibility")
+ .description("Specifies the value for 'parquet.avro.compatible' in
the underlying Parquet library")
+ .allowableValues("true", "false")
+ .defaultValue("true")
+ .required(true)
+ .build();
- public static final PropertyDescriptor COMPRESSION_TYPE = new
PropertyDescriptor.Builder()
- .name("compression-type")
- .displayName("Compression Type")
- .description("The type of compression for the file being written.")
- .allowableValues(COMPRESSION_TYPES.toArray(new AllowableValue[0]))
- .defaultValue(COMPRESSION_TYPES.get(0).getValue())
+ public static final String PARQUET_AVRO_ADD_LIST_ELEMENT_RECORDS =
"parquet.avro.add-list-element-records";
+
+ public static final PropertyDescriptor AVRO_ADD_LIST_ELEMENT_RECORDS = new
PropertyDescriptor.Builder()
+ .name("avro-add-list-element-records")
+ .displayName("Avro Add List Element Records")
+ .description("Specifies the value for
'parquet.avro.add-list-element-records' in the underlying Parquet library")
+ .allowableValues("true", "false")
+ .defaultValue("true")
+ .required(true)
+ .build();
+
+ public static final String PARQUET_AVRO_WRITE_OLD_LIST_STRUCTURE =
"parquet.avro.write-old-list-structure";
Review comment:
We are already referencing `AvroReadSupport.AVRO_COMPATIBILITY` so we might
as well reference `AvroSchemaConverter.ADD_LIST_ELEMENT_RECORDS` and
`AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE` too.
In fact I think it would be better to reference it in all the places. More
durable and maybe easier to understand the connections across the code
(property descriptor definition and config buildup):
```java
public static final PropertyDescriptor AVRO_READ_COMPATIBILITY = new
PropertyDescriptor.Builder()
.name("avro-read-compatibility")
.displayName("Avro Read Compatibility")
.description("Specifies the value for '" +
AvroReadSupport.AVRO_COMPATIBILITY + "' in the underlying Parquet library")
.allowableValues("true", "false")
.defaultValue("true")
.required(true)
.build();
public static final PropertyDescriptor AVRO_ADD_LIST_ELEMENT_RECORDS =
new PropertyDescriptor.Builder()
.name("avro-add-list-element-records")
.displayName("Avro Add List Element Records")
.description("Specifies the value for '" +
AvroSchemaConverter.ADD_LIST_ELEMENT_RECORDS + "' in the underlying Parquet
library")
.allowableValues("true", "false")
.defaultValue("true")
.required(true)
.build();
public static final PropertyDescriptor AVRO_WRITE_OLD_LIST_STRUCTURE =
new PropertyDescriptor.Builder()
.name("avro-write-old-list-structure")
.displayName("Avro Write Old List Structure")
.description("Specifies the value for '" +
AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE + "' in the underlying Parquet
library")
.allowableValues("true", "false")
.defaultValue("true")
.required(true)
.build();
...
public static void applyCommonConfig(Configuration conf, ParquetConfig
parquetConfig) {
if (parquetConfig.getAvroReadCompatibility() != null) {
conf.setBoolean(AvroReadSupport.AVRO_COMPATIBILITY,
parquetConfig.getAvroReadCompatibility().booleanValue());
}
if (parquetConfig.getAvroAddListElementRecords() != null) {
conf.setBoolean(AvroSchemaConverter.ADD_LIST_ELEMENT_RECORDS,
parquetConfig.getAvroAddListElementRecords().booleanValue());
}
if (parquetConfig.getAvroWriteOldListStructure() != null) {
conf.setBoolean(AvroWriteSupport.WRITE_OLD_LIST_STRUCTURE,
parquetConfig.getAvroWriteOldListStructure().booleanValue());
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
