dzamo commented on a change in pull request #2351:
URL: https://github.com/apache/drill/pull/2351#discussion_r754050973
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetFormatPlugin.java
##########
@@ -141,40 +145,94 @@ public AbstractWriter getWriter(PhysicalOperator child,
String location, List<St
}
public RecordWriter getRecordWriter(FragmentContext context, ParquetWriter
writer) throws IOException, OutOfMemoryException {
- Map<String, String> options = new HashMap<>();
+ Map<String, String> writerOpts = new HashMap<>();
+ OptionManager contextOpts = context.getOptions();
- options.put("location", writer.getLocation());
+ writerOpts.put("location", writer.getLocation());
FragmentHandle handle = context.getHandle();
String fragmentId = String.format("%d_%d", handle.getMajorFragmentId(),
handle.getMinorFragmentId());
- options.put("prefix", fragmentId);
-
- options.put(ExecConstants.PARQUET_BLOCK_SIZE,
context.getOptions().getOption(ExecConstants.PARQUET_BLOCK_SIZE).num_val.toString());
- options.put(ExecConstants.PARQUET_WRITER_USE_SINGLE_FS_BLOCK,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_USE_SINGLE_FS_BLOCK).bool_val.toString());
- options.put(ExecConstants.PARQUET_PAGE_SIZE,
context.getOptions().getOption(ExecConstants.PARQUET_PAGE_SIZE).num_val.toString());
- options.put(ExecConstants.PARQUET_DICT_PAGE_SIZE,
context.getOptions().getOption(ExecConstants.PARQUET_DICT_PAGE_SIZE).num_val.toString());
-
- options.put(ExecConstants.PARQUET_WRITER_COMPRESSION_TYPE,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_COMPRESSION_TYPE).string_val);
-
- options.put(ExecConstants.PARQUET_WRITER_ENABLE_DICTIONARY_ENCODING,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_ENABLE_DICTIONARY_ENCODING).bool_val.toString());
-
- options.put(ExecConstants.PARQUET_WRITER_LOGICAL_TYPE_FOR_DECIMALS,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_LOGICAL_TYPE_FOR_DECIMALS).string_val);
-
- options.put(ExecConstants.PARQUET_WRITER_USE_PRIMITIVE_TYPES_FOR_DECIMALS,
-
context.getOptions().getOption(ExecConstants.PARQUET_WRITER_USE_PRIMITIVE_TYPES_FOR_DECIMALS).bool_val.toString());
+ writerOpts.put("prefix", fragmentId);
+
+ // Many options which follow may be set as Drill config options or in the
parquet format
+ // plugin config. If there is a Drill option set at session scope or
narrower it takes precendence.
+ OptionValue.OptionScope minScope = OptionValue.OptionScope.SESSION;
Review comment:
@vdiravka Okay let's discuss a design for this for a new PR. I guess it
would impact all the existing format plugins, and how they access format
configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
