wankunde commented on code in PR #2371:
URL: https://github.com/apache/orc/pull/2371#discussion_r2324577133
##########
java/core/src/java/org/apache/orc/OrcConf.java:
##########
@@ -118,6 +118,11 @@ public enum OrcConf {
"If the number of distinct keys in a dictionary is greater than this\n" +
"fraction of the total number of non-null rows, turn off \n" +
"dictionary encoding. Use 1 to always use dictionary encoding."),
+ DICTIONARY_MAX_SIZE_IN_BYTES("orc.dictionary.maxSizeInBytes",
+ "hive.exec.orc.dictionary.maxSizeInBytes",
+ 16 * 1024 * 1024,
Review Comment:
```log
dictionary.size = 5120, row size = 5120, dictionary SizeInBytes = 360448,
rows SizeInBytes = 32768
dictionary.size = 5120, row size = 5120, dictionary SizeInBytes =
1452277760, rows SizeInBytes = 32768
dictionary.size = 1, row size = 5120, dictionary SizeInBytes = 81920, rows
SizeInBytes = 32768
dictionary.size = 661, row size = 5012, dictionary SizeInBytes = 81920, rows
SizeInBytes = 32768
dictionary.size = 5120, row size = 5120, dictionary SizeInBytes = 10223616,
rows SizeInBytes = 32768
dictionary.size = 1, row size = 5120, dictionary SizeInBytes = 81920, rows
SizeInBytes = 32768
dictionary.size = 682, row size = 5120, dictionary SizeInBytes = 147456,
rows SizeInBytes = 32768
dictionary.size = 1, row size = 5120, dictionary SizeInBytes = 81920, rows
SizeInBytes = 32768
dictionary.size = 2, row size = 5120, dictionary SizeInBytes = 81920, rows
SizeInBytes = 32768
dictionary.size = 1, row size = 5120, dictionary SizeInBytes = 81920, rows
SizeInBytes = 32768
```
The dictionary size in bytes usually less than 10MB.
If dictionary size (>16MB) / stripe size(< 128MB) > 12.5%, I think it large
enough to disable the dictionary encoding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
