morningman commented on code in PR #19434:
URL: https://github.com/apache/doris/pull/19434#discussion_r1188080216
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/PooledHiveMetaStoreClient.java:
##########
@@ -53,7 +53,8 @@ public class PooledHiveMetaStoreClient {
private static final Logger LOG =
LogManager.getLogger(PooledHiveMetaStoreClient.class);
private static final HiveMetaHookLoader DUMMY_HOOK_LOADER = t -> null;
- private static final short MAX_LIST_PARTITION_NUM = 10000;
+ // -1 means no limit on the partitions returned.
+ private static final short MAX_LIST_PARTITION_NUM = -1;
Review Comment:
better change it to a mutable FE config.
I am afraid if there are too many partitions, it will causing FE OOM.
Set as a config, and default value can be -1, if encounter error, we can set
it manually.
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -1679,6 +1679,13 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = false)
public static long max_hive_partition_cache_num = 100000;
+ /**
+ * Max cache num hive table to partition name list.
+ * Decrease this value if FE's memory is small
+ */
+ @ConfField(mutable = false, masterOnly = false)
+ public static long max_hive_table_partition_value_num = 100;
Review Comment:
```suggestion
public static long max_hive_table_catch_num = 1000;
```
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -1679,6 +1679,13 @@ public class Config extends ConfigBase {
@ConfField(mutable = false, masterOnly = false)
public static long max_hive_partition_cache_num = 100000;
+ /**
+ * Max cache num hive table to partition name list.
+ * Decrease this value if FE's memory is small
+ */
+ @ConfField(mutable = false, masterOnly = false)
+ public static long max_hive_table_partition_value_num = 100;
Review Comment:
Please refer to this https://github.com/apache/doris/pull/19246
to add config's document
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
