[ https://issues.apache.org/jira/browse/HDFS-15248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17070251#comment-17070251 ]
Wei-Chiu Chuang edited comment on HDFS-15248 at 3/30/20, 1:56 AM: ------------------------------------------------------------------ Thanks for offering the patch! I've had customers asking for extending ACL entry limit before. I'm not sure why 32, but here are a few reasons why it's probably not a good idea to extend it further (1) manageability. once you have more than a dozen ACLs per file, it becomes hard to manage, error-prone. (2) NameNode heap size. Especially in a large cluster with hundreds of millions of files, each inode occupies more bytes of heap. The memory pressure becomes even worse. (3) serialization cost. We currently serialize the files under a directory to a protobuf message, which is limited to 64mb (default), and as the result we limit the max number of files per directory to 1 million. Allowing more ACL entries per file means more serialized bytes per file, and you may run into the protobuf message limit for a large directory well before 1 million files. For these reasons I usually recommend users to use external authorization providers like Sentry or Ranger to delegate the authorization work to a separate entity. was (Author: jojochuang): Thanks for offering the patch! I've had customers asking for extending ACL entry limit before. I'm not sure why 32, but here are a few reasons why it's probably not a good idea to extend it further (1) manageability. once you have more than a dozen ACLs per file, it becomes hard to manage, error-prone. (2) NameNode heap size. Especially in a large cluster with hundreds of millions of files, each inode occupies more bytes of heap. The memory pressure becomes even worse. (3) serialization cost. We currently serialize the files under a directory to a protobuf message, which is limited to 64mb (default), and as the result we limit the max number of files per directory to 1 million. Allowing more ACL entries per file means more serialized bytes per file, and you may run into the protobuf message limit for a large directory well before 1 million files. For these reasons I usually recommend users to use external authorization providers like Sentry or Ranger to delete the authorization work to a separate entity. > Make the maximum number of ACLs entries configurable > ---------------------------------------------------- > > Key: HDFS-15248 > URL: https://issues.apache.org/jira/browse/HDFS-15248 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Reporter: Yang Yun > Assignee: Yang Yun > Priority: Minor > Attachments: HDFS-15248.001.patch, HDFS-15248.patch > > > For big cluster, the hardcode 32 of ACLs maximum number is not enough, make > it configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org