----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51578/ -----------------------------------------------------------
Review request for Ambari, Di Li and DIPAYAN BHOWMICK. Bugs: AMBARI-18071 https://issues.apache.org/jira/browse/AMBARI-18071 Repository: ambari Description ------- When HDFS is configured with Encryption Zones, Files View to browse files will give "No KeyProvider" error. Steps to reproduce this issue: 1. Configure an encrypted zone in HDFS (Transparent Data Encryption). I have used Hadoop's KMS (installed tar manually). 2. Create a Files View instance and provide a user/group with the privilege to use the instance. 3. Log into the Ambari console as the user with the Files View permission. 4. Open the Files View instance. 5. Go to the folder which is configured as an encrypted zone. 6. Try to open an existing file in this folder. 7. This throws an error - java.io.IOException: No KeyProvider is configured, cannot access an encrypted file. 8. When trying through the shell, opening this file works. This happens because Files View doesn't have enough configuration set to browse secured zone. Files view doesn't even provide an option to add this configuration. This is why we see errors "No KeyProvider is configured, cannot access an encrypted file". To work around this, you could download client configuration from HDFS service tab, and copy the core-site.xml and hdfs-site.xml files to /etc/ambari-server/conf, then restart ambari-server. After this, the user is able to open the file in the encrypted zone. Basically, the property "dfs.encryption.key.provider.uri" which provides details of the KeyProvider, is obtained from the hdfs-site.xml present in the /etc/ambari-server/conf folder. Diffs ----- contrib/views/utils/src/main/java/org/apache/ambari/view/utils/hdfs/ConfigurationBuilder.java d505e9a Diff: https://reviews.apache.org/r/51578/diff/ Testing ------- **CAUSE:** ========== The error of "No KeyProvider is configured" is seen only for those cases when the HDFS uses DistributedFileSystem for its communication. When HDFS uses WebHDFSFileSystem for communication, this error is not seen and the Ambari View instance is able to open the files in the encrypted zones. Why Ambari Views use either Distributed or WebHDFS file systems is explained below: Ambari views can be created using one of the 3 modes of configuration: 1. Local cluster 2. Remote cluster 3. Custom configuration (no cluster is associated here). The HDFS works through abstraction. For Ambari Views, the actual file system used during execution depends on whether the view instance was created using a Local/Remote cluster or using Custom configuration. For instances created using Local/Remote cluster, HDFS uses Distributed File System and for instances created using Custom configuration, HDFS uses WebHDFSFileSystem. WebHDFSFileSystem is an integrated part of the HDFS ecosystem. It is aware of all the HDFS configuration. For this reason, when a KMS is configured in HDFS, WebHDFSFileSystem is aware of the KeyProvider and no special config mapping is needed. Thus, even the view instance created using Custom configuration doesn't need any special configuration and can talk to the Encryption Zones successfully. However, for view instances created using Local/Remote cluster configuration, HDFS uses the Distributed FileSystem. This Distributed FileSystem works as an HDFS client and hence, is not fully aware of all the HDFS configuration. We need to explicitly provide HDFS properties like "dfs.encryption.key.provider.uri" to these ambari view instances to provide details of the KeyProvider. The proposed fix helps in providing this property value to the view as follows. **FIX:** ======== The proposed fix (attached as "AMBARI-18071.patch") checks if the current view instance configuration has any cluster associated in its context. If there is an associated cluster then the instance has a Local/Remote cluster configuration and needs to be provided with the HDFS KeyProvider information. Otherwise, the WebHDFSFileSystem will take care of the KeyProvider if KMS is configured. To provide the property information, the parseProperties() in ConfigurationBuilder.java looked best as we also set the defaultFS property here. If a cluster is associated with the context, and if the property "dfs.encryption.key.provider.uri" is not null, then this property is set in the Configuration object and thus made available to Distributed file system of HDFS. The Ambari VIew instance works successfully with both Local and Remote configurations. One more point to note in the configuration aspect is the addition of proxyuser to the kms-site.xml for the ambari-server daemon. Without this proxyuser even the custom configuration will not work. (I had installed hadoop's KMS on the ambari-server manually) **TESTING:** ============ I have done manual testing. Log in as a user who is allowed to access encrypted zones. Go to the Ambari View instance (with Local/Remote cluster configuration). Open the encrypted zone folder and open an existing file in this directory. The user can successfully preview the file without the "No KeyProvider configured" error. Thanks, Keta Patel