[
https://issues.apache.org/jira/browse/RANGER-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480801#comment-17480801
]
kirby zhou edited comment on RANGER-3595 at 1/24/22, 3:09 AM:
--------------------------------------------------------------
[~pradeep]
The output of security-admin is a .WAR, it have the correct struct of a webapp.
{code:java}
WEB-INF/classes/org/apache/...
WEB-INF/web.xml
WEB-INF/lib{code}
But the output of kms is .JAR, its struct looks like:
{code:java}
org/apache/... # it wont be loaded by any.
WEB-INF/web.xml # which is a dirty workaround by somebody {code}
Both kms.jar and admin.war are unpacked under 'ews/webapp', but the unpacked
kms classes are meaningless.
Actually, kms load its own class by './ews/webapp/lib/ranger-kms-*.jar',
[https://github.com/apache/ranger/blob/master/distro/src/main/assembly/kms.xml#L30]
{code:java}
<moduleSet>
<useAllReactorProjects>true</useAllReactorProjects>
<includes>
<include>org.apache.ranger:ranger-kms</include>
</includes>
<binaries>
<outputDirectory>ews/webapp/lib</outputDirectory>
<includeDependencies>true</includeDependencies>
<unpack>false</unpack>
{code}
I have tried to convert kms to WAR, so the unpacked classes can be meaningful.
But it is difficult to fix the relation of ClassLoader.
The classes under ( ews/lib/{*}, ews/webapp/lib/{*},
ews/webapp/WEB-INF/classes/lib/* ) and ( webapp/WEB-INF/lib/* ,
webapp/WEB-INF/classes ) are loaded by different ClassLoader. the first group
is system ClassLoader, the followings are Tomcat WebApp ClassLoader.
We place too many jars out of webapp/WEB-INF/lib, it makes something like
ServiceLoader works strange. So I have a simple patch to workaround, put all
classes / jar out of webapp/WEB-INF/lib. It works and pretty than current
master.
was (Author: kirbyzhou):
[~pradeep]
The output of security-admin is a .WAR, it have the correct struct of a webapp.
{code:java}
WEB-INF/classes/org/apache/...
WEB-INF/web.xml
WEB-INF/lib{code}
But the output of kms is .JAR, its struct looks like:
{code:java}
org/apache/... # it wont be loaded by any.
WEB-INF/web.xml # which is a dirty workaround by somebody {code}
Both kms.jar and admin.war are unpacked under 'ews/webapp', but the unpacked
kms classes are meaningless.
Actually, kms load its own class by './ews/webapp/lib/ranger-kms-*.jar',
I have tried to convert kms to WAR, so the unpacked classes can be meaningful.
But it is difficult to fix the relation of ClassLoader.
The classes under ( ews/lib/*, ews/webapp/lib/*,
ews/webapp/WEB-INF/classes/lib/* ) and ( webapp/WEB-INF/lib/* ,
webapp/WEB-INF/classes ) are loaded by different ClassLoader. the first group
is system ClassLoader, the followings are Tomcat WebApp ClassLoader.
We place too many jars out of webapp/WEB-INF/lib, it makes something like
ServiceLoader works strange. So I have a simple patch to workaround, put all
classes / jar out of webapp/WEB-INF/lib. It works and pretty than current
master.
> Tar of KMS contains rubbish files
> ---------------------------------
>
> Key: RANGER-3595
> URL: https://issues.apache.org/jira/browse/RANGER-3595
> Project: Ranger
> Issue Type: Improvement
> Components: kms
> Reporter: kirby zhou
> Priority: Major
>
> There are lots of .class files under ews/webapp/. They wont be loaded by any
> classpath. And they are duplicated against files inside
> ews/webapp/lib/ranger-kms-3.0.0-SNAPSHOT.jar.
> It seems dirty and may cause some security problem.
> {code:bash}
> #] tar tf target/ranger-3.0.0-SNAPSHOT-kms.tar.gz ranger-3.0.0-SNAPSHOT-kms/
> | egrep 'ews/webapp/org' | head
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/kms/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/kms/biz/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/kms/dao/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/ranger/entity/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/hadoop/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/hadoop/crypto/
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/org/apache/hadoop/crypto/key/
> //代码占位符
> {code}
> * The reason is that:
> distro/src/main/assembly/kms.xml
>
> {code:java}
> <moduleSet>
> <useAllReactorProjects>true</useAllReactorProjects>
> <includes>
> <include>org.apache.ranger:ranger-kms</include>
> </includes>
> <binaries>
> <outputDirectory>ews/webapp</outputDirectory>
> <includeDependencies>false</includeDependencies>
> <unpack>true</unpack>
> </binaries>
> </moduleSet> {code}
> Why ?
>
> The secret is in kms/scripts/setup.sh:
>
> {code:java}
> setup_kms(){
> #copying ranger kms provider
> oldP=${PWD}
> cd $PWD/ews/webapp
> log "[I] Adding ranger kms provider as services in hadoop-common jar"
> for f in lib/hadoop-common*.jar
> do
> ${JAVA_HOME}/bin/jar -uf ${f}
> META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
> chown ${unix_user}:${unix_group} ${f}
> done
> cd ${oldP}
> }
> {code}
>
>
> The code above is VERY VERY DIRTY!
> It hacks into hadoop-common.jar., Overwrite resource
> "META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory". Ensure
> the following code can load
> 'org.apache.hadoop.crypto.key.RangerKeyStoreProvider$Factory' by
> 'META-INF/.../KeyProviderFactory'.
>
>
> {code:java}
> // org.apache.hadoop.crypto.key: KeyProviderFactory.java
> private static final ServiceLoader<KeyProviderFactory> serviceLoader =
> ServiceLoader.load(KeyProviderFactory.class,
> KeyProviderFactory.class.getClassLoader());
> {code}
>
>
> But this is unnecessary.
> ServiceLoader will read all resources with the same name using the
> ClassLoader of KeyProviderFactory. We just need to put a jar contains that
> property side by side of hadoop-common.jar ( ews/webapp/lib/ ). And
> ranger-kms-3.0.0-SNAPSHOT.jar already here.
> {code:java}
> % tar tf ../target/ranger-*-kms.tar.gz | egrep 'kms[^/]*\.jar|hadoop-common'
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/lib/ranger-kms-3.0.0-SNAPSHOT.jar
> ranger-3.0.0-SNAPSHOT-kms/ews/webapp/lib/hadoop-common-3.3.0.jar
> ...
> % tar tf target/ranger-kms-3.0.0-SNAPSHOT.jar | fgrep ProviderFactory
> META-INF/services/org.apache.hadoop.crypto.key.KeyProviderFactory
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
