> On 十月 16, 2017, 12:51 p.m., Colm O hEigeartaigh wrote: > > In addition to comment above, here are some initial remarks: > > > > a) We should update to 1.99.7 as it is the latest version > > b) getLinkList/getJobList are not implemented in SqoopClient. > > c) There is a problem with the Sqoop REST API that I haven't been able to > > figure out. "http://localhost:12000/sqoop/v1/connectors"; returns a 404 with > > 1.99.7 (but http://localhost:12000/sqoop/v1/connector/kafka-connector works > > fine), so I can't get "Test Connection" to work. Same goes for the v1/jobs > > + v1/links mentioned in the REST API. > > d) It was not picking up conf/ranger-sqoop-security.xml + hence the plugin > > wasn't working. I had to copy ranger-sqoop-security.xml to the root > > directory of the Sqoop installation. Any idea why this is? > > e) After changing the policy in Ranger and having it synced down, it only > > "applies" once I exited the sqoop shell and restarted it. Any idea why this > > is? I guess it is caching the old policy somehow?
a)OK, thanks. I have updated my sqoop from 1.99.6 to 1.99.7. b)Yes, I will give another patch to implement getLinkList/getJobList after this patch recevied, because there is some work and optimization to do for connector/link/job, and it only affects resource lookup. We are now focusing on Ranger's access control over Sqoop. c)Yes, the Sqoop REST API is different between two versions. 1.99.6 : http://localhost:12000/sqoop/v1/connectors 1.99.7 : http://localhost:12000/sqoop/v1/connector/all Same goes for the jobs and links. d)I think it's because the conf directory is not in sqoop's classpath, so ranger configuration files can not be read in JVM, add conf to sqoop classpath can resolve this problem. Please add the follow configuration to function sqoop_server_classpath_set of script bin/sqoop.sh: CLASSPATH="$CLASSPATH":"$sqoop_home/conf" e)Yes, the connector will be cached in sqoop client, when Ranger change the user's connector permissions, the show connector command is not the latest result. Execute the follow command can clear sqoop client cache: set server --host zdh-11 --port 12000 --webapp sqoop Please refer the following sqoop client code which cache connectors: org.apache.sqoop.client.SqoopClient.getConnector(String) public MConnector getConnector(String connectorName) { // Firstly try if we have this connector already in cache MConnector connector = getConnectorFromCache(connectorName); if(connector != null) return connector; // If the connector wasn't in cache and we have all connectors, // it simply do not exists. if(isAllConnectors) return null; // Retrieve all connectors from server getConnectors(); return getConnectorFromCache(connectorName); } - Qiang ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/62710/#review188127 ----------------------------------------------------------- On 十月 12, 2017, 9:02 a.m., Qiang Zhang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/62710/ > ----------------------------------------------------------- > > (Updated 十月 12, 2017, 9:02 a.m.) > > > Review request for ranger, Ankita Sinha, Don Bosco Durai, Colm O > hEigeartaigh, Gautam Borad, Madhan Neethiraj, pengjianhua, Ramesh Mani, > Selvamohan Neethiraj, sam rome, Venkat Ranganathan, and Velmurugan Periasamy. > > > Bugs: RANGER-1810 > https://issues.apache.org/jira/browse/RANGER-1810 > > > Repository: ranger > > > Description > ------- > > Apache Sqoop is a tool designed for efficiently transferring bulk data > between Apache Hadoop and structured datastores such as relational databases. > You can use Sqoop to import data from external structured datastores into > Hadoop Distributed File System or related systems like Hive and HBase. > Conversely, Sqoop can be used to extract data from Hadoop and export it to > external structured datastores such as relational databases and enterprise > data warehouses.It successfully graduated from the Incubator in March of 2012 > and is now a Top-Level Apache project. > The Ranger will further expand the influence in the hadoop ecosystem if it > supports sqoop authorization. So we should develop sqoop plugin to enable, > monitor and manage apache Sqoop2. > > Our test specialists have rigorously tested this feature. > > > Diffs > ----- > > agents-common/scripts/enable-agent.sh d31a264 > > agents-common/src/main/java/org/apache/ranger/plugin/store/EmbeddedServiceDefsUtil.java > 9463ab8 > agents-common/src/main/resources/service-defs/ranger-servicedef-sqoop.json > PRE-CREATION > plugin-sqoop/.gitignore PRE-CREATION > plugin-sqoop/conf/ranger-policymgr-ssl-changes.cfg PRE-CREATION > plugin-sqoop/conf/ranger-policymgr-ssl.xml PRE-CREATION > plugin-sqoop/conf/ranger-sqoop-audit-changes.cfg PRE-CREATION > plugin-sqoop/conf/ranger-sqoop-audit.xml PRE-CREATION > plugin-sqoop/conf/ranger-sqoop-security-changes.cfg PRE-CREATION > plugin-sqoop/conf/ranger-sqoop-security.xml PRE-CREATION > plugin-sqoop/pom.xml PRE-CREATION > plugin-sqoop/scripts/install.properties PRE-CREATION > > plugin-sqoop/src/main/java/org/apache/ranger/authorization/sqoop/authorizer/RangerSqoopAuthorizer.java > PRE-CREATION > > plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/RangerServiceSqoop.java > PRE-CREATION > > plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/SqoopClient.java > PRE-CREATION > > plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/SqoopResourceMgr.java > PRE-CREATION > > plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/json/model/SqoopConnectorResponse.java > PRE-CREATION > > plugin-sqoop/src/main/java/org/apache/ranger/services/sqoop/client/json/model/SqoopConnectorsResponse.java > PRE-CREATION > pom.xml 3958014 > ranger-sqoop-plugin-shim/.gitignore PRE-CREATION > ranger-sqoop-plugin-shim/pom.xml PRE-CREATION > > ranger-sqoop-plugin-shim/src/main/java/org/apache/ranger/authorization/sqoop/authorizer/RangerSqoopAuthorizer.java > PRE-CREATION > src/main/assembly/admin-web.xml 0e97818 > src/main/assembly/plugin-sqoop.xml PRE-CREATION > > > Diff: https://reviews.apache.org/r/62710/diff/2/ > > > Testing > ------- > > Our test specialists have rigorously tested this feature. > > > Thanks, > > Qiang Zhang > >
