> On Oct. 17, 2014, 8:45 a.m., Qian Xu wrote: > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseToInitializer.java, > > line 96 > > <https://reviews.apache.org/r/26581/diff/2/?file=724138#file724138line96> > > > > I'm facing a similar situation for Kite connector. How about > > > > ` > > @Override > > public List<String> getJars(InitializerContext context, > > LinkConfiguration link, ToJobConfiguration job) { > > List<String> jars = super.getJars(context, link, job); > > > > addDesiredJars(jars, ((URLClassLoader) > > Thread.currentThread().getContextClassLoader())); > > > > return jars; > > } > > > > private void addDesiredJars(List<String> jars, URLClassLoader loader) > > { > > for (URL url : loader.getURLs()) { > > String path = url.getPath(); > > if (path.endsWith(".jar") && !jars.contains(url.getFile())) { > > jars.add(url.toString()); > > } > > } > > } > > ` > > Qian Xu wrote: > `private static void addDesiredJars(...` > > Qian Xu wrote: > I'm think why these jars are not submitted together with MR job by > default?
I did not yet reviewed entire code, so I'm just responding to this note :) 1) We are not shipping entire Sqoop Server classpath to the mapreduce because there is no good reason why to do that and there is plenty of reasons why not to do that. Albeit make it easy for connector developer is not one of them :) Server have on the classpath jars for all connectors, entire functionality and even Hadoop. That can be several megabytes worth of jars that we would transfer to each and every node in the cluster for every single mapreduce job. That is huge waste of resources. E.g. why would be shipping Tomcat jars to MR? Or HBase jars where there is no HBase involved? Why would we ship hadoop jars when Hadoop is by definition on the MR job classpath already? 2) If you look how other projects are dealing with that you'll found that most of them are exposing methods that will return list of jars that is actually required. For example HBase: https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java#L736 That downstream components are using. For example Pig: https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java#L751 - Jarek ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26581/#review57123 ----------------------------------------------------------- On Oct. 17, 2014, 6:37 a.m., Abraham Elmahrek wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/26581/ > ----------------------------------------------------------- > > (Updated Oct. 17, 2014, 6:37 a.m.) > > > Review request for Sqoop. > > > Bugs: SQOOP-1156 > https://issues.apache.org/jira/browse/SQOOP-1156 > > > Repository: sqoop-sqoop2 > > > Description > ------- > > commit b9af5c1556e2c88a3c861950a6a296733fa1e4e7 > Author: Abraham Elmahrek <[email protected]> > Date: Thu Oct 9 22:56:34 2014 -0700 > > SQOOP-1156: HBase connector > > :000000 100644 0000000... 61dd408... A connector/connector-hbase/pom.xml > :000000 100644 0000000... b0e4ea0... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseConnector.java > :000000 100644 0000000... 975cb40... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseConnectorConstants.java > :000000 100644 0000000... 2e08bfd... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseConnectorError.java > :000000 100644 0000000... cf632c9... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseConnectorRepositoryUpgrader.java > :000000 100644 0000000... bc61993... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseExecutor.java > :000000 100644 0000000... 991846f... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseLoader.java > :000000 100644 0000000... 325591c... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseToDestroyer.java > :000000 100644 0000000... 51fd885... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseToInitializer.java > :000000 100644 0000000... 6e306b1... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/configuration/LinkConfig.java > :000000 100644 0000000... a4b825c... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/configuration/LinkConfiguration.java > :000000 100644 0000000... 15feb54... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/configuration/ToJobConfig.java > :000000 100644 0000000... cc71b78... A > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/configuration/ToJobConfiguration.java > :000000 100644 0000000... 15a9425... A > connector/connector-hbase/src/main/resources/hbase-connector-resources.properties > :000000 100644 0000000... 1fc360e... A > connector/connector-hbase/src/main/resources/sqoopconnector.properties > :000000 100644 0000000... c78042a... A > connector/connector-hbase/src/test/java/org/apache/sqoop/connector/hbase/TestExecutor.java > :000000 100644 0000000... 60f8217... A > connector/connector-hbase/src/test/java/org/apache/sqoop/connector/hbase/TestLoader.java > :000000 100644 0000000... 44ffced... A > connector/connector-hbase/src/test/resources/log4j.properties > :100644 100644 e98a0fc... 35c665e... M connector/pom.xml > :100644 100644 f25a29f... a556bcf... M pom.xml > :100644 100644 67baaa5... 21a1fa9... M server/pom.xml > :100644 100644 7a80710... fbd4e84... M test/pom.xml > > > Diffs > ----- > > connector/connector-hbase/pom.xml PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseConnector.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseConnectorConstants.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseConnectorError.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseConnectorRepositoryUpgrader.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseExecutor.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseLoader.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseToDestroyer.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/HBaseToInitializer.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/configuration/LinkConfig.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/configuration/LinkConfiguration.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/configuration/ToJobConfig.java > PRE-CREATION > > connector/connector-hbase/src/main/java/org/apache/sqoop/connector/hbase/configuration/ToJobConfiguration.java > PRE-CREATION > > connector/connector-hbase/src/main/resources/hbase-connector-resources.properties > PRE-CREATION > connector/connector-hbase/src/main/resources/sqoopconnector.properties > PRE-CREATION > > connector/connector-hbase/src/test/java/org/apache/sqoop/connector/hbase/TestExecutor.java > PRE-CREATION > > connector/connector-hbase/src/test/java/org/apache/sqoop/connector/hbase/TestLoader.java > PRE-CREATION > connector/connector-hbase/src/test/resources/log4j.properties PRE-CREATION > connector/pom.xml e98a0fc > pom.xml f25a29f > server/pom.xml 67baaa5 > test/pom.xml 7a80710 > > Diff: https://reviews.apache.org/r/26581/diff/ > > > Testing > ------- > > mvn clean verify + can transfer from mysql to hbase. > > > Thanks, > > Abraham Elmahrek > >
