Thanks Abe. Right now, I am playing around with my connector. Will try out your solution when I hit production.
On Wed, Nov 12, 2014 at 2:04 PM, Abraham Elmahrek <[email protected]> wrote: > Hey there, > > It seems like you've downloaded the Sqoop2 tarball. The common.loader > should be fine. > > Here's an example setting: > > common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar > > With the above setting, you should be able to just drop your jar where the > rest of the Sqoop jars are. > > I think the best way to do this is use bigtop or one of the Hadoop > distributors to install Sqoop2 and simply place the jar in /var/lib/sqoop2. > > Sqoop uses tomcat currently and searches for connectors via classloaders. > > -Abe > > On Wed, Nov 12, 2014 at 10:20 AM, Vinoth Chandar <[email protected]> wrote: > >> Thanks for the help!!.. >> >> >> If you cannot use **any** connector, change “common.loader” to load >> correct jars will help. >> Can you please elaborate what you mean by 'any' connector? As I >> mentioned, I placed by connector jar in common.loader and I only got a >> NoClassDefFoundError on org.apache.sqoop.connector.spi.SqoopConnector, >> which is packaged into the WAR.. >> >> >> In other words, I think if my connector jar is part of the WAR, things >> will work (effectively what I achieved by adding my connector also to the >> pom.xml just like hdfs/jdbc connectors).. I am trying to understand whats >> the right way to do this.. >> >> @Ravi , I tried the lib approach also (which is based off common.loader >> again). did not help. Details below. >> >> I have the sqoop2 branch checked out. >> 1) I first did the following to build out a binary package >> $ mvn clean package -DskipTests -Dbinary >> >> 2) Hopping into the install directory from there on >> $ pwd >> ~/dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200 >> $ ls server/lib/*connector*jar >> server/lib/my-connector-0.1-SNAPSHOT.jar # placed my connector jar in lib >> $ jar tf ./server/target/sqoop.war | grep -i my # no match >> $ jar tf ./server/webapps/sqoop.war | grep -i my # no match >> >> When I start up the server, it cannot find the SqoopConnector class which >> my jar references since I am extending SqoopConnector >> >> Thanks >> Vinoth >> >> >> On Tue, Nov 11, 2014 at 7:09 PM, Xu, Qian A <[email protected]> wrote: >> >>> Hi Vinoth, >>> >>> >>> >>> If you cannot use **any** connector, change “common.loader” to load >>> correct jars will help. >>> >>> If only your connector meets NoClassDefFound issue, it means your jars >>> are not packaged with sqoop.war. >>> >>> >>> >>> --Stanley >>> >>> >>> >>> *From:* Vinoth Chandar [mailto:[email protected]] >>> *Sent:* Wednesday, November 12, 2014 9:33 AM >>> *To:* [email protected] >>> *Subject:* Install a new connector jar into sqoop2 >>> >>> >>> >>> Hey guys, >>> >>> >>> >>> I am working on a connector which connects to a sharded mysql cluster as >>> the source. I built out the connector jar with its >>> sqoopconnector.properties. >>> >>> >>> >>> And I tried to make sqoop pick up the jar a couple of ways >>> >>> >>> >>> 1) Add the path to the jar in "common.loader" property in >>> catalina.properties: This gives a NoClassDefFound on the >>> SqoopConnector.class since it is probably in tomcat's system loader, which >>> may be referenceable by external libs. >>> >>> >>> >>> 2) Randomly placing the connector jar in shared/server loader, which >>> also did not work. >>> >>> >>> >>> Wound up adding the connector as a dependency in sqoop-server's pom and >>> it worked. (sort of validates the theory in 1) >>> >>> >>> >>> I am probably a tomcat noob, so please let me know if there is a cleaner >>> way to do this or supported documentation .. Or if this is an open issue, >>> happy to work on a patch.. >>> >>> Thanks >>> >>> Vinoth >>> >> >> >
