Vinoth, We finally have this fixed in the SQOOP-1821. Kindly let us know if you run into other issues.
Thanks again for all the inputs. --Veena Best, *./Vee* On Thu, Nov 13, 2014 at 8:31 AM, Vinoth Chandar <[email protected]> wrote: > Thanks Abe. > > Right now, I am playing around with my connector. Will try out your > solution when I hit production. > > On Wed, Nov 12, 2014 at 2:04 PM, Abraham Elmahrek <[email protected]> > wrote: > >> Hey there, >> >> It seems like you've downloaded the Sqoop2 tarball. The common.loader >> should be fine. >> >> Here's an example setting: >> >> common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${catalina.home}/../lib/*.jar,/usr/lib/hadoop/*.jar >> >> With the above setting, you should be able to just drop your jar where >> the rest of the Sqoop jars are. >> >> I think the best way to do this is use bigtop or one of the Hadoop >> distributors to install Sqoop2 and simply place the jar in /var/lib/sqoop2. >> >> Sqoop uses tomcat currently and searches for connectors via classloaders. >> >> -Abe >> >> On Wed, Nov 12, 2014 at 10:20 AM, Vinoth Chandar <[email protected]> wrote: >> >>> Thanks for the help!!.. >>> >>> >> If you cannot use **any** connector, change “common.loader” to load >>> correct jars will help. >>> Can you please elaborate what you mean by 'any' connector? As I >>> mentioned, I placed by connector jar in common.loader and I only got a >>> NoClassDefFoundError on org.apache.sqoop.connector.spi.SqoopConnector, >>> which is packaged into the WAR.. >>> >>> >>> In other words, I think if my connector jar is part of the WAR, things >>> will work (effectively what I achieved by adding my connector also to the >>> pom.xml just like hdfs/jdbc connectors).. I am trying to understand whats >>> the right way to do this.. >>> >>> @Ravi , I tried the lib approach also (which is based off common.loader >>> again). did not help. Details below. >>> >>> I have the sqoop2 branch checked out. >>> 1) I first did the following to build out a binary package >>> $ mvn clean package -DskipTests -Dbinary >>> >>> 2) Hopping into the install directory from there on >>> $ pwd >>> ~/dist/target/sqoop-2.0.0-SNAPSHOT-bin-hadoop200 >>> $ ls server/lib/*connector*jar >>> server/lib/my-connector-0.1-SNAPSHOT.jar # placed my connector jar in lib >>> $ jar tf ./server/target/sqoop.war | grep -i my # no match >>> $ jar tf ./server/webapps/sqoop.war | grep -i my # no match >>> >>> When I start up the server, it cannot find the SqoopConnector class >>> which my jar references since I am extending SqoopConnector >>> >>> Thanks >>> Vinoth >>> >>> >>> On Tue, Nov 11, 2014 at 7:09 PM, Xu, Qian A <[email protected]> wrote: >>> >>>> Hi Vinoth, >>>> >>>> >>>> >>>> If you cannot use **any** connector, change “common.loader” to load >>>> correct jars will help. >>>> >>>> If only your connector meets NoClassDefFound issue, it means your jars >>>> are not packaged with sqoop.war. >>>> >>>> >>>> >>>> --Stanley >>>> >>>> >>>> >>>> *From:* Vinoth Chandar [mailto:[email protected]] >>>> *Sent:* Wednesday, November 12, 2014 9:33 AM >>>> *To:* [email protected] >>>> *Subject:* Install a new connector jar into sqoop2 >>>> >>>> >>>> >>>> Hey guys, >>>> >>>> >>>> >>>> I am working on a connector which connects to a sharded mysql cluster >>>> as the source. I built out the connector jar with its >>>> sqoopconnector.properties. >>>> >>>> >>>> >>>> And I tried to make sqoop pick up the jar a couple of ways >>>> >>>> >>>> >>>> 1) Add the path to the jar in "common.loader" property in >>>> catalina.properties: This gives a NoClassDefFound on the >>>> SqoopConnector.class since it is probably in tomcat's system loader, which >>>> may be referenceable by external libs. >>>> >>>> >>>> >>>> 2) Randomly placing the connector jar in shared/server loader, which >>>> also did not work. >>>> >>>> >>>> >>>> Wound up adding the connector as a dependency in sqoop-server's pom and >>>> it worked. (sort of validates the theory in 1) >>>> >>>> >>>> >>>> I am probably a tomcat noob, so please let me know if there is a >>>> cleaner way to do this or supported documentation .. Or if this is an open >>>> issue, happy to work on a patch.. >>>> >>>> Thanks >>>> >>>> Vinoth >>>> >>> >>> >> >
