Hi Chris, Lovely! It all makes sense now, and it works (until my next problem...).
Thanks for explaining the basics of the Maven assembly plugin; now I understand what that file was for. Just a note, no need to follow-up on it if you don't know why it happened (I doubt it's Samza-related, anyhow): My Samza code package's pom.xml already did have a dependency for the relevant jar files, and the Samza package was already being included via Maven's assembly plugin src.xml (as I am currently using the hello-samza project's "wikipedia" package to work off). The problem is that even though useTransitiveFiltering was set to true, those dependencies defined in my package's pom.xml were not being included transitively. The strange thing is that other dependencies defined in my package's pom.xml, such as jackson and slf4j libraries, do seem like they were being pulled in transitively. Maybe I'm just delusional... Anyway, this is less of a problem for me now that I understand how the assembly plugin works, although hopefully can be of help if anyone else encounters a similar problem in the future. Thank you, Jonathan ------------------------------------------------------ From: Chris Riccomini [email protected] Reply: [email protected] [email protected] Date: 10 February 2014 at 16:01:56 To: [email protected] [email protected] Subject: Re: Unable to call external library classes from within Samza > > Hey Jonathan, > > Yep, you should put your jar into the lib directory of your Samza > job > package. > > This can be done using Maven's assembly plugin. You can have the > plugin > put all runtime dependencies (including transitive dependencies) > into your > lib directory. Then you just need to add a dependency, and it'll > get > sucked in as part of the Maven build. This is what hello-samza > does. > > Here's hello-samza's assembly file: > > https://github.com/linkedin/hello-samza/blob/master/samza-job-package/src/m > ain/assembly/src.xml?source=c > > > The most important part is the "dependencySets" section. > > > lib > > org.apache.samza:samza-core_2.8.1 > org.apache.samza:samza-kafka_2.8.1 > org.apache.samza:samza-serializers_2.8.1 > org.apache.samza:samza-yarn_2.8.1 > org.slf4j:slf4j-log4j12 > samza:samza-wikipedia > org.apache.kafka:kafka_2.8.1 > > true > > > You can see that we're including transitive dependencies, and > depending on > the Samza libraries, as well as samza-wikipedia. You can either > add a new > block here for your MySQL jars, OR you can have your > Samza code > package's pom.xml depend on the MySQL jar, and it will get sucked > in > automatically (due to useTransitiveFiltering). > > > Cheers, > Chris > > On 2/10/14 3:49 PM, "Jonathan Poltak Samosir" > wrote: > > >Yep, sure enough the jar I need is not in my classpath (as per the > YARN > >container's stdout). > > > >To fix this issue, and for future reference, what is the best > way of > >adding external libraries to the classpath for Samza? > > > >Going by how the samza/bin/run-class.sh script is written, > it seems like > >it would probably work if I place the jars I need in samza/lib > directory. > >This seems rather hacky though, and I would have to do it everytime > I run > >`mvn clean package`. > > > >So is there something I'm missing in the Maven pom.xml that is > placing > >all the needed libraries, apart from the new ones I've added > as > >dependencies, into the samza/lib directory of the tarball? > (sorry if this > >is something obvious...) > > > >Thanks, > >Jonathan > > > > > >------------------------------------------------------ > >From: Chris Riccomini [email protected] > >Reply: [email protected] [email protected] > >Date: 10 February 2014 at 14:16:04 > >To: [email protected] [email protected] > >Subject: Re: Unable to call external library classes from within > Samza > > > >> > >> Hey Jonathan, > >> > >> Can you post your classpath? You can usually find this in your > >> YARN > >> container's stdout file, if you're running with YARN. If you're > >> running > >> with LocalJobFactory, it should print to SDTOUT. > >> > >> Cheers, > >> Chris > >> > >> On 2/8/14 11:54 AM, "Jonathan Poltak Samosir" > >> wrote: > >> > >> >Hello, > >> > > >> >This is a bit of a follow on on a previous thread of mine, > >> >"org.apache.hadoop.util.Shell$ExitCodeException whenever > >> Samza container > >> >launches". > >> > > >> >So I am not sure whether this is a bug, or if I am not doing something > >> > >> >correct, or it is intended, but whenever I attempt to call > >> >Class.forName() on an external library class from within > Samza, > >> I am > >> >running into a ClassNotFoundException. My Maven deps are > set > >> up > >> >correctly, and the exact same code works without any issues > >> if invoked > >> >manually through a main() method, for example, as opposed > to > >> running it > >> >through Samza. > >> > > >> >The reason I want to do this, is to test that classes for JDBC > drivers > >> > >> >can be found and used. I fully understand that this method > is > >> no longer > >> >needed as of JDBC 4.0, and that the drivers should be automatically > >> >loaded if found in the class path, although this isn't happening > >> either > >> >(an SQLException is thrown with "No suitable driver found" > >> if I leave out > >> >the Class.forName() call, which leads to the same problem). > >> Just so you > >> >know, this same code also works fine and loads the appropriate > >> JDBC > >> >driver if invoked directly through a main() method. > >> > > >> >I have also tried this with a number of different external > libraries, > >> >both pulled in using Maven and manually linking .jar files > on > >> my disk, > >> >and the same result; those classes can be found fine from a > standard > >> Java > >> >main() method call, but cannot be found running in a Samza > container. > >> > > >> >Has anyone encountered the same issue, or know of why this > could > >> be > >> >happening? > >> > > >> >I hope all that made sense, let me know if it didn't and I'll > try > >> to > >> >rephrase. > >> > > >> >Here is the class I am trying to do this in, in case anyone wants > >> to see > >> >what I'm talking about: > >> > >>>https://github.com/poltak/hello-samza/blob/database-reader/samza-wikiped > >>>ia > >> > >>>/src/main/java/samza/examples/databasereader/system/DatabaseReaderConsum > >>>er > >> >.java > >> > > >> >Anyway, thanks for your time, > >> >Jonathan > >> > >> > > > >
