Another quick follow up on this itself. If I try to run a query that invokes a M/R job, then it throws me this exception:
java.io.FileNotFoundException: File does not exist: /Users/my-classes.jar *at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:722) * at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208) at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:77) Looking at the stacktrace, it seems like it is searching for the jar on HDFS rather than local filesystem. Is that intended? All "Select *" queries that do not spawn a M/R job work fine. Thanks, On Wed, Jun 13, 2012 at 9:44 AM, [email protected] < [email protected]> wrote: > Cool. That worked! > > Thanks guys. > > > On Wed, Jun 13, 2012 at 9:35 AM, Edward Capriolo <[email protected]>wrote: > >> Right, The end result is that same. The hive shell script currently >> bulds a list of aux_lib jars and generates the auxpath arguments. but >> it is good to know these files do not need to be in a static folder. >> >> On Wed, Jun 13, 2012 at 10:32 AM, Rubin, Bradley S. >> <[email protected]> wrote: >> > Another option is to specify the path in the hive command: >> > >> > hive --auxpath ~/my-classes.jar >> > >> > -- Brad >> > >> > On Jun 13, 2012, at 9:23 AM, Edward Capriolo wrote: >> > >> >> You need to put these jars in your aux_lib folder or in your hadoop >> >> classpath. There is a subtle difference between that classpath and the >> >> classpath used by UDF and anything that involves a serde or input >> >> format needs to be in auxlib. >> >> >> >> Edward >> >> >> >> On Wed, Jun 13, 2012 at 10:20 AM, [email protected] >> >> <[email protected]> wrote: >> >>> Hello, >> >>> >> >>> In order to provide a custom "serialization.class" to a SerDe, I >> created a >> >>> jar containing all my custom serialization classes and added them to >> the >> >>> hive classpath with "ADD JAR my-classes.jar" command. Now when I try >> to use >> >>> these custom classes via CLI, it still throws me a >> "ClassNotFoundException" >> >>> for those custom classes in my jar. >> >>> >> >>> Is there something that I am missing? I confirmed that 'list jars' is >> >>> showing me the custom jar that I added. >> >>> >> >>> Any help would be appreciated. >> >>> >> >>> Thanks, >> >>> -- >> >>> Swarnim >> > >> > > > > -- > Swarnim > -- Swarnim
