Hi folks..
Is there an JDBC or other API driven, programmatic way to launch a Hive Job using SerDe's from the contrib library? I've been attempting to run a query using Hive's JDBC driver which uses the regex SerDe, however, it appears that hive doesn;t see this class. As Im launching the job from java via JDBC, im not sure where the class is not found (i added it to HIVE_HOME/lib AND HADOOP_HOME/lib, so this is *really* odd ). 2013-12-30 07:00:58.795 java[5829:1c03] Unable to load realm info from SCDynamicStore org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog. Continuing ... Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/AbstractSerDe at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:791) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:410) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:64) at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:106) at com.sun.beans.finder.ClassFinder.resolveClass(ClassFinder.java:166) at com.sun.beans.decoder.DocumentHandler.findClass(DocumentHandler.java:382) at com.sun.beans.decoder.ClassElementHandler.getValue(ClassElementHandler.java:60) at com.sun.beans.decoder.StringElementHandler.getValueObject(StringElementHandler.java:92) at com.sun.beans.decoder.ElementHandler.endElement(ElementHandler.java:169) at com.sun.beans.decoder.DocumentHandler.endElement(DocumentHandler.java:305) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1741) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2898) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302) at com.sun.beans.decoder.DocumentHandler.parse(DocumentHandler.java:356) at java.beans.XMLDecoder.parsingComplete(XMLDecoder.java:192) at java.beans.XMLDecoder.readObject(XMLDecoder.java:238) at org.apache.hadoop.hive.ql.exec.Utilities.deserializeMapRedWork(Utilities.java:542) at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:679) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:160) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.serde2.AbstractSerDe at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) ... 44 more I've seen a JIRA alluding that custom serdes may *not* work in Hive JDBC... Is that still the case https://issues.apache.org/jira/browse/HIVE-2685 ? -- Jay Vyas http://jayunit100.blogspot.com