Ah... i see whats going on. When hive finally issues a "load" statement that actually USES the SerDe, hadoop takes over.
At that point, hadoop/lib needs to have the hive serde libraries copied into it, otherwise it cant find the AbstractSerDe classes. I wonder if there is a more elegant way to integrate hive/hadoop that im not thinking of (hive_test maybe :)) On Mon, Dec 30, 2013 at 7:17 AM, Jay Vyas <jayunit...@gmail.com> wrote: > Hi folks.. > > > Is there an JDBC or other API driven, programmatic way to launch a Hive > Job using SerDe's from the contrib library? > > I've been attempting to run a query using Hive's JDBC driver which uses > the regex SerDe, however, it appears that hive doesn;t see this class. As > Im launching the job from java via JDBC, im not sure where the class is not > found (i added it to HIVE_HOME/lib AND HADOOP_HOME/lib, so this is *really* > odd ). > > 2013-12-30 07:00:58.795 java[5829:1c03] Unable to load realm info from > SCDynamicStore > org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is > not allowed in prolog. > Continuing ... > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/hive/serde2/AbstractSerDe > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:791) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:423) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:410) > at java.lang.ClassLoader.loadClass(ClassLoader.java:356) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:264) > at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:64) > at com.sun.beans.finder.ClassFinder.findClass(ClassFinder.java:106) > at com.sun.beans.finder.ClassFinder.resolveClass(ClassFinder.java:166) > at > com.sun.beans.decoder.DocumentHandler.findClass(DocumentHandler.java:382) > at > com.sun.beans.decoder.ClassElementHandler.getValue(ClassElementHandler.java:60) > at > com.sun.beans.decoder.StringElementHandler.getValueObject(StringElementHandler.java:92) > at > com.sun.beans.decoder.ElementHandler.endElement(ElementHandler.java:169) > at > com.sun.beans.decoder.DocumentHandler.endElement(DocumentHandler.java:305) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1741) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2898) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607) > at > com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210) > at > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568) > at > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:302) > at > com.sun.beans.decoder.DocumentHandler.parse(DocumentHandler.java:356) > at java.beans.XMLDecoder.parsingComplete(XMLDecoder.java:192) > at java.beans.XMLDecoder.readObject(XMLDecoder.java:238) > at > org.apache.hadoop.hive.ql.exec.Utilities.deserializeMapRedWork(Utilities.java:542) > at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:679) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:160) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.serde2.AbstractSerDe > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:423) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:356) > ... 44 more > > > I've seen a JIRA alluding that custom serdes may *not* work in Hive > JDBC... Is that still the case > https://issues.apache.org/jira/browse/HIVE-2685 ? > > -- > Jay Vyas > http://jayunit100.blogspot.com > -- Jay Vyas http://jayunit100.blogspot.com
