I don't know who owns that server, but I happened to have an old copy of the tarball lying around, so I've put it somewhere you can download it:
http://ansible.eigenbase.org/download/thrift-instant-r790732.tar.gz For the long run we'll figure out what's up with the latest thrift and how to upgrade. JVS ________________________________________ From: Pradeep Kamath [prade...@yahoo-inc.com] Sent: Wednesday, June 23, 2010 9:03 AM To: hive-user@hadoop.apache.org Subject: RE: alter table add partition error In the second step of http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code “Use the approved version of thrift. This is currently thrift-instant-r790732, which you can obtain from http://instant.thrift-rpc.org” Looks like that site is down since it is not responding to requests – is there an alternate url I can try? Also, is the approved version still thrift-instant-r790732? ________________________________ From: Paul Yang [mailto:py...@facebook.com] Sent: Tuesday, June 22, 2010 6:06 PM To: hive-user@hadoop.apache.org Subject: RE: alter table add partition error Hey Pradeep, Yep, that’s on the right path :) Since you made a change to hive_metastore.thrift, you should run the thrift compiler by calling ‘ant thriftif’ within trunk/metastore. But before you can do that, the thrift compiler has to be in your path. JVS has a nice write up on how to do this: http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code Also, I’ve set up a JIRA for this issue at: https://issues.apache.org/jira/browse/HIVE-1428 Can you post your patch there? It makes it a lot easier to track issues. Thanks! -Paul From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Tuesday, June 22, 2010 10:22 AM To: hive-user@hadoop.apache.org Subject: Re: alter table add partition error Hi Paul, I have attached a draft of a patch (incomplete). The problem I am facing is that to make this change I seem to need to change the thrift idl (hive_metastore.thrift). Looking at the code structure it seems like the generate java files are checked in rather than generated through the build (is this so that the build does not depend on the thrift compiler?). I tried downloading and building thrift but hit issues in the make install - wanted to check with you if the patch looks correct so far or if I am on the wrong path. Also if the direction is right, I need some guidance on generating the java files from the modified idl. Thanks, Pradeep Paul Yang wrote: Well, the section of code you are referring to is automatically generated by the thrift compiler so we can’t modify it. As the inability to return null is a limitation of Thrift, we can work around that issue by throwing an exception instead. Check out this thread: http://publists.facebook.com/pipermail/thrift/2007-September/000085.html From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Monday, June 21, 2010 5:46 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Hi Paul, Yes I don’t mind working on a patch though I have no knowledge of how thrift code works. I was also confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() – to first check if(result.o1 != null) throw result.o1, then to just return result.success and not throw the “unknown result” exception. I tried something along these lines and the alter table worked. Am I missing something? Thanks, Pradeep ________________________________ From: Paul Yang [mailto:py...@facebook.com] Sent: Monday, June 21, 2010 2:35 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Ah okay. So what’s going on is that getPartition() is supposed to return null if the specified partition doesn’t exist. But because Thrift can’t handle null return values, we are seeing this exception. This isn’t a problem with a local metastore because Thrift isn’t used. One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior. Pradeep, are you interested in working on a patch for this problem? From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Monday, June 21, 2010 2:11 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error I tried debugging in code a little more. Here is what I found: The code in ThriftHiveMetaStore eventually makes a call – get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn’t already exist. I added a debug line in the following code: public Partition recv_get_partition() throws MetaException, TException { TMessage msg = iprot_.readMessageBegin(); if (msg.type == TMessageType.EXCEPTION) { TApplicationException x = TApplicationException.read(iprot_); iprot_.readMessageEnd(); throw x; } get_partition_result result = new get_partition_result(); result.read(iprot_); System.err.println("XXX: result:" + result); iprot_.readMessageEnd(); if (result.isSetSuccess()) { return result.success; } if (result.o1 != null) { throw result.o1; } throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result"); } I also put a debug statements in the read() method: public void read(TProtocol iprot) throws TException { TField field; iprot.readStructBegin(); System.err.println("XXX: Reading TProtocol object:"); while (true) { field = iprot.readFieldBegin(); System.err.println("XXX: field just read:" + field); if (field.type == TType.STOP) { break; I got XXX: Reading TProtocol object: XXX: field just read:<TField name:'' type:0 field-id:0> XXX: result:get_partition_result(success:null, o1:null) The field read in the thrift response message is of type “STOP” and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception. Any pointers? TIA, Pradeep ________________________________ From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Friday, June 18, 2010 2:51 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Looks like the standalone script works fine against the existing partition: ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10 Partition(parameters={'transient_lastDdlTime': '1276881277'}, tableName='dummy', createTime=1276881277, lastAccessTime=0, values=['20100602', '100', 'view', '10'], dbName='default', sd=StorageDescriptor(outputFormat='org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat', sortCols=[], inputFormat='org.apache.hadoop.mapred.TextInputFormat', cols=[FieldSchema(comment=None, type='string', name='partition_name'), FieldSchema(comment=None, type='int', name='partition_id')], compressed=False, bucketCols=[], numBuckets=-1, parameters={}, serdeInfo=SerDeInfo(serializationLib='org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe', name=None, parameters={'serialization.format': '1'}), location='hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/20100602/100/view/10')) [prade...@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore] However when I tried to add another partition with the hive cli using thrift: hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='click',testid='10') location '/user/pradeepk/dummy/20100602/100/click/10';" 10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask tail -30 /tmp/pradeepk/hive.log at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) 2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778) at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756) ... 15 more 2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask ________________________________ From: Paul Yang [mailto:py...@facebook.com] Sent: Friday, June 18, 2010 2:19 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Looks like the thrift python libraries aren’t in your path. run: export PYTHONPATH=<path-to-trunk>/build/dist/lib/py/ before trying the remote command From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Friday, June 18, 2010 1:38 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Sorry, don’t know python – so couldn’t make sense out of the following error when I run the command you suggested: [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10 Traceback (most recent call last): File "./ThriftHiveMetastore-remote", line 11, in ? from thrift.transport import TTransport ImportError: No module named thrift.transport Thanks, Pradeep ________________________________ From: Paul Yang [mailto:py...@facebook.com] Sent: Friday, June 18, 2010 12:10 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Hi Pradeep, In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=20100602/srcid=100/action=view/testid=10 Cheers, Paul From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Friday, June 18, 2010 11:26 AM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Hi Ning, I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an “env” command printed just before the following line in execHiveCmd.sh exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@" There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build. The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost. Any pointers appreciated! Thanks, Pradeep [1] HADOOP_CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar: CLASSPATH=/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar: [2] cat /home/y/libexec/hive_client/conf/hive-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files --> <!-- that are implied by Hadoop setup variables. --> <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive --> <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized --> <!-- resource). --> <!-- Hive Execution Parameters --> <property> <name>hive.metastore.uris</name> <value>thrift://localhost:9080</value> <description>host and port for the thrift metastore server</description> </property> <property> <name>hive.metastore.local</name> <value>false</value> <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>/user/pradeepk/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> </configuration> ________________________________ From: Ning Zhang [mailto:nzh...@facebook.com] Sent: Friday, June 18, 2010 10:50 AM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: Re: alter table add partition error Pradeep, I ran the commands you provided and it succeeded with the expected behavior. One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift? Thanks, Ning On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote: I think there are two separate issues here – I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details: cat create_dummy.sql CREATE external TABLE if not exists dummy ( partition_name string ,partition_id int ) PARTITIONED BY ( datestamp string, srcid string, action string, testid string ) row format delimited stored as textfile location '/user/pradeepk/dummy'; hive -f create_dummy.sql 10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt OK Time taken: 0.627 seconds hive -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';" 10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp = '20100602', srcid = '100',action='view',testid='10') location '/user/pradeepk/dummy/20100602/100/view/10';" 10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt OK Time taken: 5.849 seconds Is there some thrift setting I am missing or is this a bug? – If it is the latter, I can open a jira with the above details. Thanks, Pradeep ________________________________ From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Thursday, June 17, 2010 1:25 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Here are the create table and alter table statements: CREATE external TABLE if not exists mytable ( bc string ,src_spaceid string ,srcpvid string ,dstpvid string ,dst_spaceid string ,page_params map<string, string> ,clickinfo map<string, string> ,viewinfo array<map<string, string>> ) PARTITIONED BY ( datestamp string, srcid string, action string, testid string ) row format serde 'com.yahoo.mySerde’ stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat' location '/user/pradeepk/mytable’; hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';" I get the following error: Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask If I don’t use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed: hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = '19174',action='click',testid='NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';" However I get errors when I try to run a query: [prade...@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp='20091101';" 10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator java.lang.IllegalArgumentException: Can not create a Path from an empty string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82) at org.apache.hadoop.fs.Path.<init>(Path.java:90) at org.apache.hadoop.fs.Path.<init>(Path.java:50) at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523) at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:603) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504) Any help is much appreciated. Pradeep ________________________________ From: Ashish Thusoo [mailto:athu...@facebook.com] Sent: Thursday, June 17, 2010 11:15 AM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error hmm... Can you send the exact command and also the create table command for this table. Ashish ________________________________ From: Pradeep Kamath [mailto:prade...@yahoo-inc.com] Sent: Thursday, June 17, 2010 9:09 AM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: RE: alter table add partition error Sorry – that was a cut-paste error – I don’t have the action part – so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it’s something to do with my Serde – unfortunately the error I see gives me no clue of what could be wrong – any help would be greatly appreciated! Thanks, Pradeep ________________________________ From: yq he [mailto:hhh.h...@gmail.com] Sent: Wednesday, June 16, 2010 5:54 PM To: hive-user@hadoop.apache.org<mailto:hive-user@hadoop.apache.org> Subject: Re: alter table add partition error Hi Pradeep, partition definition need to be key-value pairs. partition key `action` seems missed the value. Thanks Yongqiang On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <prade...@yahoo-inc.com<mailto:prade...@yahoo-inc.com>> wrote: Hi, I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition – any help would be greatly appreciated. Thanks, Pradeep == session output === [prade...@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp = '20091101', srcid = ‘10’,action) location '/user/pradeepk/mytable/20091101/10’;" 10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask [prade...@chargesize:~/dev/howl] == session output === /tmp/pradeepk/hive.log has: 2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown result at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778) at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown result at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:931) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:899) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:500) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756) ... 15 more The thrift server messages are: 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=default tbl=mytable 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called 10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=default tbl=mytable
