Hive metastore schema question related to ColumnDescriptors
Hi folks, In some recent work, I've been looking at some issues related to the Hive metastore schema. It seems more dev related, hence I'm not sure if the user list is ideal, but will ask anyway. Still, it would help me if someone can clarify a few questions as below: Firstly, I would like to know the relationship between a Table, StorageDescriptor and ColumnDescriptor, primarily from a cardinality point of view. Is it 1-1 / many-1 between them ? One specific reason I ask is related to the Hive upgrade scripts of https://issues.apache.org/jira/browse/HIVE-2246 that introduced the CDS table. In the upgrade scripts, AFAICS, we pick up non-null SD_IDs from the TBLS table, and insert them into (ultimately) into the CDS table. However, in these tables, the IDs are primary keys and hence need to be unique. This would imply that we expect the values in SD_ID column in the TBLS table to be unique as well ? If yes, there doesn't seem to be a constraint of the TBLS table to enforce this. If no, the upgrade script is doing the wrong thing. Could someone please comment on this ? Thanks Hemanth
Re: unbalanced transaction calls
Thanks, Vinod. We'll try and find this out. On Sat, Feb 23, 2013 at 3:25 AM, Vinod Kumar Vavilapalli < vino...@hortonworks.com> wrote: > If we can enable access logs on the meta store and figure out the last > query before the one that failed, it should be easy to track it. > > I did find a couple of unbalanced calls in code. Will look more carefully > and file a ticket. > > Thanks, > +Vinod Kumar Vavilapalli > Hortonworks Inc. > http://hortonworks.com/ > > On Feb 22, 2013, at 4:45 AM, Hemanth Yamijala wrote: > > Hi, > > This error is coming when we are running a CREATE TABLE script. Where are > you suggesting we make these changes ? > > > On Fri, Feb 22, 2013 at 1:31 PM, Navis류승우 wrote: > >> Could you try to make transaction related methods synchronized? >> >> It's >> openTransaction/commitTransaction/isActiveTransaction/rollbackTransaction >> on ObjectStore.java >> >> 2013/2/22 Hemanth Yamijala : >> > Hi, >> > >> > We are running into the same problem as well. Is there any clue what >> could >> > be wrong ? >> > >> > Thanks >> > hemanth >> > >> > >> > On Wed, Feb 6, 2013 at 1:51 AM, James Warren >> > wrote: >> >> >> >> As part of our daily workflow, we're running a few hundred hive >> >> queries that are coordinated through oozie. Recently we're >> >> encountering issues where on average a job or two fails - and never >> >> the same query. The observed error is: >> >> >> >> FAILED: Error in metadata: java.lang.RuntimeException: >> >> commitTransaction was called but openTransactionCalls = 0. This >> >> probably indicates that there are unbalanced calls to >> >> openTransaction/commitTransaction >> >> >> >> which I've seen was referred to in HIVE-1760 but patched in 0.7. >> >> >> >> We're running Hive 0.9 (from CDH 4.1 - will redirect to the Cloudera >> >> lists if that is a more appropriate form) - any ideas / suggestions >> >> where I should start looking? >> >> >> >> cheers, >> >> -James >> > >> > >> > > >
Re: unbalanced transaction calls
Hi, This error is coming when we are running a CREATE TABLE script. Where are you suggesting we make these changes ? On Fri, Feb 22, 2013 at 1:31 PM, Navis류승우 wrote: > Could you try to make transaction related methods synchronized? > > It's > openTransaction/commitTransaction/isActiveTransaction/rollbackTransaction > on ObjectStore.java > > 2013/2/22 Hemanth Yamijala : > > Hi, > > > > We are running into the same problem as well. Is there any clue what > could > > be wrong ? > > > > Thanks > > hemanth > > > > > > On Wed, Feb 6, 2013 at 1:51 AM, James Warren > > wrote: > >> > >> As part of our daily workflow, we're running a few hundred hive > >> queries that are coordinated through oozie. Recently we're > >> encountering issues where on average a job or two fails - and never > >> the same query. The observed error is: > >> > >> FAILED: Error in metadata: java.lang.RuntimeException: > >> commitTransaction was called but openTransactionCalls = 0. This > >> probably indicates that there are unbalanced calls to > >> openTransaction/commitTransaction > >> > >> which I've seen was referred to in HIVE-1760 but patched in 0.7. > >> > >> We're running Hive 0.9 (from CDH 4.1 - will redirect to the Cloudera > >> lists if that is a more appropriate form) - any ideas / suggestions > >> where I should start looking? > >> > >> cheers, > >> -James > > > > >
Re: unbalanced transaction calls
Hi, We are running into the same problem as well. Is there any clue what could be wrong ? Thanks hemanth On Wed, Feb 6, 2013 at 1:51 AM, James Warren < james.war...@stanfordalumni.org> wrote: > As part of our daily workflow, we're running a few hundred hive > queries that are coordinated through oozie. Recently we're > encountering issues where on average a job or two fails - and never > the same query. The observed error is: > > FAILED: Error in metadata: java.lang.RuntimeException: > commitTransaction was called but openTransactionCalls = 0. This > probably indicates that there are unbalanced calls to > openTransaction/commitTransaction > > which I've seen was referred to in HIVE-1760 but patched in 0.7. > > We're running Hive 0.9 (from CDH 4.1 - will redirect to the Cloudera > lists if that is a more appropriate form) - any ideas / suggestions > where I should start looking? > > cheers, > -James >
Re: Can we write output directly to HDFS from Mapper
Can certainly do that. Indeed, if you set the number of reducers to 0, the map output will be directly written to HDFS by the framework itself. You may also want to look at http://hadoop.apache.org/docs/stable/mapred_tutorial.html#Task+Side-Effect+Files to see some things that need to be taken care of if you are writing files on your own. Thanks hemanth On Fri, Sep 28, 2012 at 9:45 AM, Balaraman, Anand wrote: > Hi > > > > In Map-Reduce, is it appropriate to write the output directly to HDFS from > Mapper (without using a reducer) ? > > Are there any adverse effects in doing so or are there any best practices to > be followed in this aspect ? > > > > Comments are much appreciable at the moment J > > > > Thanks and Regards > > Anand B > > Confidential: This electronic message and all contents contain information > from Syntel, Inc. which may be privileged, confidential or otherwise > protected from disclosure. The information is intended to be for the > addressee only. If you are not the addressee, any disclosure, copy, > distribution or use of the contents of this message is prohibited. If you > have received this electronic message in error, please notify the sender > immediately and destroy the original message and all copies.
