Created a JIRA in Hive to track this: https://issues.apache.org/jira/browse/HIVE-12080
On Friday, October 9, 2015 11:12 AM, Mohammad Islam <[email protected]> wrote: Hi Sergio, Looks like I made some progress on this, I will let you know soon. Regards, Mohammad On Friday, October 9, 2015 9:59 AM, Sergio Pena <[email protected]> wrote: Hey Ryan, Mohammad I just left a comment on [email protected] regarding this. It may be possible to do such promotion in Hive. I don't know if Pig does such promotion. Here's the comment left on [email protected]: " I reviewed the HIVE-6784 patch and it has performance penalties because the way is implemented. I think we can do the promotion in a different way to avoid such penalties. If you take a look at ParquetStringInspector.java, this class gets a String value from different string writables (BytesWritable, Text or String). We may do something like that to return a Long from Integer and Short types. However, I am a little worried about the little overhead we will add by checking the writable type with 'instance of' everytime we get the value. I'll review the object inspector and ETypeConverter.java (parquet type coverter) to see if there's a better way to do the promotion. " I'll do my investigation, and see the best way to do it. On Fri, Oct 9, 2015 at 11:27 AM, Ryan Blue <[email protected]> wrote: > Mohammad, > > This is definitely something that I think we should support. Sergio is the > expert on the Hive implementation and can correct me, but I think we should > be able to support this in the SerDe fairly easily by adding promotion > methods to the object inspectors. > > Sergio, what do you think needs to be done here? > > rb > > > On 10/08/2015 06:06 PM, Mohammad Islam wrote: > >> Hi, >> In Hive+parquet, auto type promotion (short->int->bigint) is not >> supported. Few questions: >> 1. Does Parquet support this type of auto type widening? >> 2. is it being supported in pig or any similar project? >> >> >> I'm considering this for Hive. Other data-formats already support this >> in Hive. ORC recently added the support as well. >> I'm not expert in Parquet. >> >> Can someone please give some idea? Some relevant pointers? >> One dev tried it (https://issues.apache.org/jira/browse/HIVE-6784), but >> could not push due to performance ground. >> >> Regards, >> Mohammad >> >> > > -- > Ryan Blue > Software Engineer > Cloudera, Inc. >
