Hi, HCat 0.5 compatible with Pig 0.10? Are we making that official?
The reason I'm asking is because of this bug: https://issues.apache.org/jira/browse/HCATALOG-451 It's caused because pig does not override and call OutputCommitter.abortJob() (PIG-2712). The bug can and should be fixed in pig, but we may need to fix it in HCatalog to support unfixed versions of pig. I'd rather we not have another workaround and have HCatalog 0.5 support later versions of pig which fixes this bug. Thoughts? -Francis On 7/23/12 9:32 AM, "Travis Crawford" <[email protected]> wrote: >On Mon, Jul 23, 2012 at 8:54 AM, Alan Gates <[email protected]> wrote: > >> The issue before pig 0.10 is that the parser won't allow booleans in >> certain locations. For example, >> >> A = load 'foo'; >> B = foreach A generate mybool; -- where mybool is a boolean >> >> will fail. Half baked supported is worse than no support. In Pig 0.10 >>I >> think your changes make sense. We could declare that HCat 0.5 works wit >> Pig 0.10 and then make these changes on HCat trunk. >> >> >Cool thanks for clearing that up. I like the idea of bumping the required >pig version and simply adding boolean support. I think this means we'll >have some integration challenges with mismatched antlr versions, but we'll >have to address it at some point. > >--travis > > > > >> Alan. >> >> On Jul 20, 2012, at 2:06 PM, Travis Crawford wrote: >> >> > Hey all - >> > >> > I'm taking a look at boolean support and started by making just this >> simple >> > change. Basically just remove the bits that throw exceptions for >>boolean >> > fields. >> > >> > Using pig 0.9 I ran a simple query against a table with booleans and >>it >> > worked just fine. DataType.BOOLEAN already exists (even in pig 0.8), >> > "describe" showed the fields as boolean, and dump printed out >>true/false >> as >> > expected. >> > >> > Am I missing something? The user seems better off by not throwing an >> > exception here because if their pig can handle booleans their query >>runs. >> > >> > >> > TW-MBP13-TCrawford:hcatalog travis$ git diff >> > diff --git >> > >> >>a/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUtil. >>java >> > >> >>b/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUtil. >>java >> > index 696081f..fe686cd 100644 >> > --- >> > >> >>a/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUtil. >>java >> > +++ >> > >> >>b/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUtil. >>java >> > @@ -298,9 +298,7 @@ public class PigHCatUtil { >> > } >> > >> > if (type == Type.BOOLEAN){ >> > *- errMsg = "HCatalog column type 'BOOLEAN' is not supported in >>" + >> > - "Pig as a column type"; >> > - throw new PigException(errMsg, PIG_EXCEPTION_CODE); >> > + return DataType.BOOLEAN; >> > * } >> > >> > errMsg = "HCatalog column type '"+ type.toString() +"' is not >> > supported in Pig as a column type"; >> > @@ -409,7 +407,6 @@ public class PigHCatUtil { >> > // We don't do type promotion/demotion. >> > case SMALLINT: >> > case TINYINT: >> > *- case BOOLEAN: >> > * throw new PigException("Incompatible type found in hcat >>table >> > schema: "+hcatField, PigHCatUtil.PIG_EXCEPTION_CODE); >> > case ARRAY: >> > >> > validateHCatSchemaFollowsPigRules(hcatField.getArrayElementSchema()); >> > TW-MBP13-TCrawford:hcatalog travis$ >> > >> > --travis
