It's not really about complexity, it's just about having one too many. I'm trying to minimize our tech debt. Of course if the community thinks we should support earlier versions then I don't feel strongly enough to be against it. It's a good thing the HCat community has a number of Pig committers to chime in? ;-)
-Francis On 7/26/12 4:19 PM, "Rohini Palaniswamy" <[email protected]> wrote: >Francis, > The patch for PIG-2712 will only go into trunk(0.11) because it will >case backward compatibility break with 0.20.2 and pig 0.10 is supported >with 0.20.2. I don't understand why you think it is too much of a >workaround. The proposed fix for 451 is that abortJob() and cleanupJob() >will be doing the same thing till everyone moves to pig 0.11 as pig calls >cleanupJob() instead of abortJob(). > >commitJob() { > commit work - move from temp to final location, register partitions, >update table schema, etc >} > >abortJob(){ > cleanupJob() >} > >cleanupJob() { > delete temp dir >} > >Regards, >Rohini > >On Thu, Jul 26, 2012 at 4:07 PM, Francis Christopher Liu < >[email protected]> wrote: > >> Hi, >> >> HCat 0.5 compatible with Pig 0.10? Are we making that official? >> >> The reason I'm asking is because of this bug: >> >> https://issues.apache.org/jira/browse/HCATALOG-451 >> >> It's caused because pig does not override and call >> OutputCommitter.abortJob() (PIG-2712). The bug can and should be fixed >>in >> pig, but we may need to fix it in HCatalog to support unfixed versions >>of >> pig. I'd rather we not have another workaround and have HCatalog 0.5 >> support later versions of pig which fixes this bug. >> >> Thoughts? >> >> -Francis >> >> >> >> On 7/23/12 9:32 AM, "Travis Crawford" <[email protected]> wrote: >> >> >On Mon, Jul 23, 2012 at 8:54 AM, Alan Gates <[email protected]> >> wrote: >> > >> >> The issue before pig 0.10 is that the parser won't allow booleans in >> >> certain locations. For example, >> >> >> >> A = load 'foo'; >> >> B = foreach A generate mybool; -- where mybool is a boolean >> >> >> >> will fail. Half baked supported is worse than no support. In Pig >>0.10 >> >>I >> >> think your changes make sense. We could declare that HCat 0.5 works >>wit >> >> Pig 0.10 and then make these changes on HCat trunk. >> >> >> >> >> >Cool thanks for clearing that up. I like the idea of bumping the >>required >> >pig version and simply adding boolean support. I think this means we'll >> >have some integration challenges with mismatched antlr versions, but >>we'll >> >have to address it at some point. >> > >> >--travis >> > >> > >> > >> > >> >> Alan. >> >> >> >> On Jul 20, 2012, at 2:06 PM, Travis Crawford wrote: >> >> >> >> > Hey all - >> >> > >> >> > I'm taking a look at boolean support and started by making just >>this >> >> simple >> >> > change. Basically just remove the bits that throw exceptions for >> >>boolean >> >> > fields. >> >> > >> >> > Using pig 0.9 I ran a simple query against a table with booleans >>and >> >>it >> >> > worked just fine. DataType.BOOLEAN already exists (even in pig >>0.8), >> >> > "describe" showed the fields as boolean, and dump printed out >> >>true/false >> >> as >> >> > expected. >> >> > >> >> > Am I missing something? The user seems better off by not throwing >>an >> >> > exception here because if their pig can handle booleans their query >> >>runs. >> >> > >> >> > >> >> > TW-MBP13-TCrawford:hcatalog travis$ git diff >> >> > diff --git >> >> > >> >> >> >>>>a/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUti >>>>l. >> >>java >> >> > >> >> >> >>>>b/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUti >>>>l. >> >>java >> >> > index 696081f..fe686cd 100644 >> >> > --- >> >> > >> >> >> >>>>a/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUti >>>>l. >> >>java >> >> > +++ >> >> > >> >> >> >>>>b/hcatalog-pig-adapter/src/main/java/org/apache/hcatalog/pig/PigHCatUti >>>>l. >> >>java >> >> > @@ -298,9 +298,7 @@ public class PigHCatUtil { >> >> > } >> >> > >> >> > if (type == Type.BOOLEAN){ >> >> > *- errMsg = "HCatalog column type 'BOOLEAN' is not supported >>in >> >>" + >> >> > - "Pig as a column type"; >> >> > - throw new PigException(errMsg, PIG_EXCEPTION_CODE); >> >> > + return DataType.BOOLEAN; >> >> > * } >> >> > >> >> > errMsg = "HCatalog column type '"+ type.toString() +"' is not >> >> > supported in Pig as a column type"; >> >> > @@ -409,7 +407,6 @@ public class PigHCatUtil { >> >> > // We don't do type promotion/demotion. >> >> > case SMALLINT: >> >> > case TINYINT: >> >> > *- case BOOLEAN: >> >> > * throw new PigException("Incompatible type found in hcat >> >>table >> >> > schema: "+hcatField, PigHCatUtil.PIG_EXCEPTION_CODE); >> >> > case ARRAY: >> >> > >> >> > >>validateHCatSchemaFollowsPigRules(hcatField.getArrayElementSchema()); >> >> > TW-MBP13-TCrawford:hcatalog travis$ >> >> > >> >> > --travis >>
