[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743031#comment-13743031 ] Hudson commented on HIVE-5105: -- ABORTED: Integrated in Hive-trunk-hadoop2 #366 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/366/]) HIVE-5105 HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap (Eugene Koifman via Sushanth Sowmyan) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514929) * /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/data/schema/HCatSchema.java * /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/data/schema/TestHCatSchema.java > HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up > fieldPositionMap > - > > Key: HIVE-5105 > URL: https://issues.apache.org/jira/browse/HIVE-5105 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.12.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 0.12.0 > > Attachments: HIVE-5105.patch > > > org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema > hcatFieldSchema) makes the following call: > fieldPositionMap.remove(hcatFieldSchema); > but fieldPositionMap is of type Map so the element is not > getting removed > Here's a detailed comment from [~sushanth] > The result is that that the name will not be removed from fieldPositionMap. > This results in 2 things: > a) If anyone tries to append a field to a hcatschema after having removed > that field, it shouldn't fail, but it will. > b) If anyone asks for the position of the removed field by name, it will > still give the position. > Now, there is only one place in hcat code where we remove a field, and that > is called from HCatOutputFormat.setSchema, where we try to detect if the user > specified partition column names in the schema when they shouldn't have, and > if they did, we remove it. Normally, people do not specify this, and this > check tends to be superfluous. > Once we do this, we wind up serializing that new object (after performing > some validations), and this does appear to stay through the serialization > (and eventual deserialization) which is very worrying. > However, we are luckily saved by the fact that we do not append that field to > it at any time(all appends in hcat code are done on newly initialized > HCatSchema objects which have had no removes done on them), and we don't ask > for the position of something we do not expect to be there(harder to verify > for certain, but seems to be the case on inspection). > The main part that gives me worry is that HCatSchema is part of our public > interface for HCat, in that M/R programs that use HCat can use it, and thus, > they might have more interesting usage patterns that are hitting this bug. > I can't think of any currently open bugs that is caused by this because of > the rarity of the situation, but nevertheless, something we should fix > immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742894#comment-13742894 ] Hudson commented on HIVE-5105: -- FAILURE: Integrated in Hive-trunk-h0.21 #2274 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2274/]) HIVE-5105 HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap (Eugene Koifman via Sushanth Sowmyan) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514929) * /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/data/schema/HCatSchema.java * /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/data/schema/TestHCatSchema.java > HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up > fieldPositionMap > - > > Key: HIVE-5105 > URL: https://issues.apache.org/jira/browse/HIVE-5105 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.12.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 0.12.0 > > Attachments: HIVE-5105.patch > > > org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema > hcatFieldSchema) makes the following call: > fieldPositionMap.remove(hcatFieldSchema); > but fieldPositionMap is of type Map so the element is not > getting removed > Here's a detailed comment from [~sushanth] > The result is that that the name will not be removed from fieldPositionMap. > This results in 2 things: > a) If anyone tries to append a field to a hcatschema after having removed > that field, it shouldn't fail, but it will. > b) If anyone asks for the position of the removed field by name, it will > still give the position. > Now, there is only one place in hcat code where we remove a field, and that > is called from HCatOutputFormat.setSchema, where we try to detect if the user > specified partition column names in the schema when they shouldn't have, and > if they did, we remove it. Normally, people do not specify this, and this > check tends to be superfluous. > Once we do this, we wind up serializing that new object (after performing > some validations), and this does appear to stay through the serialization > (and eventual deserialization) which is very worrying. > However, we are luckily saved by the fact that we do not append that field to > it at any time(all appends in hcat code are done on newly initialized > HCatSchema objects which have had no removes done on them), and we don't ask > for the position of something we do not expect to be there(harder to verify > for certain, but seems to be the case on inspection). > The main part that gives me worry is that HCatSchema is part of our public > interface for HCat, in that M/R programs that use HCat can use it, and thus, > they might have more interesting usage patterns that are hitting this bug. > I can't think of any currently open bugs that is caused by this because of > the rarity of the situation, but nevertheless, something we should fix > immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742877#comment-13742877 ] Hudson commented on HIVE-5105: -- FAILURE: Integrated in Hive-trunk-hadoop1-ptest #131 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/131/]) HIVE-5105 HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap (Eugene Koifman via Sushanth Sowmyan) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514929) * /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/data/schema/HCatSchema.java * /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/data/schema/TestHCatSchema.java > HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up > fieldPositionMap > - > > Key: HIVE-5105 > URL: https://issues.apache.org/jira/browse/HIVE-5105 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.12.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 0.12.0 > > Attachments: HIVE-5105.patch > > > org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema > hcatFieldSchema) makes the following call: > fieldPositionMap.remove(hcatFieldSchema); > but fieldPositionMap is of type Map so the element is not > getting removed > Here's a detailed comment from [~sushanth] > The result is that that the name will not be removed from fieldPositionMap. > This results in 2 things: > a) If anyone tries to append a field to a hcatschema after having removed > that field, it shouldn't fail, but it will. > b) If anyone asks for the position of the removed field by name, it will > still give the position. > Now, there is only one place in hcat code where we remove a field, and that > is called from HCatOutputFormat.setSchema, where we try to detect if the user > specified partition column names in the schema when they shouldn't have, and > if they did, we remove it. Normally, people do not specify this, and this > check tends to be superfluous. > Once we do this, we wind up serializing that new object (after performing > some validations), and this does appear to stay through the serialization > (and eventual deserialization) which is very worrying. > However, we are luckily saved by the fact that we do not append that field to > it at any time(all appends in hcat code are done on newly initialized > HCatSchema objects which have had no removes done on them), and we don't ask > for the position of something we do not expect to be there(harder to verify > for certain, but seems to be the case on inspection). > The main part that gives me worry is that HCatSchema is part of our public > interface for HCat, in that M/R programs that use HCat can use it, and thus, > they might have more interesting usage patterns that are hitting this bug. > I can't think of any currently open bugs that is caused by this because of > the rarity of the situation, but nevertheless, something we should fix > immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742857#comment-13742857 ] Hudson commented on HIVE-5105: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #62 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/62/]) HIVE-5105 HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap (Eugene Koifman via Sushanth Sowmyan) (khorgath: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514929) * /hive/trunk/hcatalog/core/src/main/java/org/apache/hcatalog/data/schema/HCatSchema.java * /hive/trunk/hcatalog/core/src/test/java/org/apache/hcatalog/data/schema/TestHCatSchema.java > HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up > fieldPositionMap > - > > Key: HIVE-5105 > URL: https://issues.apache.org/jira/browse/HIVE-5105 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.12.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 0.12.0 > > Attachments: HIVE-5105.patch > > > org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema > hcatFieldSchema) makes the following call: > fieldPositionMap.remove(hcatFieldSchema); > but fieldPositionMap is of type Map so the element is not > getting removed > Here's a detailed comment from [~sushanth] > The result is that that the name will not be removed from fieldPositionMap. > This results in 2 things: > a) If anyone tries to append a field to a hcatschema after having removed > that field, it shouldn't fail, but it will. > b) If anyone asks for the position of the removed field by name, it will > still give the position. > Now, there is only one place in hcat code where we remove a field, and that > is called from HCatOutputFormat.setSchema, where we try to detect if the user > specified partition column names in the schema when they shouldn't have, and > if they did, we remove it. Normally, people do not specify this, and this > check tends to be superfluous. > Once we do this, we wind up serializing that new object (after performing > some validations), and this does appear to stay through the serialization > (and eventual deserialization) which is very worrying. > However, we are luckily saved by the fact that we do not append that field to > it at any time(all appends in hcat code are done on newly initialized > HCatSchema objects which have had no removes done on them), and we don't ask > for the position of something we do not expect to be there(harder to verify > for certain, but seems to be the case on inspection). > The main part that gives me worry is that HCatSchema is part of our public > interface for HCat, in that M/R programs that use HCat can use it, and thus, > they might have more interesting usage patterns that are hitting this bug. > I can't think of any currently open bugs that is caused by this because of > the rarity of the situation, but nevertheless, something we should fix > immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742763#comment-13742763 ] Sushanth Sowmyan commented on HIVE-5105: Committed to hive svn trunk. > HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up > fieldPositionMap > - > > Key: HIVE-5105 > URL: https://issues.apache.org/jira/browse/HIVE-5105 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.12.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 0.12.0 > > Attachments: HIVE-5105.patch > > > org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema > hcatFieldSchema) makes the following call: > fieldPositionMap.remove(hcatFieldSchema); > but fieldPositionMap is of type Map so the element is not > getting removed > Here's a detailed comment from [~sushanth] > The result is that that the name will not be removed from fieldPositionMap. > This results in 2 things: > a) If anyone tries to append a field to a hcatschema after having removed > that field, it shouldn't fail, but it will. > b) If anyone asks for the position of the removed field by name, it will > still give the position. > Now, there is only one place in hcat code where we remove a field, and that > is called from HCatOutputFormat.setSchema, where we try to detect if the user > specified partition column names in the schema when they shouldn't have, and > if they did, we remove it. Normally, people do not specify this, and this > check tends to be superfluous. > Once we do this, we wind up serializing that new object (after performing > some validations), and this does appear to stay through the serialization > (and eventual deserialization) which is very worrying. > However, we are luckily saved by the fact that we do not append that field to > it at any time(all appends in hcat code are done on newly initialized > HCatSchema objects which have had no removes done on them), and we don't ask > for the position of something we do not expect to be there(harder to verify > for certain, but seems to be the case on inspection). > The main part that gives me worry is that HCatSchema is part of our public > interface for HCat, in that M/R programs that use HCat can use it, and thus, > they might have more interesting usage patterns that are hitting this bug. > I can't think of any currently open bugs that is caused by this because of > the rarity of the situation, but nevertheless, something we should fix > immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[ https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742626#comment-13742626 ] Sushanth Sowmyan commented on HIVE-5105: +1, Thanks for the test as well, Eugene. :) > HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up > fieldPositionMap > - > > Key: HIVE-5105 > URL: https://issues.apache.org/jira/browse/HIVE-5105 > Project: Hive > Issue Type: Bug > Components: HCatalog >Affects Versions: 0.12.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 0.12.0 > > Attachments: HIVE-5105.patch > > > org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema > hcatFieldSchema) makes the following call: > fieldPositionMap.remove(hcatFieldSchema); > but fieldPositionMap is of type Map so the element is not > getting removed > Here's a detailed comment from [~sushanth] > The result is that that the name will not be removed from fieldPositionMap. > This results in 2 things: > a) If anyone tries to append a field to a hcatschema after having removed > that field, it shouldn't fail, but it will. > b) If anyone asks for the position of the removed field by name, it will > still give the position. > Now, there is only one place in hcat code where we remove a field, and that > is called from HCatOutputFormat.setSchema, where we try to detect if the user > specified partition column names in the schema when they shouldn't have, and > if they did, we remove it. Normally, people do not specify this, and this > check tends to be superfluous. > Once we do this, we wind up serializing that new object (after performing > some validations), and this does appear to stay through the serialization > (and eventual deserialization) which is very worrying. > However, we are luckily saved by the fact that we do not append that field to > it at any time(all appends in hcat code are done on newly initialized > HCatSchema objects which have had no removes done on them), and we don't ask > for the position of something we do not expect to be there(harder to verify > for certain, but seems to be the case on inspection). > The main part that gives me worry is that HCatSchema is part of our public > interface for HCat, in that M/R programs that use HCat can use it, and thus, > they might have more interesting usage patterns that are hitting this bug. > I can't think of any currently open bugs that is caused by this because of > the rarity of the situation, but nevertheless, something we should fix > immediately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-5105) HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up fieldPositionMap
[
https://issues.apache.org/jira/browse/HIVE-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741985#comment-13741985
]
Hive QA commented on HIVE-5105:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12598342/HIVE-5105.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 2880 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_parallel_orderby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported2
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/458/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/458/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
> HCatSchema.remove(HCatFieldSchema hcatFieldSchema) does not clean up
> fieldPositionMap
> -
>
> Key: HIVE-5105
> URL: https://issues.apache.org/jira/browse/HIVE-5105
> Project: Hive
> Issue Type: Bug
> Components: HCatalog
>Affects Versions: 0.12.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Fix For: 0.12.0
>
> Attachments: HIVE-5105.patch
>
>
> org.apache.hcatalog.data.schema.HCatSchema.remove(HCatFieldSchema
> hcatFieldSchema) makes the following call:
> fieldPositionMap.remove(hcatFieldSchema);
> but fieldPositionMap is of type Map so the element is not
> getting removed
> Here's a detailed comment from [~sushanth]
> The result is that that the name will not be removed from fieldPositionMap.
> This results in 2 things:
> a) If anyone tries to append a field to a hcatschema after having removed
> that field, it shouldn't fail, but it will.
> b) If anyone asks for the position of the removed field by name, it will
> still give the position.
> Now, there is only one place in hcat code where we remove a field, and that
> is called from HCatOutputFormat.setSchema, where we try to detect if the user
> specified partition column names in the schema when they shouldn't have, and
> if they did, we remove it. Normally, people do not specify this, and this
> check tends to be superfluous.
> Once we do this, we wind up serializing that new object (after performing
> some validations), and this does appear to stay through the serialization
> (and eventual deserialization) which is very worrying.
> However, we are luckily saved by the fact that we do not append that field to
> it at any time(all appends in hcat code are done on newly initialized
> HCatSchema objects which have had no removes done on them), and we don't ask
> for the position of something we do not expect to be there(harder to verify
> for certain, but seems to be the case on inspection).
> The main part that gives me worry is that HCatSchema is part of our public
> interface for HCat, in that M/R programs that use HCat can use it, and thus,
> they might have more interesting usage patterns that are hitting this bug.
> I can't think of any currently open bugs that is caused by this because of
> the rarity of the situation, but nevertheless, something we should fix
> immediately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
