[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169524#comment-16169524 ] Lefty Leverenz commented on HIVE-17344: --- [~kgyrtkirk], please add fix version 3.0.0 to this issue. Thanks. > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.2.patch, HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16159097#comment-16159097 ] Hive QA commented on HIVE-17344: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12886027/HIVE-17344.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11032 tests executed *Failed tests:* {noformat} TestAccumuloCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=230) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=61) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat] (batchId=170) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6733/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6733/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6733/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12886027 - PreCommit-HIVE-Build > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.2.patch, HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16159020#comment-16159020 ] Sergey Shelukhin commented on HIVE-17344: - +1 pending tests > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.2.patch, HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138924#comment-16138924 ] Sergey Shelukhin commented on HIVE-17344: - Yeah sorry it's allocated and reset to 0 or read the same way, so remaining == capacity. Then, it's duplicated (not copied) on read, so the original buffer never changes. Can you add a check with an error that this is the case, we don't want to cache buffers with spurious extra space. Other than that +1 > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135008#comment-16135008 ] Zoltan Haindrich commented on HIVE-17344: - [~sershe] I think it is currently true that remaining == capacity...but the underlying OrcTail resets the position to 0 when it reads explicitly...so if the bb position is not at the beginning remainig() would could underestimate the memory usage...so I think using capacity would be better...because in that case it may only overestimate memory usage...and not under (but there is the case when someone uses windows...) I haven't seen any references where a bigger bb was sliced and passed to this function...if that happens...the capacity of the sliced bb is the old limit...which is ok... I agree that currently capacity/remaining will do the same...but capacity would be more explicit > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135009#comment-16135009 ] Janos Gub commented on HIVE-17344: -- Sorry my bad. So when bytebuffers sliced position and limit will be set in a way that remaining returns the size of the slice. If there are overlapped slices it will be counted twice, but capacity would not help in this case neither. > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134858#comment-16134858 ] Janos Gub commented on HIVE-17344: -- bq. it's either allocated exactly to size If it is allocated to size, then the bb.remaining will be 0 right? bq. read from disk the same way Then here it is 0 also? The weigher in LocalCache sums up the getMemoryUsage of the TailAndFileData. If the remaining size is ALWAYS 0, how HIVE-16133 is limiting the maximum size of the cache? (Or how is it different from setting the maxsize of the cache to limit the number of elements in it) Isn't HIVE-16133 about restricting the total memory usage of cache? cc [~hagleitn] > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133458#comment-16133458 ] Sergey Shelukhin commented on HIVE-17344: - I don't think there's actually a bug since everywhere this buffer is passed in it's either allocated exactly to size, or read from disk the same way. Reads duplicate it so remaining will not change. Looking at capacity is not correct in all cases either, e.g. if someone were to cache multiple slices of the same large buffer, like LLAP allocator does in other places, it would be accounted for multiple times. So, I dunno if this change is needed. Perhaps instead at caching time it should check capacity and remaining are the same. > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131171#comment-16131171 ] Sergey Shelukhin commented on HIVE-17344: - How are those bbs allocated? > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130380#comment-16130380 ] Hive QA commented on HIVE-17344: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12882323/HIVE-17344.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10977 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1] (batchId=240) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7] (batchId=240) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_mapjoin_only] (batchId=170) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6436/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6436/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6436/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12882323 - PreCommit-HIVE-Build > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache have elements MAXWEIGHT/100 of arbitrary size. I think > the correct solution would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17344) LocalCache element memory usage is not calculated properly.
[ https://issues.apache.org/jira/browse/HIVE-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130285#comment-16130285 ] Zoltan Haindrich commented on HIVE-17344: - +1 pending tests > LocalCache element memory usage is not calculated properly. > --- > > Key: HIVE-17344 > URL: https://issues.apache.org/jira/browse/HIVE-17344 > Project: Hive > Issue Type: Bug >Reporter: Janos Gub >Assignee: Janos Gub > Attachments: HIVE-17344.patch > > > Orc footer cache has a calculation of memory usage: > {code:java} > public int getMemoryUsage() { > return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads > (semi-arbitrary). > } > {code} > ByteBuffer.remaining returns the remaining space in the bytebuffer, thus > allowing this cache to grow until MAXWEIGHT/100. I think the correct solution > would be bb.capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)