RussellSpitzer commented on a change in pull request #3073:
URL: https://github.com/apache/iceberg/pull/3073#discussion_r703638638
##########
File path: core/src/main/java/org/apache/iceberg/util/BinPacking.java
##########
@@ -169,19 +179,21 @@ public boolean hasNext() {
private static class Bin<T> {
private final long targetWeight;
+ private final int itemsPerBin;
private final List<T> items = Lists.newArrayList();
private long binWeight = 0L;
- Bin(long targetWeight) {
+ Bin(long targetWeight, int itemsPerBin) {
this.targetWeight = targetWeight;
+ this.itemsPerBin = itemsPerBin;
}
List<T> items() {
return items;
}
boolean canAdd(long weight) {
- return binWeight + weight <= targetWeight;
+ return binWeight + weight <= targetWeight && items.size() <= itemsPerBin;
Review comment:
I think there is a slight issue with this implementation and how it
interacts with lookback.
Consider we have a lookback of 3 a max number of items per bin of 2 and a
max value of bin of 10
```
// Add Item (6)
[ 8 ], [3, 4], [4, 3]
// Even if remove largest is set largest bin is [8] but we actually have 2
bins we know we can never add another item to which we should be dropping first.
I think to properly do this we need to also finish off any bins which have
reached their max item size
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
