egor-ryashin commented on a change in pull request #7185: Avoid many
unnecessary materializations of collections of 'all segments in cluster'
cardinality
URL: https://github.com/apache/incubator-druid/pull/7185#discussion_r267058115
##########
File path:
server/src/main/java/org/apache/druid/server/coordinator/helper/DruidCoordinatorSegmentInfoLoader.java
##########
@@ -42,8 +43,26 @@ public DruidCoordinatorRuntimeParams
run(DruidCoordinatorRuntimeParams params)
{
log.info("Starting coordination. Getting available segments.");
- // Display info about all available segments
- final Set<DataSegment> availableSegments =
coordinator.getOrderedAvailableDataSegments();
+ // The following transform() call doesn't actually transform the iterable.
It only checks the sizes of the segments
+ // and emits alerts if segments with negative sizes are encountered. In
other words, semantically it's similar to
+ // Stream.peek(). It works as long as
DruidCoordinatorRuntimeParams.createAvailableSegmentsSet() (which is called
+ // below) guarantees to go over the passed iterable exactly once.
+ //noinspection StaticPseudoFunctionalStyleMethod:
https://youtrack.jetbrains.com/issue/IDEA-153047
+ Iterable<DataSegment> availableSegmentsWithSizeChecking =
Iterables.transform(
Review comment:
Probably I missed something, but I cannot see how it can be iterated more
than once.
For reference:
```java
/**
* Returns a fluent iterable that wraps {@code iterable}, or {@code
iterable} itself if it
* is already a {@code FluentIterable}.
*/
public static <E> FluentIterable<E> from(final Iterable<E> iterable) {
return (iterable instanceof FluentIterable) ? (FluentIterable<E>)
iterable
: new FluentIterable<E>(iterable) {
@Override
public Iterator<E> iterator() {
return iterable.iterator();
}
};
}
```
```java
public static <F, T> Iterable<T> transform(final Iterable<F> fromIterable,
final Function<? super F, ? extends T> function) {
checkNotNull(fromIterable);
checkNotNull(function);
return new FluentIterable<T>() {
@Override
public Iterator<T> iterator() {
return Iterators.transform(fromIterable.iterator(), function);
}
};
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
