Hi Marcel, I know how to do the actual computation in parallel. My question is how to fetch and instantiate the DataObjects in parallel before I can start the computations. An iterator would only slow down the fetch because of the added roundtrips. Iterators are about reducing memory footprint, while I am not memory-constrained here.
Maik > Am 07.03.2017 um 08:30 schrieb Markus Reich <[email protected]>: > > Hi Maik, > > maybe you can use the new iterator and split the iterator for parallel > computation? > > public static <T> Stream<T> asStream(Iterator<T> sourceIterator, boolean > parallel) { > Iterable<T> iterable = () -> sourceIterator; > return StreamSupport.stream(iterable.spliterator(), parallel); > } > > found at > http://stackoverflow.com/questions/24511052/how-to-convert-an-iterator-to-a-stream > > br > Meex > > Musall, Maik <[email protected]> schrieb am Mo., 6. März 2017 um > 22:25 Uhr: > >> Hi all, >> >> I have a number of statistics functions which need to fetch large amounts >> of objects. I need the actual DataObjects because that's where the business >> logic is that I need for the computations. >> >> Let's say I need to fetch 300.000 objects. Let's also assume the database >> sits on a fast SSD array and can serve multiple connections easily. I'm >> assuming in this case the CPU time needed for DataObject instantiation is >> the main performance constraint. Is that correct? >> >> If so, how can I speed this up? Could I partition my fetch, and fetch in >> several threads in parallel into the same ObjectContext? Or is there an >> easier way to make use of multiple CPU cores for this? >> >> Thanks >> Maik >> >>
