[jira] [Commented] (IGNITE-10144) Optimize bagging upstream transformer

[Maxim Muzafarov (Jira)]

[ 
https://issues.apache.org/jira/browse/IGNITE-10144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946733#comment-16946733
 ] 

Maxim Muzafarov commented on IGNITE-10144:
--

Moved to 2.9 due to inactivity. Please, feel free to move it back if you will 
be able to complete the ticket by 2.8 code freeze date, December 2, 2019.

> Optimize bagging upstream transformer
> -
>
> Key: IGNITE-10144
> URL: https://issues.apache.org/jira/browse/IGNITE-10144
> Project: Ignite
>  Issue Type: Improvement
>  Components: ml
>Reporter: Artem Malykh
>Assignee: Artem Malykh
>Priority: Minor
> Fix For: 2.8
>
>
> For now BaggingUpstreamTransformer makes upstream sequential to make 
> transformation deterministic. Maybe we should do it other way, for example 
> use mapping of the form (entryIdx, en) -> Stream.generate(() -> en).limit(new 
> PoissonDistribution(Well19937c(entryIdx + seed), ...).sample())



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (IGNITE-10144) Optimize bagging upstream transformer

[Artem Malykh (JIRA)]

[ 
https://issues.apache.org/jira/browse/IGNITE-10144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16703391#comment-16703391
 ] 

Artem Malykh commented on IGNITE-10144:
---

According to series of quick experiments, approach used now 
{code:java}
upstream.sequential().flatMap(en -> Stream.generate(() -> 
en).limit(poisson.sample()))
{code}
 

outperforms following approaches:

1. Using hash code for determinism instead of "sequential" and then using 
parallel

 
{code:java}

PoissonWithUnderlyingRandomAccess p = new PoissonWithUnderlyingRandomAccess(
new Well19937c(123L),
0.3,
PoissonDistribution.DEFAULT_EPSILON,
PoissonDistribution.DEFAULT_MAX_ITERATIONS
upstream
.flatMap(en -> {
p.getRand().setSeed(1234L + en.hashCode());
return Stream.generate(() -> en).limit(p.sample());
}).parallel().count();{code}
2. Zipping upstream with indexes stream and then using index for determinism.

 

> Optimize bagging upstream transformer
> -
>
> Key: IGNITE-10144
> URL: https://issues.apache.org/jira/browse/IGNITE-10144
> Project: Ignite
>  Issue Type: Improvement
>  Components: ml
>Reporter: Artem Malykh
>Assignee: Artem Malykh
>Priority: Minor
> Fix For: 2.8
>
>
> For now BaggingUpstreamTransformer makes upstream sequential to make 
> transformation deterministic. Maybe we should do it other way, for example 
> use mapping of the form (entryIdx, en) -> Stream.generate(() -> en).limit(new 
> PoissonDistribution(Well19937c(entryIdx + seed), ...).sample())



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)