[jira] [Commented] (DATAFU-16) weighted reservoir sampling with exponential jumps UDF
[ https://issues.apache.org/jira/browse/DATAFU-16?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13895791#comment-13895791 ] jian wang commented on DATAFU-16: - Matt, Do you think we go ahead to implement the exponential jump only for the accumulate-based model? And for algebraic, we still use the weighted reservoir sampling without exponential jump. The good part of introducing the exp jump: it could improve the job performance, especially when there is a lot of data to process, without sacrificing much on the sampling precision(per-item sampling probability is close to w/sum(w)). The not good part: the chance of using accumulate-based model may not be as many as algebraic, so is it worthwhile to introduce this enhancement? > weighted reservoir sampling with exponential jumps UDF > -- > > Key: DATAFU-16 > URL: https://issues.apache.org/jira/browse/DATAFU-16 > Project: DataFu > Issue Type: New Feature > Environment: Mac, Linux > pig-0.11 >Reporter: jian wang >Priority: Minor > Attachments: ScoredExpJmpReservoir.java, ScoredReservoir.java, > WeightedSamplingCorrectnessTests.java > > > Create a weightedReservoirSampleWithExpJump UDF to implement the weighted > reservoir sampling algorithm with exponential jumps. Investigation is tracked > in https://github.com/linkedin/datafu/issues/80. This task is part of > experiment of different weighted sampling algorithms. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (DATAFU-28) Tests are too slow
[
https://issues.apache.org/jira/browse/DATAFU-28?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13895817#comment-13895817
]
jian wang commented on DATAFU-28:
-
Matt, do you have stats of individual test cases for
datafu.test.pig.stats.entropy and datafu.test.pig.sampling?
Which ant option or tool do you use to measure the running duration of the test
cases?
> Tests are too slow
> --
>
> Key: DATAFU-28
> URL: https://issues.apache.org/jira/browse/DATAFU-28
> Project: DataFu
> Issue Type: Bug
>Reporter: Matthew Hayes
>
> I ran the tests on my laptop and it took nearly 2 hours.
> The worst offenders are {{datafu.test.pig.sampling}},
> {{datafu.test.pig.stats}}, and {{datafu.test.pig.stats.entropy}}.
> ||Package ||Tests|| Failures|| Duration|| Success rate||
> |datafu.test.pig.bags|27 |0| 1m10.72s|100%|
> |datafu.test.pig.geo |1 |0 |9.757s |100%|
> |datafu.test.pig.hash|4 |0 |41.039s| 100%|
> |datafu.test.pig.linkanalysis|5 |0| 32.677s |100%|
> |datafu.test.pig.random |1| 0| 11.789s|100%|
> |datafu.test.pig.sampling |25|0 |38m25.81s| 100%|
> |datafu.test.pig.sessions |7 |0 |2m50.67s |100%|
> |datafu.test.pig.sets |9 |0 |5m46.70s |100%|
> |datafu.test.pig.stats| 52| 0 |26m11.98s| 100%|
> |datafu.test.pig.stats.entropy|40|0 |31m30.97s |100%|
> |datafu.test.pig.urls|1 |0 |1m35.24s |100%|
> |datafu.test.pig.util|21 |0| 4m51.64s|100%|
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
