Hi Aja,
Can you measure the prediction accuracy of the mixture (3)? I suspect its
prediction accuracy is lower than 44%.
Result is as follows.
Mix is highest. And CNN is not 44% but 39.3%. Maybe it is because
Detlef made CNN from KGS games, but I tested by pro games.
Christopher Clark's DCNN is 41.4% for pro games, 44.3% for KGS games.
It seems pro games is more difficult for DCNN.
Aya and CNN accuracy is similar, but CNN cumulative accuracy is much
better.
Detlef's CNN uses only black and white, two 19x19 binary. So I think
Aya's tactical features(string capture, ko threat) may help.
From pro 4000 games
prediction accuracy
Aya policy 38.8%
CNN policy 39.3%
Mixture 44.1%
Cumulative accuracy
move rank Aya CNN Mixture
1 38.8 39.3 44.1
2 51.1 53.6 57.9
3 57.8 62.0 65.7
4 62.5 67.7 70.8
5 65.9 71.9 74.6
6 68.8 75.2 77.7
7 71.1 78.0 80.3
8 73.2 80.2 82.3
9 74.8 82.1 84.0
10 76.3 84.0 85.4
11 77.6 85.4 86.7
12 78.9 86.7 88.0
13 80.0 87.7 89.0
14 81.0 88.7 89.8
15 82.0 89.6 90.6
16 82.9 90.3 91.3
17 83.7 91.0 91.9
18 84.4 91.6 92.5
19 85.1 92.2 93.0
20 85.8 92.7 93.5
Regards,
Hiroshi Yamashita
----- Original Message -----
From: "Aja Huang" <ajahu...@google.com>
To: <computer-go@computer-go.org>
Sent: Saturday, October 10, 2015 3:44 AM
Subject: Re: [Computer-go] Detlef's DCNN data
Hi Hiroshi,
On Fri, Oct 9, 2015 at 2:10 PM, Hiroshi Yamashita <y...@bd.mbn.or.jp> wrote:
Mix MM gamma with CNN is clery stronger. But using only CNN is still
stronger than Aya without CNN. I had thought too small CNN's
probability was bad for Mix, but it is not.
Aya with CNN vs Aya without CNN, 10000 playouts/move
winrate wins/games min p
0.956 284/297 (p < 0.0001 ) p*=50 Mix MM gamma with CNN's p
0.914 85/ 93 (p < 0.001 ) p*=50 Mix MM gamma with CNN's p
0.923 275/298 (p < 0.01 ) p*=50 Mix MM gamma with CNN's p
0.570 158/277 (p < 0.00001) p*=1 Use only CNN's p
0.720 206/286 (p < 0.00001) p*=10 Use only CNN's p
0.777 226/286 (p < 0.00001) p*=100 Use only CNN's p
Thanks for the results. Very interesting. I still don't understand how can
possibly the mixture can do better than CNN alone since CNN policy is much
stronger than Aya policy.
So now, you have three different policies:
1. Aya policy pi_aya, prediction accuracy is 35% if I remember correctly.
2. CNN policy pi_cnn, prediction accuracy is 44%.
3. Mixture of pi_aya and pi_cnn.
Can you measure the prediction accuracy of the mixture (3)? I suspect its
prediction accuracy is lower than 44%.
Aja
_______________________________________________
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go
