[MediaWiki-commits] [Gerrit] wikimedia...wmf[master]: db1047 => db1108

Bearloga (Code Review) Mon, 13 Nov 2017 11:12:47 -0800

Bearloga has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/391063 )


Change subject: db1047 => db1108
......................................................................

db1047 => db1108

Bug: T156844
Change-Id: I81f0f93a97f7467e1fcf30e20c252fc044bbbd31
---
M DESCRIPTION
M NEWS.md
M R/mysql.R
M man/mysql.Rd
A vignettes/interleaved.R
A vignettes/interleaved.html
6 files changed, 475 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/wikimedia/discovery/wmf 
refs/changes/63/391063/1

diff --git a/DESCRIPTION b/DESCRIPTION
index d0d3314..57e4d2f 100644
--- a/DESCRIPTION
+++ b/DESCRIPTION
@@ -1,8 +1,8 @@
 Package: wmf
 Type: Package
 Title: R Code for Wikimedia Foundation Internal Usage
-Version: 0.3.0
-Date: 2017-11-01
+Version: 0.3.1
+Date: 2017-11-13
 Authors@R: c(
     person("Mikhail", "Popov", email = "mikh...@wikimedia.org", role = 
c("aut", "cre")),
     person("Oliver", "Keyes", role = "aut", comment = "No longer employed at 
the Foundation"),
diff --git a/NEWS.md b/NEWS.md
index c3927ce..dfd22b8 100644
--- a/NEWS.md
+++ b/NEWS.md
@@ -1,3 +1,7 @@
+wmf 0.3.1
+=========
+* Switched host name from db1047.eqiad.wmnet to db1108.eqiad.wmnet per 
[T156844](https://phabricator.wikimedia.org/T156844)
+
 wmf 0.3.0
 =========
 * C++-based `exact_binomial()` to quickly estimate sample size for exact 
binomial tests
diff --git a/R/mysql.R b/R/mysql.R
index 4594a36..725b8c4 100644
--- a/R/mysql.R
+++ b/R/mysql.R
@@ -33,7 +33,7 @@
 #' @export
 mysql_connect <- function(
   database, default_file = NULL,
-  hostname = ifelse(database == "log", "db1047.eqiad.wmnet", 
"analytics-store.eqiad.wmnet")
+  hostname = ifelse(database == "log", "db1108.eqiad.wmnet", 
"analytics-store.eqiad.wmnet")
 ) {
   # Begin Exclude Linting
   if (is.null(default_file)) {
diff --git a/man/mysql.Rd b/man/mysql.Rd
index 09c9606..2032aa8 100644
--- a/man/mysql.Rd
+++ b/man/mysql.Rd
@@ -11,7 +11,7 @@
 \title{Work with MySQL databases}
 \usage{
 mysql_connect(database, default_file = NULL, hostname = ifelse(database ==
-  "log", "db1047.eqiad.wmnet", "analytics-store.eqiad.wmnet"))
+  "log", "db1108.eqiad.wmnet", "analytics-store.eqiad.wmnet"))
 
 mysql_read(query, database, con = NULL)
 
diff --git a/vignettes/interleaved.R b/vignettes/interleaved.R
new file mode 100644
index 0000000..d903b20
--- /dev/null
+++ b/vignettes/interleaved.R
@@ -0,0 +1,38 @@
+## ---- echo=FALSE---------------------------------------------------------
+set.seed(0)
+
+## ------------------------------------------------------------------------
+data(interleaved_data, package = "wmf") # no preference
+data(interleaved_data_a, package = "wmf") # preference for A
+data(interleaved_data_b, package = "wmf") # preference for B
+
+## ---- results='asis'-----------------------------------------------------
+knitr::kable(head(interleaved_data_b))
+
+## ------------------------------------------------------------------------
+library(wmf)
+
+## ----no_pref-------------------------------------------------------------
+x <- interleaved_data[interleaved_data$event == "click", ]
+x <- x[order(x$session_id, x$timestamp), ]
+boot_x <- interleaved_bootstraps(x$session_id, x$ranking_function)
+hist(boot_x, col = "gray70", border = NA, main = "No preference", xlab = 
"Bootstrapped preferences")
+abline(v = quantile(boot_x, c(0.025, 0.975)), lty = "dashed")
+abline(v = interleaved_preference(x$session_id, x$ranking_function), lwd = 2)
+
+## ----a_pref--------------------------------------------------------------
+y <- interleaved_data_a[interleaved_data_a$event == "click", ]
+y <- y[order(y$session_id, y$timestamp), ]
+boot_y <- interleaved_bootstraps(y$session_id, y$ranking_function)
+hist(boot_y, col = "gray70", border = NA, main = "Preference for A", xlab = 
"Bootstrapped preferences")
+abline(v = quantile(boot_y, c(0.025, 0.975)), lty = "dashed")
+abline(v = interleaved_preference(y$session_id, y$ranking_function), lwd = 2)
+
+## ----b_pref--------------------------------------------------------------
+z <- interleaved_data_b[interleaved_data_b$event == "click", ]
+z <- z[order(z$session_id, z$timestamp), ]
+boot_z <- interleaved_bootstraps(z$session_id, z$ranking_function)
+hist(boot_z, col = "gray70", border = NA, main = "Preference for B", xlab = 
"Bootstrapped preferences")
+abline(v = quantile(boot_z, c(0.025, 0.975)), lty = "dashed")
+abline(v = interleaved_preference(z$session_id, z$ranking_function), lwd = 2)
+
diff --git a/vignettes/interleaved.html b/vignettes/interleaved.html
new file mode 100644
index 0000000..2e4d725
--- /dev/null
+++ b/vignettes/interleaved.html
@@ -0,0 +1,429 @@
+<!DOCTYPE html>
+
+<html xmlns="http://www.w3.org/1999/xhtml";>
+
+<head>
+
+<meta charset="utf-8" />
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<meta name="generator" content="pandoc" />
+
+<meta name="viewport" content="width=device-width, initial-scale=1">
+
+<meta name="author" content="Mikhail Popov" />
+
+<meta name="date" content="2017-11-13" />
+
+<title>Estimating Preference For Ranking Functions With Clicks On Interleaved 
Search Results</title>
+
+
+
+<style type="text/css">code{white-space: pre;}</style>
+<style type="text/css">
+div.sourceLine, a.sourceLine { display: inline-block; min-height: 1.25em; }
+a.sourceLine { pointer-events: none; color: inherit; text-decoration: inherit; 
}
+.sourceCode { overflow: visible; }
+code.sourceCode { white-space: pre; }
+@media print {
+code.sourceCode { white-space: pre-wrap; }
+div.sourceLine, a.sourceLine { text-indent: -1em; padding-left: 1em; }
+}
+pre.numberSource div.sourceLine, .numberSource a.sourceLine
+  { position: relative; }
+pre.numberSource div.sourceLine::before, .numberSource a.sourceLine::before
+  { content: attr(data-line-number);
+    position: absolute; left: -5em; text-align: right; vertical-align: 
baseline;
+    border: none; pointer-events: all;
+    -webkit-touch-callout: none; -webkit-user-select: none;
+    -khtml-user-select: none; -moz-user-select: none;
+    -ms-user-select: none; user-select: none;
+    padding: 0 4px; width: 4em; }
+pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; color: 
#aaaaaa;  padding-left: 4px; }
+@media screen {
+a.sourceLine::before { text-decoration: underline; color: initial; }
+}
+code span.kw { color: #007020; font-weight: bold; } /* Keyword */
+code span.dt { color: #902000; } /* DataType */
+code span.dv { color: #40a070; } /* DecVal */
+code span.bn { color: #40a070; } /* BaseN */
+code span.fl { color: #40a070; } /* Float */
+code span.ch { color: #4070a0; } /* Char */
+code span.st { color: #4070a0; } /* String */
+code span.co { color: #60a0b0; font-style: italic; } /* Comment */
+code span.ot { color: #007020; } /* Other */
+code span.al { color: #ff0000; font-weight: bold; } /* Alert */
+code span.fu { color: #06287e; } /* Function */
+code span.er { color: #ff0000; font-weight: bold; } /* Error */
+code span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* 
Warning */
+code span.cn { color: #880000; } /* Constant */
+code span.sc { color: #4070a0; } /* SpecialChar */
+code span.vs { color: #4070a0; } /* VerbatimString */
+code span.ss { color: #bb6688; } /* SpecialString */
+code span.im { } /* Import */
+code span.va { color: #19177c; } /* Variable */
+code span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
+code span.op { color: #666666; } /* Operator */
+code span.bu { } /* BuiltIn */
+code span.ex { } /* Extension */
+code span.pp { color: #bc7a00; } /* Preprocessor */
+code span.at { color: #7d9029; } /* Attribute */
+code span.do { color: #ba2121; font-style: italic; } /* Documentation */
+code span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* 
Annotation */
+code span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* 
CommentVar */
+code span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* 
Information */
+</style>
+
+
+
+<style type="text/css">body {
+background-color: #fff;
+margin: 1em auto;
+max-width: 700px;
+overflow: visible;
+padding-left: 2em;
+padding-right: 2em;
+font-family: "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
+font-size: 14px;
+line-height: 1.35;
+}
+#header {
+text-align: center;
+}
+#TOC {
+clear: both;
+margin: 0 0 10px 10px;
+padding: 4px;
+width: 400px;
+border: 1px solid #CCCCCC;
+border-radius: 5px;
+background-color: #f6f6f6;
+font-size: 13px;
+line-height: 1.3;
+}
+#TOC .toctitle {
+font-weight: bold;
+font-size: 15px;
+margin-left: 5px;
+}
+#TOC ul {
+padding-left: 40px;
+margin-left: -1.5em;
+margin-top: 5px;
+margin-bottom: 5px;
+}
+#TOC ul ul {
+margin-left: -2em;
+}
+#TOC li {
+line-height: 16px;
+}
+table {
+margin: 1em auto;
+border-width: 1px;
+border-color: #DDDDDD;
+border-style: outset;
+border-collapse: collapse;
+}
+table th {
+border-width: 2px;
+padding: 5px;
+border-style: inset;
+}
+table td {
+border-width: 1px;
+border-style: inset;
+line-height: 18px;
+padding: 5px 5px;
+}
+table, table th, table td {
+border-left-style: none;
+border-right-style: none;
+}
+table thead, table tr.even {
+background-color: #f7f7f7;
+}
+p {
+margin: 0.5em 0;
+}
+blockquote {
+background-color: #f6f6f6;
+padding: 0.25em 0.75em;
+}
+hr {
+border-style: solid;
+border: none;
+border-top: 1px solid #777;
+margin: 28px 0;
+}
+dl {
+margin-left: 0;
+}
+dl dd {
+margin-bottom: 13px;
+margin-left: 13px;
+}
+dl dt {
+font-weight: bold;
+}
+ul {
+margin-top: 0;
+}
+ul li {
+list-style: circle outside;
+}
+ul ul {
+margin-bottom: 0;
+}
+pre, code {
+background-color: #f7f7f7;
+border-radius: 3px;
+color: #333;
+white-space: pre-wrap; 
+}
+pre {
+border-radius: 3px;
+margin: 5px 0px 10px 0px;
+padding: 10px;
+}
+pre:not([class]) {
+background-color: #f7f7f7;
+}
+code {
+font-family: Consolas, Monaco, 'Courier New', monospace;
+font-size: 85%;
+}
+p > code, li > code {
+padding: 2px 0px;
+}
+div.figure {
+text-align: center;
+}
+img {
+background-color: #FFFFFF;
+padding: 2px;
+border: 1px solid #DDDDDD;
+border-radius: 3px;
+border: 1px solid #CCCCCC;
+margin: 0 5px;
+}
+h1 {
+margin-top: 0;
+font-size: 35px;
+line-height: 40px;
+}
+h2 {
+border-bottom: 4px solid #f7f7f7;
+padding-top: 10px;
+padding-bottom: 2px;
+font-size: 145%;
+}
+h3 {
+border-bottom: 2px solid #f7f7f7;
+padding-top: 10px;
+font-size: 120%;
+}
+h4 {
+border-bottom: 1px solid #f7f7f7;
+margin-left: 8px;
+font-size: 105%;
+}
+h5, h6 {
+border-bottom: 1px solid #ccc;
+font-size: 105%;
+}
+a {
+color: #0033dd;
+text-decoration: none;
+}
+a:hover {
+color: #6666ff; }
+a:visited {
+color: #800080; }
+a:visited:hover {
+color: #BB00BB; }
+a[href^="http:"] {
+text-decoration: underline; }
+a[href^="https:"] {
+text-decoration: underline; }
+
+code > span.kw { color: #555; font-weight: bold; } 
+code > span.dt { color: #902000; } 
+code > span.dv { color: #40a070; } 
+code > span.bn { color: #d14; } 
+code > span.fl { color: #d14; } 
+code > span.ch { color: #d14; } 
+code > span.st { color: #d14; } 
+code > span.co { color: #888888; font-style: italic; } 
+code > span.ot { color: #007020; } 
+code > span.al { color: #ff0000; font-weight: bold; } 
+code > span.fu { color: #900; font-weight: bold; }  code > span.er { color: 
#a61717; background-color: #e3d2d2; } 
+</style>
+
+</head>
+
+<body>
+
+
+
+
+<h1 class="title toc-ignore">Estimating Preference For Ranking Functions With 
Clicks On Interleaved Search Results</h1>
+<h4 class="author"><em>Mikhail Popov</em></h4>
+<h4 class="date"><em>2017-11-13</em></h4>
+
+
+
+<section id="introduction" class="level2">
+<h2>Introduction</h2>
+<p>The way Search and Discovery’s Analysts have been assessing changes to 
search has so traditionally relied on A/B testing wherein the control group 
receives results using the latest configuration and the test group (or groups) 
receives results using the experimental configuration. Another way to evaluate 
the user-perceived relevance of search results from the experimental 
configuration relies on a technique called <em>interleaving</em>. In it, each 
user is their own baseline – we perform two searches behind the scenes and then 
interleave them together into a single set of results using the team draft 
algorithm described by Chapelle et al. (2012):</p>
+<ol type="1">
+<li><strong>Input</strong>: result sets <span class="math inline">\(A\)</span> 
and <span class="math inline">\(B\)</span>.</li>
+<li><strong>Initialize</strong>: an empty interleaved result sets <span 
class="math inline">\(I\)</span> and drafts <span class="math inline">\(T_A, 
T_B\)</span> for keeping track of which results belong to which team.</li>
+<li>For each round of picking:
+<ol type="a">
+<li>Randomly decide whether we first pick from <span class="math 
inline">\(A\)</span> or from <span class="math inline">\(B\)</span>.</li>
+<li>Without loss of generality, if <span class="math inline">\(A\)</span> is 
randomly chosen to go first, grab top result <span class="math inline">\(a \in 
A\)</span>, append it to <span class="math inline">\(I\)</span> and <span 
class="math inline">\(T_A\)</span>: <span class="math inline">\(I \gets a, T_A 
\gets a\)</span>.</li>
+<li>Take the top result <span class="math inline">\(b \in B\)</span> such that 
<span class="math inline">\(b \neq a\)</span> and append it to <span 
class="math inline">\(I\)</span> after <span class="math inline">\(a\)</span> 
and to <span class="math inline">\(T_B\)</span>: <span class="math inline">\(I 
\gets b, T_B \gets b\)</span>.</li>
+<li>Update <span class="math inline">\(A = A \setminus \{a, b\}\)</span> and 
<span class="math inline">\(B \setminus \{a, b\}\)</span> so the two results 
that were just appended to <span class="math inline">\(I\)</span> are not 
considered again.</li>
+<li>Stop when <span class="math inline">\(|I| = \text{maximum per 
page}\)</span>, so only the first page contains interleaved results.</li>
+</ol></li>
+<li><strong>Output</strong>: interleaved results <span class="math 
inline">\(I\)</span> and team drafts <span class="math inline">\(T_A, 
T_B\)</span>.</li>
+</ol>
+<p>By keeping track of which results belong to which ranking function when the 
user clicks on them, we can estimate a preference for one ranker over the 
other. The preference statistic <span class="math 
inline">\(\Delta_{AB}\)</span> is described by Chapelle et al. as</p>
+<p><span class="math display">\[
+\Delta_{AB} = \frac{\text{wins}_A + \frac{1}{2} \text{ties}}{\text{wins}_A + 
\text{wins}_B + \text{ties}} - 0.5,
+\]</span></p>
+<p>where wins are calculated by counting clicks on the results from teams “A” 
and “B”. A positive value of <span class="math inline">\(\Delta_{AB}\)</span> 
indicates that <span class="math inline">\(A \succ B\)</span>, a negative value 
indicates that <span class="math inline">\(B \succ A\)</span>. We performed two 
types of calculations: per-session and per-search. In 
<strong>per-session</strong>, “A” has won if there are more clicks on team “A” 
results than team “B” results and <span class="math 
inline">\(\text{wins}_A\)</span> is incremented by one for each such session. 
In <strong>per-search</strong>, “A” has won if there are more clicks on team 
“A” results in each search, thus any one session can contribute multiple points 
to the overall <span class="math inline">\(\text{wins}_A\)</span>.</p>
+<p>In order to obtain confidence intervals for the preference statistic, we 
utilize <a 
href="https://en.wikipedia.org/wiki/Bootstrapping_(statistics)">bootstrapping</a>
 with <span class="math inline">\(m\)</span> iterations.</p>
+<ol type="1">
+<li>For bootstrap iteration <span class="math inline">\(i = 1, \ldots, 
m\)</span>:
+<ol type="a">
+<li>Sample unique IDs with replacement.</li>
+<li>Calculate <span class="math inline">\(\Delta_{AB}^{(i)}\)</span> from new 
data.</li>
+</ol></li>
+<li>The confidence intervals (CIs) are calculated by finding percentiles of 
the distribution of bootstrapped preferences <span class="math 
inline">\(\{\Delta_{AB}^{(1)}, \ldots, \Delta_{AB}^{(m)}\}\)</span> – e.g. the 
2.5th and 97.5th percentiles for a 95% CI.</li>
+</ol>
+</section>
+<section id="simulated-data" class="level2">
+<h2>Simulated Data</h2>
+<p>This package provides simulated search and click data. The three built-in 
datasets have simulated users that (1) exhibit no preference, (2) exhibit 
preference for the ranking function “A”, and (3) exhibit preference for the 
ranking function “B”.</p>
+<pre class="sourceCode r" id="cb1"><code class="sourceCode r"><div 
class="sourceLine" id="cb1-1" data-line-number="1"><span 
class="kw">data</span>(interleaved_data, <span class="dt">package =</span> 
<span class="st">&quot;wmf&quot;</span>) <span class="co"># no 
preference</span></div>
+<div class="sourceLine" id="cb1-2" data-line-number="2"><span 
class="kw">data</span>(interleaved_data_a, <span class="dt">package =</span> 
<span class="st">&quot;wmf&quot;</span>) <span class="co"># preference for 
A</span></div>
+<div class="sourceLine" id="cb1-3" data-line-number="3"><span 
class="kw">data</span>(interleaved_data_b, <span class="dt">package =</span> 
<span class="st">&quot;wmf&quot;</span>) <span class="co"># preference for 
B</span></div></code></pre>
+<p>Here are the first few rows of the third dataset:</p>
+<pre class="sourceCode r" id="cb2"><code class="sourceCode r"><div 
class="sourceLine" id="cb2-1" data-line-number="1">knitr<span 
class="op">::</span><span class="kw">kable</span>(<span 
class="kw">head</span>(interleaved_data_b))</div></code></pre>
+<table>
+<thead>
+<tr class="header">
+<th style="text-align: left;">session_id</th>
+<th style="text-align: left;">timestamp</th>
+<th style="text-align: left;">event</th>
+<th style="text-align: right;">position</th>
+<th style="text-align: left;">ranking_function</th>
+</tr>
+</thead>
+<tbody>
+<tr class="odd">
+<td style="text-align: left;">rusjmp29aw</td>
+<td style="text-align: left;">2018-08-01 00:00:47</td>
+<td style="text-align: left;">serp</td>
+<td style="text-align: right;">NA</td>
+<td style="text-align: left;">NA</td>
+</tr>
+<tr class="even">
+<td style="text-align: left;">rusjmp29aw</td>
+<td style="text-align: left;">2018-08-01 00:05:18</td>
+<td style="text-align: left;">click</td>
+<td style="text-align: right;">18</td>
+<td style="text-align: left;">B</td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;">gdkhnfo5ts</td>
+<td style="text-align: left;">2018-08-01 00:01:14</td>
+<td style="text-align: left;">serp</td>
+<td style="text-align: right;">NA</td>
+<td style="text-align: left;">NA</td>
+</tr>
+<tr class="even">
+<td style="text-align: left;">gdkhnfo5ts</td>
+<td style="text-align: left;">2018-08-01 00:02:44</td>
+<td style="text-align: left;">click</td>
+<td style="text-align: right;">7</td>
+<td style="text-align: left;">B</td>
+</tr>
+<tr class="odd">
+<td style="text-align: left;">gdkhnfo5ts</td>
+<td style="text-align: left;">2018-08-01 00:03:13</td>
+<td style="text-align: left;">click</td>
+<td style="text-align: right;">11</td>
+<td style="text-align: left;">B</td>
+</tr>
+<tr class="even">
+<td style="text-align: left;">gdkhnfo5ts</td>
+<td style="text-align: left;">2018-08-01 00:03:42</td>
+<td style="text-align: left;">click</td>
+<td style="text-align: right;">14</td>
+<td style="text-align: left;">A</td>
+</tr>
+</tbody>
+</table>
+</section>
+<section id="estimation" class="level2">
+<h2>Estimation</h2>
+<pre class="sourceCode r" id="cb3"><code class="sourceCode r"><div 
class="sourceLine" id="cb3-1" data-line-number="1"><span 
class="kw">library</span>(wmf)</div></code></pre>
+<p>To calculate <span class="math inline">\(\Delta_{AB}\)</span> with 
<code>interleaved_preference</code>, we will need to use the clicks. We also 
use bootstrapping via <code>interleaved_bootstraps</code> which resamples 
sessions (with replacement) to obtain a distribution of the preference 
statistic <span class="math inline">\(\Delta_{AB}\)</span>. After we plot each 
bootstrapped sample, we mark the 95% confidence interval bounds. 
<strong>Note</strong> that <code>interleaved_confint</code> outputs the 
<code>quantile</code>-based CI and uses the same bootstrap function 
internally.</p>
+<section id="no-preference" class="level3">
+<h3>No preference</h3>
+<p>When users click on the interleaved results <em>without</em> a preference, 
the resulting preference statistic is close to 0 and the confidence interval 
covers 0:</p>
+<pre class="sourceCode r" id="cb4"><code class="sourceCode r"><div 
class="sourceLine" id="cb4-1" data-line-number="1">x &lt;-<span class="st"> 
</span>interleaved_data[interleaved_data<span class="op">$</span>event <span 
class="op">==</span><span class="st"> &quot;click&quot;</span>, ]</div>
+<div class="sourceLine" id="cb4-2" data-line-number="2">x &lt;-<span 
class="st"> </span>x[<span class="kw">order</span>(x<span 
class="op">$</span>session_id, x<span class="op">$</span>timestamp), ]</div>
+<div class="sourceLine" id="cb4-3" data-line-number="3">boot_x &lt;-<span 
class="st"> </span><span class="kw">interleaved_bootstraps</span>(x<span 
class="op">$</span>session_id, x<span class="op">$</span>ranking_function)</div>
+<div class="sourceLine" id="cb4-4" data-line-number="4"><span 
class="kw">hist</span>(boot_x, <span class="dt">col =</span> <span 
class="st">&quot;gray70&quot;</span>, <span class="dt">border =</span> <span 
class="ot">NA</span>, <span class="dt">main =</span> <span class="st">&quot;No 
preference&quot;</span>, <span class="dt">xlab =</span> <span 
class="st">&quot;Bootstrapped preferences&quot;</span>)</div>
+<div class="sourceLine" id="cb4-5" data-line-number="5"><span 
class="kw">abline</span>(<span class="dt">v =</span> <span 
class="kw">quantile</span>(boot_x, <span class="kw">c</span>(<span 
class="fl">0.025</span>, <span class="fl">0.975</span>)), <span class="dt">lty 
=</span> <span class="st">&quot;dashed&quot;</span>)</div>
+<div class="sourceLine" id="cb4-6" data-line-number="6"><span 
class="kw">abline</span>(<span class="dt">v =</span> <span 
class="kw">interleaved_preference</span>(x<span class="op">$</span>session_id, 
x<span class="op">$</span>ranking_function), <span class="dt">lwd =</span> 
<span class="dv">2</span>)</div></code></pre>
+<p><img 
src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASAAAAEgCAYAAAAUg66AAAAEDWlDQ1BJQ0MgUHJvZmlsZQAAOI2NVV1oHFUUPrtzZyMkzlNsNIV0qD8NJQ2TVjShtLp/3d02bpZJNtoi6GT27s6Yyc44M7v9oU9FUHwx6psUxL+3gCAo9Q/bPrQvlQol2tQgKD60+INQ6Ium65k7M5lpurHeZe58853vnnvuuWfvBei5qliWkRQBFpquLRcy4nOHj4g9K5CEh6AXBqFXUR0rXalMAjZPC3e1W99Dwntf2dXd/p+tt0YdFSBxH2Kz5qgLiI8B8KdVy3YBevqRHz/qWh72Yui3MUDEL3q44WPXw3M+fo1pZuQs4tOIBVVTaoiXEI/MxfhGDPsxsNZfoE1q66ro5aJim3XdoLFw72H+n23BaIXzbcOnz5mfPoTvYVz7KzUl5+FRxEuqkp9G/Ajia219thzg25abkRE/BpDc3pqvphHvRFys2weqvp+krbWKIX7nhDbzLOItiM8358pTwdirqpPFnMF2xLc1WvLyOwTAibpbmvHHcvttU57y5+XqNZrLe3lE/Pq8eUj2fXKfOe3pfOjzhJYtB/yll5SDFcSDiH+hRkH25+L+sdxKEAMZahrlSX8ukqMOWy/jXW2m6M9LDBc31B9LFuv6gVKg/0Szi3KAr1kGq1GMjU/aLbnq6/lRxc4XfJ98hTargX++DbMJBSiYMIe9Ck1YAxFkKEAG3xbYaKmDDgYyFK0UGYpfoWYXG+fAPPI6tJnNwb7ClP7IyF+D+bjOtCpkhz6CFrIa/I6sFtNl8auFXGMTP34sNwI/JhkgEtmDz14ySfaRcTIBInmKPE32kxyyE2Tv+thKbEVePDfW/byMM1Kmm0XdObS7oGD/MypMXFPXrCwOtoYjyyn7BV29/MZfsVzpLDdRtuIZnbpXzvlf+ev8MvYr/Gqk4H/kV/G3csdazLuyTMPsbFhzd1UabQbjFvDRmcWJxR3zcfHkVw9GfpbJmeev9F08WW8uDkaslwX6avlWGU6NRKz0g/SHtCy9J30o/ca9zX3Kfc19zn3BXQKRO8ud477hLnAfc1/G9mrzGlrfexZ5GLdn6ZZrrEohI2wVHhZywjbhUWEy8icMCGNCUdiBlq3r+xafL549HQ5jH+an+1y+LlYBifuxAvRN/lVVVOlwlCkdVm9NOL5BE4wkQ2SMlDZU97hX86EilU/lUmkQUztTE6mx1EEPh7OmdqBtAvv8HdWpbrJS6tJj3n0CWdM6busNzRV3S9KTYhqvNiqWmuroiKgYhshMjmhTh9ptWhsF7970j/SbMrsPE1suR5z7DMC+P/Hs+y7ijrQAlhyAgccjbhjPygfeBTjzhNqy28EdkUh8C+DU9+z2v/oyeH791OncxHOs5y2AtTc7nb/f73TWPkD/qwBnjX8BoJ98VVBg/m8AADOjSURBVHgB7Z0HuNTE4vaHItJ7VXoV6QgoSBcpiggWiijlchUsKGLBKzYuyKMigqhYLyjIBQUEAZEmHUVBUFC4CEgXEekdhPnmnc/kn92T3eScLdnsvvM8e04ymfqb5M20zGSSyggaEiABEvCAQGYP4mSUJEACJKAJUIB4I5AACXhGgALkGXpGTAIkQAHiPUACJOAZAQqQZ+gZMQmQAAWI9wAJkIBnBChAnqFnxCRAAhQg3gMkQAKeEaAAeYaeEZMACVCAeA+QAAl4RoAC5Bl6RkwCJEAB4j1AAiTgGQEKkGfoGTEJkAAFiPcACZCAZwQoQJ6hZ8QkQAIUIN4DJEACnhGgAHmGnhGTAAlQgHgPkAAJeEaAAuQZekZMAiRAAeI9QAIk4BkBCpBn6BkxCZAABYj3AAmQgGcEKECeoWfEJEACFCDeAyRAAp4RoAB5hp4RkwAJUIB4D5AACXhGgALkGXpGTAIkQAHiPUACJOAZAQqQZ+gZMQmQAAWI9wAJkIBnBChAnqFnxCRAAhQg3gMkQAKeEaAAeYY+bcTHjx8XLVu21L9evXoJKWUaRwMHDtTX+/Tpk+aaXy3Wrl0rbr75ZlGsWDFRoUIFcffdd/s1K0x3OglkTad7Oo8hgQsXLoglS5aYMTRp0kQEC8369evF0qVLxcGDB013fj44ceKEaN26tThy5IjOxh9//CHKlSvn5ywx7ekgwBpQOmDF2+mgQYPEoUOH4h1tXOP75ZdfTPHp3bu32L9/v5g4cWJc08DIvCNAAfKOvWPMEB+IUHrNxYsXxY4dOwT+R2Lgf9euXbZNweBwjx07JlCbCTZoRu7cuVOcPn06+JI+37dvn2nfrl07Ubx4cVGiRAnTDgdOYQQ4/vtk7969aawzEg78gMH58+fThGe1ACvk0w1zNLX37Nlj9Z66xwowTYIQ+PPPP9HpE/DLlCmTXLlypZnC5s2b6+vVq1c37YyDL7/8UrZt21bmyZNHu7n88stlq1at5JYtWwwnIf+rpp3MlSuX/i1cuFD27dtX5s2bV4eTP39+OXr06AC/7du31247deokX3nlFZklSxZ52WWXyRdffFG7O3v2rBwwYICZlsyZM8tatWrJr7/+2gxH9XdJpNHIM46Rhnnz5rkOAw4RppH2OXPmyMqVK+swS5UqJZWISzdpQTjbt283w5k9e7Z87rnnZKFChXRY2bJlk6pfTp48eRJOTaOEXjM38lGgQAF5++23y88++8x0YxwgnTVq1JAoU+QZXB955BGdPsNNqv3H24UmQQhYBahNmzYSDy1uVNy0qn9IpzKUAM2dO1fiITEeZuv/HDlyyOXLl4fN5ffff2/6LV++vD6GoFjDGT58uBkGhA3XihQpYj5QOIcAwNxwww2mXwiikZesWbPKBQsWaDcNGzY03Vjj+eKLL1yHAYcQaMO/6sg2j+vVq5eucLZt22b6NUQM+TPSjjiefPJJHSb+qBqPKdK4lj17dtM/RGb69Omm28WLF2uRNtKZL18+022zZs1Md6l2QAFKoBK3CtDzzz8v+/XrZ96kr776qk6pnQDt3r3bvPnx1h8/frw8cOCAfPnll2Xu3Ll1GJUqVZKqGRQyt1YBwsMzatQoqZpUcsaMGbJkyZI6DISlOol1GIYA4YFq1KiRFhXUhCCUn3zyiZlu1IJQa0DtombNmtr+mmuu0WEgje+//77p9t1335XIy5kzZ1yHgYCsAoT0oLb2+eefS9QI3aYF4VgFCEJpCOHGjRslRBxhW2ued9xxh7ZD7Q9pB98NGzaYtb6qVavKS5cuyb/++kv7g3/VxJRq1E9zQjphh9+sWbOQhJQzFKAEKvJgAVIjQ9J4o+PhV/0G0k6Axo0bZ97IEA6r6dGjh3nNaNpYrxvHVgEyBMK49vbbb5thzJw5U1tbBWjNmjWGU/2/S5cu2j0eTEOwcOH11183w9m0aZN2i/CMh9BaY0hPGFYBUsP5OlzjT3rCsQpQ165djSD0/+uvv16nE+UBo/p6TNHv0KGDtjP+oIaHZiwEFmbz5s1mHvv372840yJUuHBhfa1z586mfSodcBhe3f2JalQfgRg5cqSeF6NqEUL1F9gmFcPyhrn11luNQ/0f5xMmTNDH6k0uVNMu4LrdCTqDraZ27drmqXpIzWMcqJqCsF6H3datW/FPqKaLaNGihT7GH2tHNDphVQ3BvBZ8kNEw6tevHxBURsMpW7ZsQDiqP0yfnzt3Tv9H+lXfkj5WzbUAtzfeeGPAuZEGWCqRFao5Zl5XtT19nKqd0hQg81ZIzIPu3bsL1aQSX331lVAdm8J4EKypVc0D81TVOsxjHKjmVMC5mxPVlxHSGUTFalTNTIuQ1c54qJAWXDcMjosWLapPnUaLMhoGRNtqMhpOMINgrqppZUZjiJJpEXRgpAHWOXPmDGCimnTadfDIX1AQSXtKAfJB0Y4dO1ao/hOBGx1DuDCqmm6mXHXmCtUHoc9V34dQ1XzzmupbMI9VZ7Z5HO4AM5OtRjWXzNOKFSuaxzhQHd8B5zjBRELV7BCq70OoTmmhmhnaDdKuOraFVTDTeP7bIqNhBKcno+E4CXeZMmWEGvnSZfLzzz8HZEP1hQnVpBXVqlUTmLmONBhGjR4K1Uw2ToVqpukZ4KZFih0Evs5SLPN+yS6q+E899VTI5DZu3Fg3d+BAdWyKqVOn6jk5b731lq41wR7C0bRpUxw6GogWwoBB806NfuljNZolIHZWE1wzwDWjGQgBGjFihH5IUWOAPWpBderUMYXUGpb1OKNhBKcno+FY02J3jJqg0ZxFkwqMjh49KiDeaiqC+PTTT/WESuQXTdTSpUvrYKZMmaLFGSfwh3lPqLXhJZOSJpU6vBI9r8Gd0Nb0Yi4LRrLUTap/6u1qvSzV7OGAYV71IJpuMUdl2bJlAe6DT6yd0MZwsnWoGPHaDcOrpkNwUFLV1KQSGTN+DGUbnekIB6N0hgnVCZ2eMKyd0BiNspr0hGPthB4yZIg1GIl5T0g75u4YBu5Vk8rMp5U5RhKVCBlO5X//+9+A8sEImTG8j+NTp06ZblPpgKNgCVTa4QQIycTISigBwnUMmWNI3JgPhAdC1Y6kaiLgclhjFaCXXnpJqm/QzAcED5mqyQT4N0bB7AQIDvFA9ezZ0xySxsOGeTkYqreaUAKUnjDCCVB6wkmvACFsjHCBMQTHKBvVPNOCg+tWg/LD/CLDLSY5YrTNTflYw0mm40zIjAJHk0QEMDqD0Sp8We6mvwVZX7dunVDD75oCmnEYccOnFRidQRMQo10ZMbi91Bwg3cww+oLSG040wkCc0QrHLv1KcDVzVdvTn5KE60MCV3wqUqVKFbPpbBdmKthRgFKhlF3k0U6AXHijExKIiAA7oSPCR88kQAKREMhYvTqSGOk3IQmojzlFgwYNdNpUh3FCppGJSj4CbIIlX5kyRyTgGwJsgvmmqJhQEkg+AhSg5CtT5ogEfEOAAuSbomJCSSD5CFCAkq9MmSMS8A0BCpBviooJJYHkI0ABSr4yZY5IwDcEKEC+KSomlASSjwAFKPnKlDkiAd8QoAD5pqiYUBJIPgIUoOQrU+aIBHxDgALkm6JiQkkg+QhQgJKvTJkjEvANAQqQb4qKCSWB5CNAAUq+MmWOSMA3BChAvikqJpQEko8ABSj5ypQ5IgHfEKAA+aaomFASSD4CFKDkK1PmiAR8Q4AC5JuiYkJJIPkIUICSr0yZIxLwDQEKkG+KigklgeQjQAFKvjJljkjANwQoQL4pKiaUBJKPAAUo+cqUOSIB3xCgAPmmqJhQEkg+AhSg5CtT5ogEfEOAAuSbooosoRcuXBDHjx+PLBCPfG/fvl3Mnz9f/P777x6lIP3Rnjt3Tpw6dSr9HlPMBwUoRQp88eLF4s477/RlbkeNGiXatm0rpk+f7pv0f/jhh2LgwIG+Sa9XCaUAeUWe8ZIACQgKEG8CEiABzwhQgDxDz4hJgASyEkFqELj66qvFww8/nNCZnTRpkm36tmzZou3XrFkjQrmBg+7du9v698KySZMmomrVql5E7as4KUC+Kq6MJ7ZUqVICP5r4EIDg0zgTYBPMmRFdkAAJxIiA7wVISikOHjwoDh8+HCNEDJYESCBWBHwpQPv27RODBg0SZcuWFdmyZRNFixYVhQoVEvny5RO1atUSjz32mDh58mSsmDFcEiCBKBHwXR/Qrl27ROPGjUWmTJn0xLry5cuLggUL6nPUgnbs2CGmTZumJ6199dVXokKFClFC5e9gjhw5Ivbs2SNq1qzp74z4JPX79+8XJ06cEJUrV/ZJir1Jpu8EaMSIEbrms2jRInH55ZfbUhs+fLho166dmDBhghgyZIitm1Sz/O6778Rrr72mP2lItbx7kd9Zs2aJdevWiXfffdeL6H0Tp++aYD/88IPo0aNHSPEB+csuu0z06tVLzJs3zzcFwYSSQCoS8J0ANWrUSKxatcqxrPDt05VXXunojg5IgAS8I+C7Jthdd90lIEIHDhzQE8/Qx4MO6MyZM+uRsJ07d+rJanPnzhVoptGQAAkkLgHfCVDt2rXFhg0bRN++fUXPnj3FpUuX0tBt1aqVWLBggWjWrFmaa7SILYFwM5VjGzND9yMB3wkQIFesWFFghOv8+fNi9+7dArUerHdzxRVXiJIlS+oakR8LI5ZpbtmypWjYsGEso2DYFgJ4Od59990WGx7aEfClABkZwRwgNMEw/ydLlix6ON64xv+BBNAxj18ym0hqX9H+jix79uzJjDpqefNdJzRyzomIUSt/BkQCnhJwrAGtWLFCfPbZZ3pYG7OMvTaciOh1CTB+EogeAUcByp8/v568Nnr0aP2ZA+bXoLpapEiR6KUiHSHFeiIiJuxNmTLFMUXr168Xw4YNE9dff72jWzogARKwJ+DYBKtRo4bYtGmTwFosTZs2FS+++KKeX9OxY0cxY8YM3flrH3RsbGM9ETF37tx62Qpj+YpQ//HJx9GjR2OTyRiEunnzZvHmm2/GIGQGaUdg5cqVYvLkyXaXaGch4FgDMtzWq1dP4Ddy5EhdI4L4oDaEjmDUiPr16yeuuuoqw3nM/hsTEe+9996wcWR0IiLWcXGzlsunn37qq9E2jBbOnj1bPPTQQ2G58WJ0CPz888/6U4xu3bpFJ8AkDcW1ABn5x42M5ge+c8E2Lw0aNBDffvutGDNmjBg8eLAYOnSo4TQm/zkRMSZYGSgJeELAlQD9+eef4pNPPhEff/yxWL16tShWrJi455579Hm1atV0wqdOnSo6d+4sOnXqJOrWrRuzzHAiYszQMmASiDsBRwFavny5wMximPbt2wt85YsvzbNmDfR60003aTf4RCLWhhMRY02Y4ZNAfAgEqohNnDlz5hSvvPKK48gXlsbAzpWoHcXLYGIdxAi/YHPmzBltlSNHjuBLKXmO7+USYRpFqsDHh9DcGdW5tB1HwdDx/Mgjj4iFCxeKX3/91QzxgQceEP/73//Mc9SI4iU+WOenXLlyArNN8XmB3dfx6CvCdHia/08A5YgXCU18CKC1wJ1RnVk7ChCCaNOmjR7xMvbmxjrMP/74o6hevbrA/KB4GgghhKVMmTLiiSee0OtBY3oAh5jjWQqMiwSiQ8BRgLZu3SqWLVsmfvrpJ70MBqLFcqiodWCFPYx84aPQeBmsMAdBXLp0qZ4IiPQhDf379xcfffRRvJLBeEiABKJAwFGA8NU5qu92a9via9/Tp0/rL9KjkBZXQeBTDFRvDQMx/Pe//y2efvppgblBXAPIIMP/JJD4BBwFCJML8XkCFjQPNnPmzNFfocdz5UEsuYFJhsEGM7S7du0q7rjjDr1eUPB1npMACSQeAUcBuu6660Tx4sX1vB88+Hv37hXbtm3T08yfeeYZvTNFPEea0LmMGb2o7WBCpNWMGzdONG/eXC9EtnHjRuullD9esmSJ6NKlS8pziBcADJQ8/vjj8YrOt/E4DsNjpOnrr78WN998s7jhhhsCMoraxtixYwPsYn2Ch+iXX37R36TlyZNH1KlTx4wSI3GYMInPQj788MOYTog0I/XJAfrp/PTtmk+whkwmpoFgWx6a8AQcBQje8UEmlkFF7Qcfg2LxLzTNMBTuhXn22Wf1CJjdA4X5SOPHj9ciFI9JkV7kn3GSQLIQcCVARmax3Cl+iWBQM0PTMJS59tprQ12iPQmQQIIQcCVAaIJhvg9qQHZD7mvXrk2Q7DAZJEACfiLgKEAY/Wrbtq1edxmzjvPmzeun/DGtfxPAEiMPP/wwecSJQJMmTUTVqlXjFJt/o3EUIMwDwp5b6AMqUKCAf3Oa4ik3FlZLcQxxy76bNaXilpgEjshxGB6fXaDfh+KTwKXIpJGATwk4ClCLFi30R6hY4Y2GBEiABKJJwLEJhtEmTP7DLqOYg4PaEIbhrebJJ5+0nvKYBEiABFwRcBQgfPWO1Q5hsCKinUkmAcIs6xdeeMEumwF2WIoEM8IxU9wP5siRI/pzmpo1a/ohub5P4/79+/VERLtvKH2fuShmwFGA8OX5sWPHohhlYgeFRe/ff/99x0RiQf5EmRPlmFjlAN/zYfWC+fPnu3FONxESwMqhWDcdqzfQhCbgKEBWr1jhbfv27aJ8+fJ6m1/MOk42g5UD8XMyuXLl0guiObnjdRIggdAEHDuh4RUTEPHdF/bMwrKe2Cds0KBB+mM7LMdBQwIkQAIZIeAoQJj53KFDB/0B6KhRowTWiIbBRKsPPvhADBgwICPx0g8JkAAJCMcmGBb42rdvnxagfPnyieeff15ju/322/WsaCyPirlCWBiMhgRIgATSQ8BRgLDkKfb+gvgEm/r16wv09u/cudOzL+OD08RzewItW7bUC/jbX6VttAngxYwVQ2nCE3BsgmHJDaz/fPDgwTQhTZkyRe8PFs8VEdMkghauCGALI37H5wpVVBxh/hwGKmjCE3CsAeHNieFmbDyIbUYuXbokUCvCfJl33nlHT1LE/vA0JEACJJBeAo4ChJGvGTNm6G15MCMaxqhaduzYUbz++uvpjTOq7tH/hK2jMTu7YMGCUQ2bgaUOgUmTJkWU2e7du0fkP1U9OwoQwGD2LNb8+f7773XtBzUe9At5tdwAOsXHjBmjl1/F8V9//aXLD02MsmXL6q2khwwZoqcNpGrBMt8k4AcCrgQIGcGSHOh0xs9Lg215GjdurEfd7rzzTj0pEjUfjMIdPnxY7NixQ0ybNk1Mnz5dYCmRChUqeJnchIl78+bNmsdDDz2UMGlK5oSsXLlSf/rSrVu3ZM5mxHlzFKDVq1eLp556KmxE2CQwXmbEiBG6loPpAaFmYg8fPly0a9dOYGcC1IRohN67Df12FKD43A1YPQKfYlCAwvN2HAXDQ16iRImAH3r3sVLiN998I+K99jIWxe/Ro0dI8UF2MeKDb7XmzZsXPve8SgIk4CkBxxoQtr2ZPHlymkSi8xfbIe/evTvNtVha4GNRTAvAvmDhDPYw4/SAcIR4jQS8J+AoQKGSiD4XbLxWsWJF/fU4RsviYTASBxHCljsYeUAfDz4eRR8V+oAwKRIjGnPnzuU2zRkokEhHgzIQJb2kMIEMCxCYQQQuXryol+uIlwDVrl1br0/dt29fgdmmmJcUbFq1aiUWLFigF1ELvpaq5xBpfEhMEx8CqH1j9Qia8AQcBQg1CqxtYjUQHdQ2sG4ONiiMd1MHtS6McOFDWTQBkcYLFy4I7BuPSZNultOw5icVjuvVqyfwo4kPgfbt28cnIp/H4ihAWHrjiSeeSJNNdESjNuLlRETMR0ITDN+pcSJimiKiBQkkPAFHAcInGOfOnUuojHAiYkIVBxNDAhkm4ChAGQ45Rh45ETFGYBksCXhAwFGA3ExEtKZ73Lhxenay1S6ax7GeiIjPOs6cOeOYZPSD2XWAO3qkAxIgAZOA40REbEiYI0cOsXz5cj3BDyMpRYoUEdgtY9myZXr42zpRMWtWR00zI8/IQawnImKJEXSqO/3wXRx2xvCLWbJkid5WyS/p9Xs6MQsf01RowhNwVAsMr+Ohx6cPWJrDMKgBYFr/xo0bbScqGu6i/T/WExHxpb/xtX+4tDds2FD4aftdjBgePXo0XJZ4LYoEUIs+ceJEFENMzqAcBQjzaTAb2io+QIFRp2eeeUYPe2NVRNSC4mE4ETEelBkHCcSHgKMAYYkL7Cl18uTJNMtb/P7777oJFq9JiEDCiYjxuTEYCwnEg4CjAGFW8dmzZ0Xv3r11jQfzbrAVD5YbQBu3c+fOIk+ePPFIqxkHJyKaKHhAAr4m4ChAmOSHjz9vueUWXfuw5hbi4+XOj5iICDHCjyY8AfRXPfzww+Ed8WrUCGDbKq8W7ItaJuIQkKMAIQ0Y+cKOqFjjBB3SmAVdo0YN/RlGHNLIKKJAoFSpUgI/mvgQ8NMARXyI2MfiSoDgFWvsVKpUSff5GFsz2wdJWxIgARJwR8BxHhCCSaStmfFBJXZndfNzM5zuDhNdkQAJxIKAYw3I2JoZM4SxNfPgwYN1OtDG7dOnjzh+/Lh47733YpE22zCRBuzKinRhGgDWAQplKleuHOoS7UmABBKAgKMAJdrWzBA+jMBhOB6TITnb1N1ddOTIEb2MLnY4oYk9AcyNw0REvgTDsw5dffjbn9utmcNHE92rKNShQ4eKYcOGiUOHDkU38CQNDXO57JZVSdLsep4trKE1cuRIz9OR6AlwrAFZt2bGN2BW4+XWzAMGDBB169bVtSBrmnhMAiTgHwKOApSoWzPjU5AWLVr4hzRTSgIkkIaAowDhM4tE3po5TY5oQQIk4BsCjgKEL6gxARGzoX/66aeE2JrZN3SZUBIggbAEHAXoiy++0MtTYCPCRNiaOWxueDEkATSlsYQITXwIYMcWzkNzZu04Cla4cGEdyrFjx5xDo4uEJYCZ7FjZgCY+BLJnz64/WYpPbP6NxbEGhPk2WIOnadOmolOnTnoXiuA92QcOHOhfAkw5CZCAZwQcBWj9+vVi5syZOoF2WzTjAgXIs/JjxCTgawKOAtS2bVvu8OjrImbiSSBxCdgKEL77wg/t2FQz+PAWS444GXza4Kc1fzdv3qx3k8U63jSxJ4DPhTBw061bt9hH5uMYbAUIu51iGvlvv/1mZg0PJh64ZF9kCTtduFlk7cCBA3p7ahNQgh9gC+vZs2frjQQSPKlJkTxMXVm3bh0FyKE0bQXIzg++Qsc2PGvXrrW7nDR2WIIWPyeDIe0yZco4OeN1EiCBMAQch+HD+OUlEiABEoiIAAUoInz0TAIkEAkB102wSCKhX+8JFCpUSK/t7X1KUiMF2Fn31KlTqZHZCHJJAYoAnp+8Yilb/GhiQ2DSpElpAi5WrJiws0/jUFl0797dzjrp7UIK0MGDB0X16tVNANiEEIputTMu4iNVmsQg4PaGT4zUMhWpTsBWgLDPVvv27QPYYEcMGhIgARKIJgFbAbr11lsFfjQkQAIkEEsCHAWLJV2GTQIkEJYABSgsnuS5uGnTJvHGG28kT4YSPCcrVqxw3QGd4FmJafIoQDHFmziB49u+06dPJ06Ckjwl2Lfu7NmzSZ7LyLNHAYqcIUMgARLIIAEKUAbB0RsJkEDkBChAkTNkCCRAAhkkYDsMn8Gw6C2BCeDTgNatWydwCpMraVWqVBFgThOeAAUoPJ+kuYpvwfCjiQ+BkiVLxicin8dCAUrAAuTnFAlYKExSTAj4vg9ISinw3drhw4djAoiBkgAJxI6ALwVo3759YtCgQaJs2bIiW7ZsomjRorp5kS9fPr3kxGOPPSZOnjwZO2oMmQRIICoEfNcE27Vrl2jcuLHIlCmTuPPOO0X58uVFwYIF9TlqQTt27BDTpk0T06dP14uwV6hQISqg/B4IVjI4dOiQKF26tN+z4ov0Y9MCTEQsUaKEq/RG0uz281IevhOgESNG6JrPokWLRPAGiUZJDx8+XLRr105MmDBBDBkyxLB29R9rXn/yySeObiGEWKLEzmBh+0Qz27dvF19++aWuOSZa2pIxPViQfufOnaJPnz7JmL2o5cl3AoQtc7DvdijxARlsQ9yrVy/97VN6BQhbEWEhKSfTvHlzLYR27q666io7a9d2kfq3iwhbbIOdH9+W33zzjcALp379+r5JP7oAcB/6kbfd/RMrO98JUKNGjcSqVavEvffeG5bJ4sWLMzQPAwuu2S26FjYyXiQBEsgQAd8JEPaphwhhXy68XdDHg/ktmTNn1iNhqPaiPT137lz91swQFXoiARKICwHfCVDt2rXFhg0bRN++fXVT7NKlS2lAYV+vBQsWiGbNmqW5lugWyFflypWjvistluPA5oRvvfVWSATbtm0TOXPmFFdccUVIN7G6gAEEbIRpV/vcuHGjjhb70sXKLF++XDRt2jRqwRs7o4bjbUSG/PXu3Vtce+21hlXK/M+k5tFIv+YWSx7goUKt58KFC/rBwQxUP8/4zZ8/v+jQoYPImzdv1IsFRY3Rw1AGDzhGFGvUqBHKSczsUYZYWzx4KWBEePHiRV2+mHKBmm60DbiMHTtWPPjgg9EO2lV4s2bNEs8++6xjt4KrwHzmyNcC5DPWrpJbVs1tghB4sevqAw88oMXn/vvvd5XWaDqaM2eO3hIb20fH26AWjQ5jCJ0Xpm3btuLRRx8Vbdq08SJ6T+OM/uvE0+wwchIgAT8RoAD5qbSYVhJIMgIUoCQrUGaHBPxEgALkp9JiWkkgyQhQgJKsQJkdEvATAQqQn0qLaSWBJCNAAUqyAmV2SMBPBChAfiotppUEkowAJyImWIFiKQ9834aJcfE2WOgtR44cejZ0vOM+ceKEXq8IEzG9MPgcwosZ4Mgr1rAqUqSIyJ07txdZ9zROCpCn+Bk5CaQ2ATbBUrv8mXsS8JQABchT/IycBFKbAAUotcufuScBTwlQgDzFz8hJILUJUIBSu/yZexLwlAAFyFP8jJwEUpsABSi1y5+5JwFPCVCAPMXPyEkgtQlQgFK7/Jl7EvCUAAXIU/yMnARSmwAFKM7ln5FNSNz4ceMmzlkNiC4j6XPjx25bpoCIPTpxk/bgpLnxk6j5Dc6L23MKkFtSEbjDNr2DBg0SlSpV0h963nbbbfrDS6cgP/roI4EtoLFXV4MGDcTSpUsDvOBmxNbT+Hg1T548omvXrmncBHjw4MQpD3ZJWr9+vd50skCBAqJ8+fJi6NChaZxNnjxZ7yGGLbqxlVCXLl3E3r1707iLt0Wq5Tdivkp1aWJMoH///lKJhFy4cKFUW+7IWrVqSbXBolQCEjJmuFP7YMk33nhDqgdS9uvXT6p96+WPP/5o+nnooYdkrly55Pjx4+WKFStkx44dpXoYpfqy3HTj5YGbPASn79SpU1KJjuzWrZtct26dzpsSYDls2DDTqdq6B3vZyZ49e0olyvK9997TftTX7PLcuXOmu3gfpFp+o8FXRCMQhhGagNrFVarN9OTMmTNNR2qXUv0AzZs3z7QLPqhatapUW08HWKtdQ+U//vEPbff7779L9faX48aNM92cPn1aC516C5t2Xh445cEubc8//7xUmzLKM2fOmJdVLU8WLlxYnj17VtupfbSkWrYjQMCnTJmimUIEvDKplt9ocKYARYNimDBGjhypay7Bb2a1/bJEzcjO7NmzRz9M06ZNC7g8ePBgWbx4cW03ZswY/VCGq0UFeI7ziZs82CXpuuuuk6o5FXDphx9+0DxQy4OZPn26nD9/foCbb7/9VrtRu4wG2MfrJNXyGy2u7AOKuBEbPgDst47FprCtsNVg//UDBw5YrczjrVu36uMrr7zStMMBzg8ePCjQ94MtqatVqya2bNkiVFNEqGad6NGjR0L0gyCtbvIAd8EGvOzyDXeq1qedow+tdevWAV4nTZoksmbNKurVqxdgH6+TVMtvtLhSgKJFMkQ4x48ft11hEB2soQQIqwPCBO9xDz/YPvjQoUPit99+E4cPH9ad1BCj+vXri6lTp+oHENe9Nm7yYJdG8ArOd/78+bXTULxWrVqlt3UeOHCgKFGihF2wMbdLtfxGC2jWaAWU6uFcuHBBYElTq8GbHEurqj4gq7U+zpQpkzh//nwae1jgTQ4DN3YG/vCgYhnRl156SY+wwR0eQNSKRowYoe3t/MbCDvlG/g0DwXCTB8O99T94BefbOLfGYfhZvXq1uOWWW4RquukRQcM+3v9TLb/R4pv2yYhWyCkWzvbt20W5cuUCfmhOqD4bceTIkTQ0YKc6W9PYw8J4ix89ejTguhEO/BUrVkwLG5pdhrn66qtFzZo1xZo1awyruPxXncIB+Vb9U67yYJc4O14GB0w1sJrFixeLVq1aCTWiKObMmSPUKKH1clyP3ZSZXYL8ml+7vGTEjjWgjFCz8YMbUI1IBVxBPw/s0W+DplOWLFnM6+jPwBwfO4ObEmb//v0Bl+FHjQbpOT8IGw9c0aJFA9zA3q6mEOAoyieYp2OII4KuU6eOFkgch8sDrgcb5N3o6zGuGWFgvpNhlixZItq3b6/7gtQImKfigzS5KTMj7db/fs2vNQ8RHUerN5vh2BP46aef9OjMokWLTAeqtqTtVJ+NaRd8oJpSsm/fvgHWdevWlZ07d9Z2mP+iCl6uXLnSdKMmPOqh+aefftq08/LAKQ92acMwvOq0l3/99Zd5WTUz9dC8anZqOzDF3CDMFbK6Mz14dJBq+Y0GZg7DR4OiQxgtWrSQmMOjRqykmq0rW7ZsKRs3bmzOY1EjKFpYVJPCDOntt9/Ww/cYij927JgcPny4Pod4GUbNjtYT8DAEjTAwfI05NJhnlAjGTR4mTpyo8445TDCqc12qfiD54IMPStXJricaYnLl6NGjzSypZpdU2wdJTEXAJETrT21rZLqL90Gq5TcafClA0aDoEAZEp1GjRrrGojor5Y033hggEsYclg8++MAMCW921amsH0bUdDDJLXiC4R9//CHbtWunJzrCDWZbB8+PMQP04MBNHh5//HHNxajdIJmqP0fPcUKeID733XefWdOBQME+1M/KMN5ZTrX8RoMv9wVTd3K8zJ9//qk7jvHtklujZv8KJTSidOnSIb3gWzN01JYsWTKkGy8vuMlDcPrUzS127dql82SMMAW7SdTzVMtvJOVAAYqEHv2SAAlERIDD8BHho2cSIIFICFCAIqFHvyRAAhERoABFhI+eSYAEIiFAAYqEHv2SAAlERIACFBE+eiYBEoiEAAUoEnr0SwIkEBEBClBE+OiZBEggEgIUoEjo0S8JkEBEBChAEeGjZxIggUgIUIAioUe/JEACERGgAEWEj55JgAQiIUABioQe/ZIACUREgAIUET56JgESiIQABSgSevRLAiQQEQEKUET46JkESCASAhSgSOjRLwmQQEQEKEAR4aNnEiCBSAhQgCKhR78kQAIREaAARYSPnkmABCIhQAGKhB79kgAJRESAAhQRPnomARKIhAAFKBJ69EsCJBARAQpQRPjomQRIIBICFKBI6NEvCZBARAQoQBHho+dkI3DhwgWxfft2ceLEiWTLWkLmhwLkUCw1atQQmTJlMn+XX365qF69uujbt684deqUg+/0X7506ZJ4++23Bbb3dWuw5bPaN96t84R199lnn2nOhw4d8iSNH3/8sciXL5+oWLGiGDx4sCdpSLVIuTWzQ4lDgCA6AwYM0C6xDzv2LH/jjTdE3bp1xfLlyx1CSN/lKVOmiG7dugnEkytXLleeb7/9dnHmzBkxd+5cV+4T1REECHmBoBYqVCjuyaxSpYooWrSoePfdd0WxYsU8SUPcM+1xhFk9jt8X0ZcqVUrcfffdAWktU6aMuP/++8WOHTtEuXLlAq5FcoIaUHoN/KCWRhMZgT/++EP07NlTXH311ZEFRN+uCbAJ5hpVoMPChQtriyxZspgXDhw4IHr16iWuvPJKUaRIEXHrrbfq/gTTgToI52bWrFnimWee0c4bN24sJk6cqI+/++470axZM5EnTx4tdojj8OHD+lr//v3F0qVLxcqVK0W9evXE/v37xQsvvCCee+458eSTT+o3+qBBg7TbX3/9Vdx2222ibNmyOqxrrrlGTJo0SV/Dn2+++UaHsWbNGtGwYUOBPHbp0kWsW7fOdDN06FCB8F5//XXdVKlQoYIYOHBgmuboxo0bRevWrUXBggVFpUqVxNNPPy3Qv2I1c+bMEW3atNE1DbACm3DGTfpC5R01ygceeEDnHfnq2LGjrskivq1bt+p8Hz9+XIwdO9Y8xjWnfKCcUG7gBRZffvklvDn6A0c088C/Tp06In/+/KJdu3Zi586d2r/xB2WBMkDNrEWLFuKtt94S1peUU/rC3TtGHJ7+lzRhCaj+HnnLLbdI1cTRP/Xgy2+//VbWrl1btm3b1vR7/vx5Wa1aNakebvnhhx/KTz/9VNavX1+qPgW5b98+7c7JjXoQZL9+/aS6IeT7778vf/75Z6keHKmaI/Kmm26SU6dOlW+++aYsUaKEGfeiRYukagrKmjVrygkTJmj36i0ur7jiClm6dGl5zz33yBEjRsijR49KdZNL9cDo9CH8Ro0a6bjUTazTN2/ePH2ubnY5atQoqYRNx1uyZEmpmkXaTZ8+fXSelKhIJSBy2rRpOi7EY5hNmzbJHDlySCWIcvz48VI1aaR66GXXrl0NJ3L9+vVSNW1l79695ezZs6WqTcps2bLp+I24TMd/H7hJn13e1QMrGzRoIJEvJcxSNfVk06ZNpXpJSNXfJFGmYIc0I404PnfunHSTj5w5c+r8IzzcJz/88IMrf+CIclQ1aTlkyBDNG+UDZoZRLwzNrX379vLzzz+XSlw1I/Vi0k6c0ud07xjxePlfeBm5H+KGAEEQgn+qj0AeOXLEzMKYMWOkqg3JX375xbTDzQ27++67T9u5caPeiDou3DwwEDvErWoh+hx/8AA99thjEg8WjHqba6HQJ+oPHkLVJJNbtmwxrKR6M8vKlStL1X9l2kHwEPZ7772n7YwHfNiwYaYbCC/y+vjjj2s7PDiZM2eWmzdvNt0sWLBAh6PettpO1bK0aKqOdNMNhApxff/999ruhhtu0CJgOlAHnTp10m6cBChc+uzyPnnyZB2u6iMzozt9+rRUfWzyX//6l2mXN29eLdaGhZt8QIDwArh48aLhTbrxB464N7Zt22b6e+2113Q6IYgwqi9QXnXVVQFhq1qcyc0pHjf3jhm5RwfsA1JPhZOpVauW7u+BO/VmFLt37xbqphaqBiG++OIL3SxCVVnVeHRzwwgPzQ/15hVff/21tnLjxvBr/Fe1KpE7d27ddPrnP/8pOnToINSDqn+GG7v/xYsXF0pwzEuqtiaUIOlzDDErARHqBhVKTIR6GE13OEAnuGGyZ88u4FfVWAwr3WRQD4Z5fuONNwr1IOqmGhigY/76668XSmxMNxhdQlyrV6/WnfeqppBmpAlNjRkzZph+Qh04pS847ytWrNCjW2jCGmWBsFGu1vPg+NzkA37QDEPeDOPWn6pZ6mab4Q+jbzAonwIFCggwQllbw0YTzDBO8SgxztC9Y4Qfj/8UIBeU0cmMYXerUTUQgY7o//znP0K9kXVnNPp+gg36bjCqAoMOayc3wf4xEvbVV1+JBx98UPcPoY8IfSqvvvqqFqNg98Y5bu5gg34bjN6hL0g1N7SAqhcfasEBToP9ov8B/UyGscsD3Ki3uRYzjGKhXwS/YIM+DvRfYagdI01Wo5qN1tOQx07pC76OOI8dOyaaNGmSJkz0h9kZiLJTPgx/GKQwTHr8oZ/QajDaCqNqU7qfR9WmBcTUzriJJ6P3jl18sbL7P9mOVQxJGq5qvwvUTvB2hcHNhAcr2OBtZrzZ3LgJ9o9z1KJQe8KDpPqA9LQAvBkhaKGMtXMcbiCUjz76qO58XbVqlVB9QkL1K2jvwQKEh9VqIBbWkb7g63CLh7V8+fJa2CBuEGh0lgb/XnnlFf12R20kmJfbyX/B8QenLzjvGNLHy8J4sK1pghjbGeTBKR+GP2t86fFn+Lf7j1qPahLqcrJex9wzDDS4jScj9441vlgfU4AySPjgwYN69MRo5kCM1q5dK1S/kBkibnTUXlQHsbZz48YYTodfGIhF8+bN9QOOhwg1IdVJqh/sDRs2aDfwg4crnMEcIQwvo+aEEZvLLrtMV/EhPsF+DVFFeEjHsmXLhOp0N4P/8ccfA2YKI98YZUKTBmlBPKoTXrvHOX54m6t+Hz3ShnO4Xbx4sRkmDpBXN8YpfcFhgDvmbqHJaaQH+b7rrrs0j2D3OIc7p3xE059dWIgfo5tWo/qJdBMWXQFO6XNz71jD9uRYFQRNGALohMaIFzpq8XvnnXckOkHRoaseYqlqJtq3GkKW6q0ub775ZqlqKlLNKZFKLCQ6KdXwsWs3GBFSN4JUw8FSfRIg1RtPdwKrh0V3/Kpaj1R9QTJr1qw6DgTco0cPqZodEp3I6LxGR+x1112n4zT+jB49Wnd6qodeYjRO9X3oETvEhVEYGKMTWtVkpBIVPTqkhvml6svSaYEbdJ7CD0aLkEd0RhsjgkrI4ER3ksMN0oGRPHSgY8QNLNVQvHaDETR0wmJUTw1/S9Vc0/zgz6kTOlz67PKOwQKMJKrhbj2ahFHJJ554Qsdv7dwP7oRGZ79TPlC+6Dy2Gjf+wNE64gX/8+fP1/Fh9AsGI1+IHyOSGMUEM9Ukk+oloq87xePm3tEBefiHo2AO8INHwVTVWA+fqo5XiSFwq4HQqLeSvmnUG1QPy+OhthonN3hY8EDjxsOIB8zMmTOl6tTVogN7iM3ChQvNYJEOPDy4tmTJEv3gBwsQwsVomaq663BwI6tPD6SahyPV/BMdliFAal6PVJ3POjzkR9XizLjw4FStWlWqeTt6pA0ijOkICN9qINYQLqQJcUKYIWpWo/qj9BA23GCY/uWXX9bunQQoXPrsBAhxYtgfAoS4UDYYucLLxGqCBQjXnPJhJ0Bu/LkRIISDF5FRtkg7ytAYIXUTj9O9gzC8NBSgGNBXzTOzdhIqeCc3eKCN2oIRhur7kL/99ptxGvBfNZX0nJYAS5sTvBX37Nljc+X/akCoZWEIfe/evWncWR8c5AG1l3AGw/6qwzSkE6QbNUb8dzKGQIZLn1MY4Kr6UJycpbnulI80Hv62yKg/a3hgg1pRsMhb3TjFE+7esYYT72OOgqnXSrQNZto6GSc3mBkbbNSbUHdMBtvjHH0WGPZ3Mhgux8/JYETGbrTL6s8pD3CrJkNavaQ5RrrRt5Ve4yZ9dmHacbVzF2znlI9g98Z5Rv0Z/vEfjKyDANZrxrFTPOHuHSMML/6zE9oL6oyTBEhAE+DX8LwRTAIYyVLNG4FJhhglszOqWaaXCjGmFti5iZWdm/TFKm6GGxsCFKDYcGWoJEACLgiwCeYCEp2QAAnEhgAFKDZcGSoJkIALAhQgF5DohARIIDYEKECx4cpQSYAEXBCgALmARCckQAKxIUABig1XhkoCJOCCAAXIBSQ6IQESiA0BClBsuDJUEiABFwQoQC4g0QkJkEBsCFCAYsOVoZIACbggQAFyAYlOSIAEYkOAAhQbrgyVBEjABQEKkAtIdEICJBAbAhSg2HBlqCRAAi4IUIBcQKITEiCB2BCgAMWGK0MlARJwQYAC5AISnZAACcSGAAUoNlwZKgmQgAsCFCAXkOiEBEggNgT+H3lVvO5lFtFoAAAAAElFTkSuQmCC"
 /><!-- --></p>
+</section>
+<section id="preference-for-a" class="level3">
+<h3>Preference for A</h3>
+<p>When users click on the interleaved results <em>with</em> a preference for 
A, the resulting preference statistic is <em>positive</em> and the confidence 
interval does <em>not</em> cover 0:</p>
+<pre class="sourceCode r" id="cb5"><code class="sourceCode r"><div 
class="sourceLine" id="cb5-1" data-line-number="1">y &lt;-<span class="st"> 
</span>interleaved_data_a[interleaved_data_a<span class="op">$</span>event 
<span class="op">==</span><span class="st"> &quot;click&quot;</span>, ]</div>
+<div class="sourceLine" id="cb5-2" data-line-number="2">y &lt;-<span 
class="st"> </span>y[<span class="kw">order</span>(y<span 
class="op">$</span>session_id, y<span class="op">$</span>timestamp), ]</div>
+<div class="sourceLine" id="cb5-3" data-line-number="3">boot_y &lt;-<span 
class="st"> </span><span class="kw">interleaved_bootstraps</span>(y<span 
class="op">$</span>session_id, y<span class="op">$</span>ranking_function)</div>
+<div class="sourceLine" id="cb5-4" data-line-number="4"><span 
class="kw">hist</span>(boot_y, <span class="dt">col =</span> <span 
class="st">&quot;gray70&quot;</span>, <span class="dt">border =</span> <span 
class="ot">NA</span>, <span class="dt">main =</span> <span 
class="st">&quot;Preference for A&quot;</span>, <span class="dt">xlab =</span> 
<span class="st">&quot;Bootstrapped preferences&quot;</span>)</div>
+<div class="sourceLine" id="cb5-5" data-line-number="5"><span 
class="kw">abline</span>(<span class="dt">v =</span> <span 
class="kw">quantile</span>(boot_y, <span class="kw">c</span>(<span 
class="fl">0.025</span>, <span class="fl">0.975</span>)), <span class="dt">lty 
=</span> <span class="st">&quot;dashed&quot;</span>)</div>
+<div class="sourceLine" id="cb5-6" data-line-number="6"><span 
class="kw">abline</span>(<span class="dt">v =</span> <span 
class="kw">interleaved_preference</span>(y<span class="op">$</span>session_id, 
y<span class="op">$</span>ranking_function), <span class="dt">lwd =</span> 
<span class="dv">2</span>)</div></code></pre>
+<p><img 
src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASAAAAEgCAYAAAAUg66AAAAEDWlDQ1BJQ0MgUHJvZmlsZQAAOI2NVV1oHFUUPrtzZyMkzlNsNIV0qD8NJQ2TVjShtLp/3d02bpZJNtoi6GT27s6Yyc44M7v9oU9FUHwx6psUxL+3gCAo9Q/bPrQvlQol2tQgKD60+INQ6Ium65k7M5lpurHeZe58853vnnvuuWfvBei5qliWkRQBFpquLRcy4nOHj4g9K5CEh6AXBqFXUR0rXalMAjZPC3e1W99Dwntf2dXd/p+tt0YdFSBxH2Kz5qgLiI8B8KdVy3YBevqRHz/qWh72Yui3MUDEL3q44WPXw3M+fo1pZuQs4tOIBVVTaoiXEI/MxfhGDPsxsNZfoE1q66ro5aJim3XdoLFw72H+n23BaIXzbcOnz5mfPoTvYVz7KzUl5+FRxEuqkp9G/Ajia219thzg25abkRE/BpDc3pqvphHvRFys2weqvp+krbWKIX7nhDbzLOItiM8358pTwdirqpPFnMF2xLc1WvLyOwTAibpbmvHHcvttU57y5+XqNZrLe3lE/Pq8eUj2fXKfOe3pfOjzhJYtB/yll5SDFcSDiH+hRkH25+L+sdxKEAMZahrlSX8ukqMOWy/jXW2m6M9LDBc31B9LFuv6gVKg/0Szi3KAr1kGq1GMjU/aLbnq6/lRxc4XfJ98hTargX++DbMJBSiYMIe9Ck1YAxFkKEAG3xbYaKmDDgYyFK0UGYpfoWYXG+fAPPI6tJnNwb7ClP7IyF+D+bjOtCpkhz6CFrIa/I6sFtNl8auFXGMTP34sNwI/JhkgEtmDz14ySfaRcTIBInmKPE32kxyyE2Tv+thKbEVePDfW/byMM1Kmm0XdObS7oGD/MypMXFPXrCwOtoYjyyn7BV29/MZfsVzpLDdRtuIZnbpXzvlf+ev8MvYr/Gqk4H/kV/G3csdazLuyTMPsbFhzd1UabQbjFvDRmcWJxR3zcfHkVw9GfpbJmeev9F08WW8uDkaslwX6avlWGU6NRKz0g/SHtCy9J30o/ca9zX3Kfc19zn3BXQKRO8ud477hLnAfc1/G9mrzGlrfexZ5GLdn6ZZrrEohI2wVHhZywjbhUWEy8icMCGNCUdiBlq3r+xafL549HQ5jH+an+1y+LlYBifuxAvRN/lVVVOlwlCkdVm9NOL5BE4wkQ2SMlDZU97hX86EilU/lUmkQUztTE6mx1EEPh7OmdqBtAvv8HdWpbrJS6tJj3n0CWdM6busNzRV3S9KTYhqvNiqWmuroiKgYhshMjmhTh9ptWhsF7970j/SbMrsPE1suR5z7DMC+P/Hs+y7ijrQAlhyAgccjbhjPygfeBTjzhNqy28EdkUh8C+DU9+z2v/oyeH791OncxHOs5y2AtTc7nb/f73TWPkD/qwBnjX8BoJ98VVBg/m8AADcCSURBVHgB7V0J3A5V+76FolAklJJooVDSIolIZSu0aBP6qxT1q1QqadGioqJCoSISSlpJijb1yZLSvgiJQihlSen8z3V/zXzzzPu8z8zzerZ53uv+/d73mTlzlvtcZ+aa+9znzDkljBWhEAEiQASygMBOWSiTRRIBIkAEFAESEG8EIkAEsoYACShr0LNgIkAESEC8B4gAEcgaAiSgrEHPgokAESAB8R4gAkQgawiQgLIGPQsmAkSABMR7gAgQgawhQALKGvQsmAgQARIQ7wEiQASyhgAJKGvQs2AiQARIQLwHiAARyBoCJKCsQc+CiQARIAHxHiACRCBrCJCAsgY9CyYCRIAExHuACBCBrCFAAsoa9CyYCBABEhDvASJABLKGAAkoa9CzYCJABEhAvAeIABHIGgIkoKxBz4KJABEgAfEeIAJEIGsIkICyBj0LJgJEgATEe4AIEIGsIUACyhr0LJgIEAESEO8BIkAEsoYACShr0LNgIkAESEC8B4gAEcgaAiSgrEHPgokAESAB8R4gAkQgawiQgLIGPQsmAkSABMR7gAgQgawhUCprJRejgtetWydnn312TI1LlCghO++8s+y2225yyCGHyPXXXy977LFHTJxUnSxYsEBuu+02wW+5cuXkuOOOk6effjpV2edNPrmCU69eveSrr75SXHv37i1nnnlm3mBcoCKGknYEVq5caSzwCf+qV69uZs2alXJdNm7caCpWrBhT9kknnZTycqKeYa7g9N133xn7cnLb65hjjok6tAn1pwVUgJLTG1C7dm1p3bq1bN++XTZv3ixvvPGG/PTTT2JJSq688kr5/PPPU6rAN998Ixs2bNA8L7roIhk4cKDYOyKlZeRDZrmC05NPPhnTPvPmzZNPPvlEDj/88HyAuUAd6AMqAEl6Axo2bCjDhg2TRx99VJ566imxbzz35vriiy9k6dKlcRX47bff5Pfffy9wDWSybNkyJbMCF20AiM2RNm3aSLVq1WTvvfd2gvQ3KI+YyP+e/Pjjj/GCxVoSsmLFirjX4gWi7OXLl8u2bdviXXbDQNioJ36DJFkdkF8YnBBv7dq1gi51WCms3eKlR93Gjh2rl6xF7EYZOXKke5x3B/YGoKQZAW8X7KyzzipQ2s033+ya3OPHj9fr7du3N9Y/ZDp16mQGDRpkSpYsaUqXLm3uvvtuvb5161Zz9dVXm/Lly2vanXbaydi3pPnggw/c/Fu2bGl22WUXN28cI88ZM2aEzgMRkSfS4e/VV181Bx98sOa53377Gfswal6IU79+fbf7YP1Z5qqrrjLQ05ElS5a4+bzyyivm1ltvNXvuuafmZf1hpnv37uaPP/5wouuvJWRjLUa3HuhOWp+ImTp1akw8nITRoUAiGxCEk7VUjbVOTZ06dVws0WV+4IEHjCWNmCyD2i0msu8EmFiC0T/7gjL16tXT4woVKhTAxZc0sqcw9yhpRiARAeEBtU5h98abP3++atOqVSsN22uvvdyHGjcnCAACP45zs4KEQEA4L1WqlJk5c6bG8ebrxMXvtGnTQueBiHPmzHHLqlq1qnt81FFHaT6zZ89WgnTK2H333d04zZs31zj4B/+GE8chMdTP0R3X+vbt68a3Fo/Bw+ekKVOmjHsMP8nzzz/vxg2rg5vAc5AIpy1bthinLRw9vL8gHK84cQtrN29c/3GHDh20fiBjEPvgwYPd+j7++OP+6HlxTgLKQDN6CQhvUVg8N9xwg7GjHaZu3bruTVa5cmVjuyKqkXMj42Zv0qSJkgosob/++stMnjzZTQMrCFYDrIsGDRpoeKNGjTSP1atXm9GjR7txrSlvfvjhB4OHKmweyMhLQNBn6NCh5qWXXjKvvfaa+fvvv903te3eGTuSpDoijvOgvvzyy6qPl4BAlA4Rfvrpp6Zs2bIaH299R2AtIg9Yf9AdlsjixYtdqw/Y/fPPP0np4OTt/U2EE9rIqUfXrl3Nl19+ad577z21mpxwYOxIonZz4sT7tX5AfXkgT1i9kFWrVrnEnq/OaBJQvLshxWFeAnJuWv8v3u540B3x3siOVeRcO+ecc9wHc82aNU6weeihh9yHxfqTNPzFF190w7wWQzJ5eAmoXbt2bnk4wAPp1AXdFEdAlCBUXOvcubMGewno3HPPdaLq7/HHH69xYWFB0LVxLJ7TTz9dw5x/sPCs896AOCDJ6ODk4f8tDKcaNWqoXugqgmwdAeE79W7cuLETHGMt+dvNjRTn4N5773Xz83Yvrd/ODV+0aFGclNEO4iiYvYsyKbZ7IjVr1hRnHpD1acgRRxwhmPthb/YCqlhLQa97L3z77bd6arsu0qJFC/cSRtUcgSPYWgjOaYHfouZx9NFHx+Tl5INAS3Biu0LudWtp6XE8pzQw8Irtaunpn3/+qb9IY7unemy7a96ocvLJJ8ecF1WHmEzinGBAwFqMesUSr1hLzI1Vq1YtsT4vsdabfPbZZzpyhTZ1JF67Odfi/T7xxBMajDxQH2tB6rm1DN3oo0aNkhEjRrjn+XBAAspwK+Lhee6550KXiomDuJm94jzYeCBw3REcV6lSRU+DRouKmod/sqSTDwrdddddY/Sx3SnVxT/qhkBr3eg155/34UaY7Vo5l8QhJTfAd1BUHXzZFDj1Pvz+NkBkh3CcX28G8drNe917/M477yjpIMzaM2K7597L7vGECRPE+oV08qobGPGD2Ds74pXJR/UxW9ovBxxwgNhuh9gugVintNiujkbB8LMdKRPvg+NP65wXNQ+/PsjHEeuQlSFDhjinYrtIYrtU7rn3IN5D672+//77ix21U/Lxz42yvjBZuHChHHbYYdKnTx8pqg7e8uIdY8oCLDUM/9sRKiVFWJ0QTB2w/ig9hh7++vhx0oiF/HOsH1y2I4vilOFEx/SL9evX6xSHSZMmSY8ePZxLkf/lPKAcb0K/ZQB17WiJag0CwhsRFgIsBoTjzYu5RiCjRFLUPPz6oPvodB3xcIAYIeiK4QGGxVSUbgMewlNPPdXNCxMof/31V/2cxE5FkGeffVbslAWtb7p0QOHNmjVTHTD/B59FgHisL0a7zHrB/otHCH6cnLj+X9RpypQpGoyuJrp8IDzvn50h7ybLuzlB0XZhRUN7rxM63jygeLVwnNC2+1LgsiUcY0nGdU5iyNc7PD5mzBg3TWHO1WTy8DqhMRrll2eeecYdrbFPio7sWQJR/TBStWnTJk3idUIPGDAgJhsMZyMt5g85gvi2W+fW0z7U7rG1OIwlISeqCauDm8B3UBhOdha5OfbYY91yvTpA37Zt28bklKjdYiL+ezJ8+HA37zvuuCNeFA3ztnc+OaNpAbnvlugcwLy3pCDdunUTOwdIZ+fiDW3n5Qi6J3ZCX2BlUpGHU8h5550ndnKj4A2OrgisIDjX7UiXvt3hGyqK4LMVdLWaNm2q+Tp+LXTP4A/xfuCbLh1gweFzGXzGsu+++7ozsVE/WGV2ikFRquamsfN79Bi4denSxQ33H3itrHyygkqAXv2V5Xl0EEDz2SFh7eo4vqBktU9FHk6Z8FfgMw184e/3ZThxivJrrSj9bMVae/opid/n4s0zXTqgDHSNUC+n2+ktl8fJI0ACSh4zpiACRCBFCLALliIgmQ0RIALJI0ACSh4zpiACRCBFCJCAUgQksyECRCB5BEhAyWPGFESACKQIARJQioBkNkSACCSPAAkoecyYgggQgRQhQAJKEZDMhggQgeQRIAEljxlTEAEikCIESEApApLZEAEikDwCJKDkMWMKIkAEUoQACShFQDIbIkAEkkeABJQ8ZkxBBIhAihAgAaUISGZDBIhA8giQgJLHjCmIABFIEQIkoBQByWyIABFIHgESUPKYMQURIAIpQoAElCIgmQ0RIALJI0ACSh4zpiACRCBFCJCAUgQksyECRCB5BEhAyWPGFESACKQIARJQioBkNkSACCSPAAkoecyYgggQgRQhQAJKEZDMhggQgeQRIAEljxlTEAEikCIESEApApLZEAEikDwCJKDkMWMKIkAEUoQACShFQDIbIkAEkkeABJQ8ZkxBBIhAihAgAaUISGZDBIhA8giQgJLHLHIp1q1bFzmdvQqvWrVKXn/9dVm6dKk3OKePt27dKps3b85pHXNBORJQLrRCGnXYvn27VKlSJY0lpD/rSZMmSevWrWXYsGHpLyxFJQwfPlxuueWWFOWWv9mQgPK3bVkzIpDzCJCAcr6JqCARyF8ESED527asGRHIeQRK5byGVHCHENhpp51kxIgRO5RHqhNPmDAh1VnmXH6tWrWiEzpEq5CAQoAU5SglSpSQnj17RrkKkdT98MMPj6TemVaaXbBMI87yiAARcBEgAblQ8IAIEIFMI0ACyjTiLI8IEAEXARKQC0X+Hrzzzjv5W7kcrdmPP/4oS5YsyVHtcketyBOQMUbWrl0r69evzx1Uc0gTzIRu2bJlDmlUPFSZPHlyzo0+5iLykRwFW7lypTz88MOCRsbx33//rdhWqFBBatasKRgCHTBggJQrVy4XMadOaUYg2WH+Cy64IM0aMfvCEIgcAS1fvlyaNm0qGF4+++yzpVatWlKpUiU9hxWEDxanTJkizz//vMyaNUtq165dWN0ZTgSIQJYRiBwBDR48WK2cN998U3bZZZe48A0cOFDatGkj48aNU0sobiQGEgEikHUEIucD+vjjj6Vr166Fkg8QLV26tHTv3l1mzJiRdYCpABEgAoUjEDkLqEmTJvL+++/LJZdcUnit7JXZs2dL9erVE8YpDhdLliwpa9asKQ5Vzak69u7dWzAAQEmMQOQI6PzzzxeQ0OrVqwXOQ/h49txzT8E3T/ABLVu2TOCEnD59uqCbRhHFJ8o4LFy4UNX/8ssvtW2jUJcyZcpEQc2s6xg5AjriiCNk8eLF+n1Tt27d5J9//ikAIkbBZs6cKc2bNy9wjQFEgAjkDgKBBPTee+/J1KlT1aeSKx/YHXjggTrCtW3bNvnhhx/U6vnrr79kn332kX333Tfyb/zcuT2oCRFILwKBTug99thD1+OF5YG/oUOH6sS/9KoVLvedd95ZQEaweGDtYD4QumHse4fDj7GIQLYRCCSg+vXryxdffCHz58+XZs2ayd13363O3Y4dO8oLL7wgsDwyKQ888ECBtXbvuecenQt01FFHycEHHyyHHHKIdsEyqVeuloWZ4ldccUWuqpe3emEOGnoOlMQIBBKQkxwPN2YfY4cCAAvHL4a60e25+uqr5auvvnKipvUX/h/HKYmCxowZI/369dPJiU888YQMGTJEdQNBYsi+uAt8ZI8++mhxhyHj9ce9h9FaSmIEAn1A/uTwuSxatEg++ugj2bhxoxxzzDHy4YcfKjndfPPNcuedd/qTpPV89OjR0rhx45gRLxDikUceqd3FsWPHJlU+6oQRtiD5+eefBdYhuqgUIkAEioZAKAL65Zdf9Lurp59+WubOnStVq1aVCy+8UHB+2GGHacnPPfecdO7cWTp16qQPf9HUST7Vr7/+Kphz4RfMExo1apQ/OPAcQ/c33nhjYLwVK1Zo/sCBQgSIQNEQCCSgd999V528yL59+/by8ssv62cOpUrFJm3btq1qEMZ6KJqq/0u1ZcsW9T1hxnOHDh30g9T/Xf3vEbpplStX9gcHnp9xxhmCvyA57rjj5KCDDgqKxutEgAgkQCCWReJE3HXXXWXQoEE66W+vvfaKE+O/QfguC90SWEfpFHyE+vbbb0v58uWlQYMG2gX64IMP9MPUhg0bCtZhgaN8/PjxkdrILl2YAa8WLVqkK3vmWwgCNWrUEDw7lMQIBDqh4Xy+6qqr5I033pDvv//eza1Xr14xjmdYROkmHxSObhV8UI899pjACvnzzz8FZTuLP02bNk2voQvWo0cPV9/ieoAZ4pwRnvnWx0oNl19+eeYLjliJgRYQ6nPqqaeq1QHLA8tfYGj3k08+kXr16sn999+vo2CZqjfm/jhzkjAK54izJtBpp50m7dq10wmJzjX+EgEikJsIBFpA3377rWBJz88++0y/wUI1YNZjiPHBBx8UjHxhRnK2xfFJObOhs60PyycCRCAYgUALCBOqnAl+/uy6dOmi3TMMzWNGMqV4IpDsCoTFEyXWOh4CgRZQnTp1ZN68eYJhZ7+8+uqrguUeuOyFHxmeEwEiEAaBQALCJL9q1arpvB+ssYNRpu+++04mTpwo/fv319GnsmXLhikrJXFgjWF0IcwfLLTiLpgJjfajZBYBzD6/7bbbMltoBEsL7IJhXRMMc8Oxe9JJJ8VU8ayzzsr4yv/41OLMM89UvxMIEKM8hQm+Cyvu4uwaUtxxyHT9N2/eLH/88Uemi41ceYEEhBrtt99+ugYPrB9844JuF7pmBxxwQMYrfMIJJ8icOXN0JAxfvV933XUZ14EFEgEikBoEQhGQUxTW2sFftgWWDb45u/322+Xiiy/m+j/ZbhCWTwSKiEAoAkIXDOsAwQKKN+S+YMGCIhZf9GTOB6dc+6foGDIlEcg2AoEEhNGv1q1by+67764zj7H5Xy4IuoH8xCC4JeAjGzFiRHBExkgpAlgkD34gSmIEAgkI84BwE2MdnooVKybOjVdzDgFMGu3Zs2fO6ZXvCuXK8sW5jnPhQ0j/ao5RFPh9SD653pTUjwhED4FAAkI3Bx+hfv7559GrHTUmAkQgpxEI7IJhHhD24sKi7+ecc45aQ/C/eKVv377eUx4TASJABEIhEEhA+Oodqx1CsAJiPCEBxUMld8LwMTH3SMtse2DEGEvFYONMSuEIBBIQluL47bffCs+BV3IaAUxTaNmyJbcqynArTZ48WTdwwC4ulMIRCPQBeZNu2rRJR8MwxRzsTiECRIAI7AgCoQgI5iS++ypXrpxgeBH7hN1www36GQTnOuwI/ExLBIo3AoEEhJnPp59+unzzzTe655azzi2+yXr88cczuhpi8W4q1p4I5B8CgT4grCe8cuVKJSDMhnaWGMAX6ZgV3a1bN12iFRPeKESACBCBZBAIJCAsyYq9v0A+fjn66KPlp59+kmXLlmXly3i/PjwviACmTKxZs6bgBYakFQHsVcfvFIMhDiQgLLmB9Z/Xrl0r/m15Jk2apDtScEXEYKCzGQPbaFMKRyDZJWUvuOCCwjP79wrmz1GCEQgkIAzh4lMMbDzYp08fwQp7sIpeeeUV3f4GkxSxUwWFCBABIpAsAoEEhJGvF154Qbp3764zolGAs9Rpx44d5aGHHkq2zJyOD38XdgAJEmwJna8r3iVrEQRhxetEoDAEAgkICbEDKdb8wXbHsH5g8cAvVLdu3cLyjWw4phhg2dcgge/rl19+CYrG60SACCRAIBQBIT2W5IDTGX/5LCeffLLgL0iwK2vNmjWDomX9OlYzuPLKK7lNdYZbAsvY4AuCM844I8MlR6u4QAKaO3eu3HjjjQlrhR1TKbmJAHx22KFh2LBhualgnmqFtdNXrVpFAgpo30AC2mWXXWTvvfeOyWbjxo26LzxmSGNpVAoRIAJEoCgIBBJQw4YNdQ8wf+aOaY9dUSlEgAgQgaIgEPgpRmGZYuYztsTBUh35OhpUWN0ZTgSIQGoQCLSAEhWzevVqne0JZxuG6ym5hwBeFFy8P/PtUqNGDd29N/MlR6vEQALCZxYvv/xyTK0wxXz9+vUyevRo3aCQM6Fj4MmpE4xe4ns+SmYROPvsszNbYERLCyQgzIu5/vrrC1Rvt912091J820iYoGKMoAIEIG0IRBIQPgEg4uPpQ1/ZkwEijUCRXZCF2vUWHkiQARSgkCgBRRmIqJXkyeffFJq1arlDeIxESACRCAuAoEWEDYkLFu2rLz77rtSunRpXZIVy3JgtwzstgAnJyYqOn+lSgVyWlxFGJgeBDATulq1aunJnLkWigBmnzuL9xUaiRckkC0wvI5p5RhJwdIcjmAk7IorrpBPP/007kRFJx5/s4sAJoxiLSdKZhHAWumcHxeMeSABzZw5UzAb2ks+yBYr7fXv31/XCsKX4f7PNYKLZgwiQASKOwKBXTCs+zxv3ry4bP7zzz9rF4yTEIv7bcT6E4GiIRBIQK1atZKtW7fKRRddpH4fmJVYY3jq1KmCyVadO3eW8uXLF610piICRKBYIxDYBcNi9FgT+rTTTtOJh160QD4jR470BvE4xxDAIMGIESNyTKv8Vwcvbu6ZF9zOgQSELLAZ4ZIlS+Tzzz9XhzRmQdevX18/wwgugjGyiQC+BevZs2c2VSiWZeOZoQQjEIqAkA2G4A866CD1+WCeD84pRIAIEIEdQSDQB4TMc3lrZmeYGR/HUogAEYgWAoEElItbM2PnCuxNX9OuyYwF8qtUqSLY+wr+Kpi+1157bdxRu2g1DbUlAvmPQGAXLNe2Zl6+fLk0bdpU4NvAKBy6g5UqVdJzWEFLly6VKVOmyPPPPy9YGLx27dr534oBNcSM9ebNmwfE4uVUIoBeAz7i5v2XGNVAAsq1rZkHDx6slg+IEetVx5OBAwdKmzZtZNy4cTJgwIB4UYpNGGasYxIptwnObJNPnjxZF6V/4IEHMltwxEoL7IJ5t2b21y0bWzPjs5CuXbsWSj7QEQ5ybKQ4Y8YMv8o8JwJEIIcQCCQgvD2drZknTpzobs18yy23CP4yvTVzkyZNdF5SEIazZ88WrtQYhBKvE4HsIhDYBcu1rZlBeCAhrEd9wQUXaB8bDmhMuIMPCEvIYmvh6dOncynS7N5bLJ0IBCIQSEDYAx0TEDEbGnumZ3tr5iOOOEIWL16sk+u6deumFpm/lpiFio9oi+J4/frrrwXWU5DgOzgO/QehxOtEIDECgQQ0bdo06dKli6xYsSJntmY+8MADdYQLUwSwLxmsnt9//12/WcOoQ6NGjfRr/cRVj38VpIIlRoJky5YtkViqFqsW4Ns9SmYR6N27Nx3/ISAPJKDKlStrNth6B76gbAtGFWCV3XnnnToHCGSEvckw2uWsXQ0SwvdPp5xyStLqYs93/AXJokWLIrMECbqolNQhgC5+MgJXASU+AoEEhC4P/C7NmjWTTp06qc/FP/zdp0+f+LmnIRTdL+8CW2PGjJF+/frJSSedpHpi22g4yzt27CgffPBBgQ9o06ASsyQCRKCICAQSEN70L774omaPBzueZJKA/OVjb7LGjRvHOJyxX/2RRx4pQ4cOlbFjx/qT8JwIEIEcQSCQgFq3bi2bNm3KEXULqoHuGPrbfrnkkktk1KhR/mCeEwEikEMIxJ0H9Pfff6tDN4f0jFEFDuC//vpLwzp06CD4NswvCxcuFMd/5b9WnM7xsS7W7qZkFgGMGM+fPz+zhUawtLgWEHY7hbN31apVbpXwbQtGmurWreuGZeMA34C9/fbbugpjgwYNZI899lBfD74Lw9rV0PPuu++W8ePHy7Bhw7KhYk6ViV0x4JAP41jPKcUjrgxGZzds2KAjxxGvSlrVj2sBxStxyJAhcuGFF8a7lNEwdKvgl3rsscf0ocLIF7YCwoJpEEwbwDV0wXr06JFR3VgYESACySEQ1wJKLovMxsbyGxiZwx++93IE3UYIlo5t165dTkwZcHTjLxEgAvERiBwBxa+GqBWEa/vss09hURhOBIhAjiGQNwSUY7jmjDrwmR166KE5o09xUQSTP2GtUxIjQAJKjE/kr+Ij3Ztuuiny9YhaBY499tioqZwVfQslIMw2rlevnqsUPr7EfCBvmHMRQ44UIkAEiECyCMQlIHxf1b59+5i8sCMGhQgQASKQSgTiEhAm9+GPQgSIABFIJwKh5wGlUwnmTQSIQPFEgASU5+2OmdC9evXK81rmXvWwaQJ2ZqEkRoAElBifyF/Ft2D4hIaSWQSwWN7WrVszW2gESyMBRbDRqDIRyBcESED50pKsBxGIIAJxR8EiWI9ipXIyS4LCB0TJLgLJtBc0LU5LuJKAsntvpr10fIrh/Wg37QWyAEXgsMMOE/iBKIkRIAElxifyV0FAWC+bklkE9t9//8wWGNHSSEA50HDJmug5oDJVIAIpQSDyTmgMM+O7NW4SmJL7gZkQgYwiEEkLCGtAP/zwwzJ58mRdD9pZjKxChQpSs2ZNwc6o2CcM20pTiEDUEEjWIo6y0zpyBLR8+XJp2rSpwLeBdaBr1aollSpV0nNYQUuXLpUpU6boLNRZs2bpPmZRuwFTre+XX36Z9bW8U12nXM9v3bp1ghdj1apVc13VrOoXOQIaPHiwWjmY6u7fINFBcuDAgdKmTRsZN26cWkJOeJjfBQsWqGUVFBdEiCVK4kmyb7B4eaQqDMPwwAOL9FMyh8CHH36oi9JH2TrJBFqRI6CPP/5YunXrVij5ALTSpUvr0PMjjzySNAGVKVMm1FvrxBNPVCKM10i5dNNt375dscglneJhlihs9erV8swzz6gVF5V64OWEXWWiom8i/NN5LXIE1KRJE3n//fd114tEwMyePVuqV6+eKErca1hwLd6ia3EjM5AIEIEdQiByBIR96kFCeCvi7VK7dm3B+rtYehQ+oGXLlgm6QNOnT4/ZrnmHUGJiIkAE0oJA5AgI2/EsXrxYevbsqV2xeJ8aYBRs5syZ0rx587SAlijTJ554Qr755hupUaNGomhFvoaNFzHDFs73MAJ8MFVh+PDhYaKrX2vjxo1y8MEHh4qfbKRffvlFXx6YKRxW5syZo1HR/Q6qB7bqhn/u8MMPD5t9UvH++OMP+frrr6VRo0YJ08FK/+233wL19WeCfe4waDB69Gj/pbw8L2FvThPVmuFBxA6UsHqwVTO25Nl3333VIspWnTp37qw6HXXUUWlRAZsybt68WY4//vjQ+aOJMWoYRr744gsloZYtW4aJnnSc77//Xr766itp27Zt6LTwY6F94dsrWbJkwnSYojFv3jzp1KlTwnhFvQgCxQDIueeeW9QsEqYDwU2dOlXwEigWAgKipA6Bfv36Gbs1dOoy9OVkRwHNdddd5wtN3am14Mz//d//pS5DX0724TKWHHyhqTu123Yba/mmLkNfTp988omxW4L7QlN3umLFCmNfoqnLMMdzivxM6GLxlmAliUCeIkACytOGZbWIQBQQIAFFoZWoIxHIUwRIQHnasKwWEYgCAiSgKLQSdSQCeYoACShPG5bVIgJRQIAEFIVWoo5EIE8RIAHlacOyWkQgCghEeiZ0LgKMb9Qw67hKlSppUQ8zcTEreO+9905L/hs2bJBNmzbpjPJ0FIAZvigjXWsmQ3e0QdhPVZKtIzYbtJMF5aCDDko2aaj4aNslS5ZInTp1QsWPeiQSUNRbkPoTgQgjwC5YhBuPqhOBqCNAAop6C1J/IhBhBEhAEW48qk4Eoo4ACSjqLUj9iUCEESABRbjxqDoRiDoCJKCotyD1JwIRRoAEFOHGo+pEIOoIkICi3oLUnwhEGAESUIQbj6oTgagjQAKKegtSfyIQYQRIQCEaz67rHSJWbJQwaeJtKRSbS2rOwujiL6koafx5pPK8KPqESRMmTirqUZRywqTJ1D2UCgzi5UECiofKv2HYAgebH1asWFE/brzzzjsTxP7vJWw706dPH90XrFq1anLhhRcW2GJl4sSJuvsq9ravVKmSnHPOOYL9vlIt6dLfq+c777yjm0K+++673uCUHGOLmhtuuEE//AROZ5xxhqxbty5h3viYc8iQIdK4cWMpV66c7g2Hfdq9god2wIABuqll+fLldYsdu5uGN0pKjtOp/x133KHbUGGrokMOOUS3rk6J0pnOxLIsJQ4C9qtqY7+oNuedd5756KOPzJgxY8yuu+5q7rrrrjix/xfUrFkzc/TRR5tp06aZ559/3tivmo19GNwIr7zyCswpY/e3N9hCZtSoUVpO/fr1jd2Uzo23owfp0t+r1++//24OOOAArY8lIu+llBxfeeWVxu58a9544w2D/O1mg8ZuTGksgRSa/7Bhw8zuu+9u7AaGxm4OaCy5m912283Yzf7cNFdccYWGoU3fe+8907FjR2MJzqA+qZR06X/fffcZu+KCbv80d+5cc8kll2gbzJgxI5XqZyQv7JpJiYPAbbfdZipUqGC2bNniXrVvTVO5cmVjl2Rww7wHr7/+ut4I8+fPd4OttaNhdrdUDWvdurWpWbNmzEM0adKklD/E6dLfrZg9uOyyy4zd4TTluqMMu/utsdttmxdffNEt0m6aqGUV9qDZDRuNtTpN37593TR2d1IlG2evtp9//tlYy9M8+eSTbhykA9E99dRTbtiOHqRLf+jVsGFDg/vIkb///ttUr17dWGvbCYrML7tghZiclkykTZs2UqZMGTdGhw4dBOvxWIJxw7wHJ598sixdulS8u6JibRoITGWIfVvJyJEjY3YqtVaEXsNWvqmSdOnv6IfdQSdPniwPPfSQE5TSX2v1yM4776xt4GRct25d3TLaWpdOUMxv2bJl5dNPP5X+/fu74Vh7CNsdO/g/++yzgm5X9+7d3ThI991330nXrl3dsB09SJf+0GvPPffUbZ8dHS3biCUhsS9MJygyvySgQpoKN6R9q8Rcdc7tWzQm3DnBQmTWutFT3PjWspFBgwbJ6aef7obDj3HKKac4SfR3woQJUqpUqRjiiolQhJN06Q9VsKiY3T1VHnjgAfVDFEG9wCTQf6+99lIS8kbG9tsOqXvDnWNroSrBwBcEv9Tll1+u+Zx99tkaBVt5Y1967O9uu8G6hzyIJ9U+uHTpj0qgTtYtIDfeeKO89dZb0qNHDyVZL6k6eOT8b2RstQwrat++BbZYtje1dgHgZwiSRo0aaVzrwDbLly8vNPqcOXO0S+DtNhQaOYkL6dQfWzdbElVtnG5Rqn1A1vmvPh9/lbGtc5itl+EnsQ+f/mG7aUfOP/98A39b1apVzYknnmjsw2uslavn1rp1ou3wb7r0dxTD/eLUD7/eLqUTJwq/tIAKeUXAZIdF4xXnHG/XIBk/frzADD/22GP1LTtv3rwCSawDUU477TQdscGoTColXfpPnz5dnnvuObHO81SqWyAv6G99QAXC0Qbbtm0rEO4PsIMHYv0wcv3116vFYAcPNAqsN3TTrrnmGrUeHn/8cVm4cKFaVYMHD/ZnU+TzdOkPhWB9WsKR+++/X+xe9WLJSKxjXR555JEi65u1hFFgyWzoCKekvXljisYb0jaUsTdtTHiiE3vD6+gZRkS8MmvWLHWOtmjRIuWjLygnHfrbYWVju0A6gmdNf4O/sWPHKiZDhw41CxYs8FZxh45t90Kd9f5MgNepp57qD054DqvJrqGtjn9YPHBur1q1KiZNgwYNTMuWLWPCduQkXfrjHoT+AwcOjFEPlh3aJmpS8BWTNSrMrYIxh8fv6/npp59USftwx1X222+/lZkzZ8Zcg8PzmGOOEa8FhH57+/btpVWrVgKLAvNVUi3p0H/NmjViH1yxo0ViiUD/HL/D1VdfrZZGquqBRffXrl0r27dvj8kSbVLYgvNw4sNB7XfmA2e0Hfw88CFhYMG/aQDCHQs3psAinqRLf1jNmMcEv6JXMECCtsE8tEhJ1BgzU/piGNs6QQ2GOB259957dWgeVk08sRMV9e3kDLkjzvr169XH06VLF03y2WefqUWE+UXevOPltyNh6dAfPjD4s7x/djRMLSA7umTsQ74jKsekBU72QTLI3xG7W4SG2S6gExTz+9VXX+n1e+65Jya8bdu2ijnwxtwr5AvfmyOw7DA0369fPydoh3/TpT+G96G/HYGM0dE6pk3JkiUNphRESTgPqJDWgolu+/Gmd+/eSiK4cTFZDV0NR6yfx3Tu3NltdDwg1prRLoL1MxjMB7I+HmNHuHQyI9LZt7Gxw77m4Ycf1kmImIjo/OEBSpWE0d9abKr/7Nmztdgw+vv1S5cTGuWgu1WvXj1jR6yMtV60i9S0aVN3DpVff6SBgxoO5pdeesmsXLnS3H777fpg3nLLLbisYi1SnfxpZ0gb5IHJipjzhbqkUoL0R1m9evWKGewI0h+TMOE8txacQbvZmeHG+n5Uf0xIjJqQgBK02KuvvqoTD/HGAflceumlMVbLddddp28jr0WEyYh2zysNR7oaNWoY5AMBKSCssL9kfEsJ1HYvBemPBxC6eMtNpL+bsecgnQQE0mnSpInqCBK386xiSCKe/sAYk/QcjGHZ3HzzzcY6rl2tbVfS2Dleaq0iHvxlqHeqJUh/lIf7w+vTCqM/4tgumM6Ghv6wfHBvpnomd6rxiJcf9wWzLZhILGhiuxy6UR/m6oQRpMF8E3tjpG2DvzB6IE7U9UcdMPkTI2L4Hiys2K6v+pDgL8KIVDzBt1q//vpr2tsonfpbK0/nmOG7wigKCSiKrUadiUCeIMBRsDxpSFaDCEQRARJQFFuNOhOBPEGABJQnDclqEIEoIkACimKrUWcikCcIkIDypCFZDSIQRQRIQFFsNepMBPIEARJQnjQkq0EEoogACSiKrUadiUCeIEACypOGZDWIQBQRIAFFsdWoMxHIEwRIQHnSkKwGEYgiAiSgKLYadSYCeYIACShPGpLVIAJRRIAEFMVWo85EIE8QIAHlSUOyGkQgigiQgKLYatSZCOQJAiSgPGlIVoMIRBEBElAUW406E4E8QYAElCcNyWoQgSgiQAKKYqtRZyKQJwiQgPKkIVkNIhBFBEhAUWw16kwE8gQBElCeNCSrQQSiiAAJKIqtRp2JQJ4gQALKk4ZkNVKDwF9//SVLliwRu81xajJkLgkRIAElhEekfv36UqJECfcPW+DWq1dPevbsKZs2bQpInfzlf/75Rx599FHZunVr6MTY+vepp54KHT9XI06dOlVxXrduXVZUfPrpp2X33XeXAw88UOx+8lnRobgVyq2ZA1ocBATSufrqqzUm9hPHXvGPPPKIHHnkkfLuu+8G5JDc5UmTJsl5550nKGe33XYLlfjMM8+ULVu2yPTp00PFz9VIICDUBYS65557ZlzNQw45RKpUqSIjR46UqlWrZkWHjFc6ywWWynL5kSh+v/32ky5dusTouv/++8vll18uS5culQMOOCDm2o6cwAJKVpAGVhplxxBYs2aNdOvWTQ499NAdy4ipQyPALlhoqGIjVq5cWQNKlizpXli9erV0795dqlevLnvttZd06NBB/QluBHuQKM7LL78s/fv31+hNmzaV8ePH6/G8efOkefPmUr58eSU7lLF+/Xq9duWVV8rbb78tc+bMkaOOOkp++uknuf322+XWW2+Vvn376hv9hhtu0Ljff/+9nHHGGVKzZk3Nq1GjRjJhwgS9hn//+c9/NI/58+fLcccdJ6jjOeecIx999JEb58477xTk99BDD2lXpXbt2tKnT58C3dFPP/1UTjnlFKlUqZIcdNBB0q9fP4F/xSuvvvqqnHrqqWppACtgk0jC6FdY3WFR9urVS+uOenXs2FEtWZT37bffar03btwoI0aMcI9xLageaCe0G/ACFq+99hqSBaYDjujmAf+GDRvKHnvsIW3atJFly5Zpeucf2gJtAMusRYsWMnz4cPG+pIL0S3TvOGVk9ddQEiJg/T3mtNNOM7aLo3/2wTcffvihOeKII0zr1q3dtNu2bTOHHXaYsQ+3GTt2rHn22WfN0UcfbaxPwaxcuVLjBcWxD4K57LLLjL0hzOjRo83nn39u7INjbHfEtG3b1jz33HNm2LBhZu+993bLfvPNN43tCpoGDRqYcePGaXz7Fjf77LOPqVGjhrnwwgvN4MGDza+//mrsTW7sA6P6If8mTZpoWfYmVv1mzJih5/ZmN0OGDDGW2LTcfffd19hukcbp0aOH1smSirEEYqZMmaJloRxHvvjiC1O2bFljCdGMGTPG2C6NsQ+9Offcc50oZtGiRcZ2bc1FF11kXnnlFWOtSbPzzjtr+U5ZbuR/D8LoF6/u9oE1xxxzjEG9LDEb29UzzZo1M/YlYay/yaBNgR10ho44/vPPP02Yeuy6665af+SH++Tjjz8OlQ44oh2tJW0GDBigeKN9gJkj9oWhuLVv39689NJLxpKrYmRfTBolSL+ge8cpJ5u/ks3Co1A2CAiE4P+zPgKzYcMGtwoPP/ywsdaQ+eabb9ww3NwIu/TSSzUsTBz7RtSycPNAQHYo21oheo5/eICuvfZagwcLYt/mShR6Yv/hIbRdMvP11187Qca+mc3BBx9srP/KDQPhIe9Ro0ZpmPOA33XXXW4cEC/qet1112kYHpyddtrJfPnll26cmTNnaj72bath1spS0rSOdDcOiAplLVy4UMNOOukkJQE3gj3o1KmTxgkioET6xav7xIkTNV/rI3OL27x5s7E+NnPTTTe5YRUqVFCydgLC1AMEhBfA9u3bnWQmTDrgiHvju+++c9M9+OCDqicIEWJ9gaZOnToxeVsrzsUtqJww945beJYO6AOyT0WQHH744ervQTz7ZpQffvhB7E0t1oKQadOmabcIprK1eLS74eSH7od988oHH3ygQWHiOGmdX2tVSbly5bTrdPHFF8vpp58u9kHVPydOvN9q1aqJJRz3krXWxBKSnmOI2RKI2BtULJmIfRjdeDiAE9yRMmXKCNJai8UJ0i6DfTDc85NPPlnsg6hdNWAAx/zxxx8vlmzcOBhdQllz585V5721FAqMNKGr8cILL7hpCjsI0s9f9/fee09Ht9CFddoCeaNdvef+8sLUA2nQDUPdHAmbzlqW2m1z0mH0DYL2qVixogAjtLU3b3TBHAkqx5Jxke4dJ/9M/JKAQqAMJzOG3b1iLRCBI/qJJ54Q+0ZWZzR8P36B7wajKhA4rIPi+NNjJGzWrFnSu3dv9Q/BRwSfyv33369k5I/vnOPm9gv8Nhi9gy/IdjeUQO2LD1ZwTFR/Wvgf4GdyJF4dEMe+zZXMMIoFvwj+/AIfB/xXGGrHSJNXbLfRe1rocZB+/uso87fffpMTTjihQJ7wh8UTkHJQPZx0GKRwJJl08BN6BaOtEGtNqZ/HWtMCMo0nYcop6r0Tr7x0hf2PttNVQp7ma/vvAusEb1cIbiY8WH7B28x5s4WJ40+Pc1hRsJ7wIFkfkE4LwJsRhFaYeJ3jiAOivOaaa9T5+v7774v1CYn1K2hyPwHhYfUKyMI70ue/jrh4WGvVqqXEBnIDQcNZ6v8bNGiQvt1hjfjxCjv5z1++Xz9/3TGkj5eF82B7dQIZxxPUIageTjpvecmkc9LH+4XVY7uE2k7e65h7hoGGsOUU5d7xlpfuYxJQERFeu3atjp443RyQ0YIFC8T6hdwccaPDerEOYg0LE8cZTkdaCMjixBNP1AccDxEsIesk1Qd78eLFGgdp8HAlEswRwvAyLCeM2JQuXVpNfJCPP61DqsgPerzzzjtine5u9p988knMTGHUG6NM6NJAF5RjnfAaH+f4w9vc+n10pA3niDt79mw3TxygrmEkSD9/HsAdc7fQ5XT0Qb3PP/98xcMfH+eIF1SPVKaLlxfKx+imV6yfSLuwcAUE6Rfm3vHmnZVj2xCUBAjACY0RLzhq8ffYY48ZOEHh0LUPsbGWiaa2Q8jGvtVNu3btjLVUjJ1TYixZGDgp7fBx6DgYEbI3grHDwcZ+EmDsG0+dwPZhUcevtXqM9QWZUqVKaRnIuGvXrsZ2OwycyHBewxHbuHFjLdP5N3ToUHV62ofeYDTO+j50xA5lYRQG4jihrSVjLKno6JAd5jfWl6W6IA6cp0iD0SLUEc5oZ0TQEhmiqJMccaAHRvLgQMeIG7C0Q/EaByNocMJiVM8OfxvbXVP8kC7ICZ1Iv3h1x2ABRhLtcLeOJmFU8vrrr9fyvc59vxMazv6geqB94Tz2Sph0wNE74oX0r7/+upaH0S8IRr5QPkYkMYoJzGyXzNiXiF4PKifMvaMZZfEfR8ECwPePglnTWIdPrePVYAjcKyAa+1bSm8a+QXVYHg+1V4Li4GHBA40bDyMekBdffNFYp66SDsJBNm+88YabLfTAw4Nrb731lj74fgJCvhgts6a75oMb2X56YOw8HGPnn2heDgHZeT3GOp81P9THWnFuWXhw6tata+y8HR1pAwljOgLy9wrIGsQFnVAmiBmk5hXrj9IhbMTBMP19992n8YMIKJF+8QgIZWLYHwSEstA2GLnCy8QrfgLCtaB6xCOgMOnCEBDywYvIaVvojjZ0RkjDlBN07yCPbAoJKA3o2+6Za50Uln1QHDzQjrXg5GF9H2bVqlXOacyv7SrpnJaYwDgneCuuWLEizpX/WUCwsjCE/uOPPxaI531wUAdYL4kEw/7WYVpoFOgNixG/QeIQZCL9gvIArtaHEhStwPWgehRI8G9AUdN58wM2sIr8JO+NE1ROonvHm0+mjzkKZl8rqRbMtA2SoDiYGesX+yZUx6Q/HOfwWWDYP0gwXI6/IMGITLzRLm+6oDogrp0M6U1S4Bh6w7eVrITRL16e8XCNF88fFlQPf3znvKjpnPT4BUbeQQDvNec4qJxE946TRzZ+6YTOBuoskwgQAUWAX8PzRnARwEiW7d4IJhlilCye2G6ZLhXiTC2IFyddYWH0S1fZzDc9CJCA0oMrcyUCRCAEAuyChQCJUYgAEUgPAiSg9ODKXIkAEQiBAAkoBEiMQgSIQHoQIAGlB1fmSgSIQAgESEAhQGIUIkAE0oMACSg9uDJXIkAEQiBAAgoBEqMQASKQHgRIQOnBlbkSASIQAgESUAiQGIUIEIH0IEACSg+uzJUIEIEQCJCAQoDEKESACKQHARJQenBlrkSACIRAgAQUAiRGIQJEID0IkIDSgytzJQJEIAQCJKAQIDEKESAC6UGABJQeXJkrESACIRAgAYUAiVGIABFIDwIkoPTgylyJABEIgQAJKARIjEIEiEB6EPh/LQtHt2fWwCcAAAAASUVORK5CYII="
 /><!-- --></p>
+</section>
+<section id="preference-for-b" class="level3">
+<h3>Preference for B</h3>
+<p>When users click on the interleaved results <em>with</em> a preference for 
B, the resulting preference statistic is <em>negative</em> and the confidence 
interval does <em>not</em> cover 0:</p>
+<pre class="sourceCode r" id="cb6"><code class="sourceCode r"><div 
class="sourceLine" id="cb6-1" data-line-number="1">z &lt;-<span class="st"> 
</span>interleaved_data_b[interleaved_data_b<span class="op">$</span>event 
<span class="op">==</span><span class="st"> &quot;click&quot;</span>, ]</div>
+<div class="sourceLine" id="cb6-2" data-line-number="2">z &lt;-<span 
class="st"> </span>z[<span class="kw">order</span>(z<span 
class="op">$</span>session_id, z<span class="op">$</span>timestamp), ]</div>
+<div class="sourceLine" id="cb6-3" data-line-number="3">boot_z &lt;-<span 
class="st"> </span><span class="kw">interleaved_bootstraps</span>(z<span 
class="op">$</span>session_id, z<span class="op">$</span>ranking_function)</div>
+<div class="sourceLine" id="cb6-4" data-line-number="4"><span 
class="kw">hist</span>(boot_z, <span class="dt">col =</span> <span 
class="st">&quot;gray70&quot;</span>, <span class="dt">border =</span> <span 
class="ot">NA</span>, <span class="dt">main =</span> <span 
class="st">&quot;Preference for B&quot;</span>, <span class="dt">xlab =</span> 
<span class="st">&quot;Bootstrapped preferences&quot;</span>)</div>
+<div class="sourceLine" id="cb6-5" data-line-number="5"><span 
class="kw">abline</span>(<span class="dt">v =</span> <span 
class="kw">quantile</span>(boot_z, <span class="kw">c</span>(<span 
class="fl">0.025</span>, <span class="fl">0.975</span>)), <span class="dt">lty 
=</span> <span class="st">&quot;dashed&quot;</span>)</div>
+<div class="sourceLine" id="cb6-6" data-line-number="6"><span 
class="kw">abline</span>(<span class="dt">v =</span> <span 
class="kw">interleaved_preference</span>(z<span class="op">$</span>session_id, 
z<span class="op">$</span>ranking_function), <span class="dt">lwd =</span> 
<span class="dv">2</span>)</div></code></pre>
+<p><img 
src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASAAAAEgCAYAAAAUg66AAAAEDWlDQ1BJQ0MgUHJvZmlsZQAAOI2NVV1oHFUUPrtzZyMkzlNsNIV0qD8NJQ2TVjShtLp/3d02bpZJNtoi6GT27s6Yyc44M7v9oU9FUHwx6psUxL+3gCAo9Q/bPrQvlQol2tQgKD60+INQ6Ium65k7M5lpurHeZe58853vnnvuuWfvBei5qliWkRQBFpquLRcy4nOHj4g9K5CEh6AXBqFXUR0rXalMAjZPC3e1W99Dwntf2dXd/p+tt0YdFSBxH2Kz5qgLiI8B8KdVy3YBevqRHz/qWh72Yui3MUDEL3q44WPXw3M+fo1pZuQs4tOIBVVTaoiXEI/MxfhGDPsxsNZfoE1q66ro5aJim3XdoLFw72H+n23BaIXzbcOnz5mfPoTvYVz7KzUl5+FRxEuqkp9G/Ajia219thzg25abkRE/BpDc3pqvphHvRFys2weqvp+krbWKIX7nhDbzLOItiM8358pTwdirqpPFnMF2xLc1WvLyOwTAibpbmvHHcvttU57y5+XqNZrLe3lE/Pq8eUj2fXKfOe3pfOjzhJYtB/yll5SDFcSDiH+hRkH25+L+sdxKEAMZahrlSX8ukqMOWy/jXW2m6M9LDBc31B9LFuv6gVKg/0Szi3KAr1kGq1GMjU/aLbnq6/lRxc4XfJ98hTargX++DbMJBSiYMIe9Ck1YAxFkKEAG3xbYaKmDDgYyFK0UGYpfoWYXG+fAPPI6tJnNwb7ClP7IyF+D+bjOtCpkhz6CFrIa/I6sFtNl8auFXGMTP34sNwI/JhkgEtmDz14ySfaRcTIBInmKPE32kxyyE2Tv+thKbEVePDfW/byMM1Kmm0XdObS7oGD/MypMXFPXrCwOtoYjyyn7BV29/MZfsVzpLDdRtuIZnbpXzvlf+ev8MvYr/Gqk4H/kV/G3csdazLuyTMPsbFhzd1UabQbjFvDRmcWJxR3zcfHkVw9GfpbJmeev9F08WW8uDkaslwX6avlWGU6NRKz0g/SHtCy9J30o/ca9zX3Kfc19zn3BXQKRO8ud477hLnAfc1/G9mrzGlrfexZ5GLdn6ZZrrEohI2wVHhZywjbhUWEy8icMCGNCUdiBlq3r+xafL549HQ5jH+an+1y+LlYBifuxAvRN/lVVVOlwlCkdVm9NOL5BE4wkQ2SMlDZU97hX86EilU/lUmkQUztTE6mx1EEPh7OmdqBtAvv8HdWpbrJS6tJj3n0CWdM6busNzRV3S9KTYhqvNiqWmuroiKgYhshMjmhTh9ptWhsF7970j/SbMrsPE1suR5z7DMC+P/Hs+y7ijrQAlhyAgccjbhjPygfeBTjzhNqy28EdkUh8C+DU9+z2v/oyeH791OncxHOs5y2AtTc7nb/f73TWPkD/qwBnjX8BoJ98VVBg/m8AADZuSURBVHgB7Z0J3E3V/v+/iQZzxookmhAyVJKoyFxSkSZ0aaJuUtItKbfoXiqaFA2KBglNSJokuchQhjQbkpK5okGs//p8/+392+c85zl7n+Ocs895ns/39Xqes4e1vmut9977u9f6rrXX2s9YEQoJkAAJhECgSAhpMkkSIAESUAI0QLwRSIAEQiNAAxQaeiZMAiRAA8R7gARIIDQCNEChoWfCJEACNEC8B0iABEIjQAMUGnomTAIkQAPEe4AESCA0AjRAoaFnwiRAAjRAvAdIgARCI0ADFBp6JkwCJEADxHuABEggNAI0QKGhZ8IkQAI0QLwHSIAEQiNAAxQaeiZMAiRAA8R7gARIIDQCNEChoWfCJEACNEC8B0iABEIjQAMUGnomTAIkQAPEe4AESCA0AjRAoaFnwiRAAjRAvAdIgARCI0ADFBp6JkwCJEADxHuABEggNAI0QKGhZ8IkQAI0QLwHSIAEQiNAAxQaeiZMAiRAA8R7gARIIDQCNEChoWfCJEACNEC8B0iABEIjQAMUGnomTAIkQAPEe4AESCA0AkVDS7kQJbxlyxbp0qVLRIn3228/OeCAA6REiRJy3HHHyYABA6Rs2bIRYVK1s2jRIrnzzjsFvyVLlpRTTz1VnnvuuVSpLzB6wuI0cuRIeeONN/JwLFq0qBQvXlxq164tV1xxhRxzzDF5wuT8AUNJO4Hvv//e2Bsl7l+VKlXMu+++m/K8/Pzzz+aQQw6JSLtly5YpTyfXFYbJ6Zprrom4PrHulWLFipk333wz1zHnyT9rQBl+hdSsWVPatm0re/bskV27dsnbb78tP/zwg1gjJddff72sXLkypTn68ssvZdu2baoTb9Fhw4aJvQtSmkZBUJYtnHr06CEVK1bUa/TXX3/JkiVL5MMPP5Tdu3dL9+7dZePGjYLac4GRPCaJB1JOwFsDuvDCCyP079y509SvX999A3777bcR552d7du3G7ylo2Xv3r1m9erVBnpiyWuvvebqnjRpUqwgxk9HrEjfffddrMNmx44dZt26dTHPxTqItNesWWP++OOPWKfdY/Zh1HLi108SzQP0BeGEcD/99JPZvHkzNgNJftfNG9lbA/r000+9p3Qb94w1OPpnX1B5zufyAVhaSpoJxDNASPr22293b7AJEyZobjp27Gisf8h07tzZDB8+3Oy///4G1fChQ4fq+d9//93069fPlCpVSuMWKVJEDdm8efP0PP6dddZZ5sADD3R1Yxs6Z86cGVgHAkIn4uFv2rRp5thjj1WdRxxxhLH+LdWFMHXr1jX27aznrD/L3HDDDQb5dOSbb75x9Vifhxk8eLApX768hrf+MNOzZ0/z66+/OsH1F8bV1hjdcqA5ecEFF5ipU6dGhMNOkDzkiWQP+HGyNVVja6fm+OOPd1miyXz//fcbW5ONUOl33SIC/73jZ4B69+6t6eL6RfOJpS+XjtEAZeBqxTNAeECtU9i9sT/++GPNUatWrfSYrY67DzXegjAAEPhxnLcijBAMEPat49LMmjVLw3j1OmHxO3369MA6EHDu3LluWpUrV3a3GzdurHree+89NZBOGmXKlHHDtGjRQsPg39dff+0ed4wYyufkHfFvueUWNzxqRqVLl3bjHHTQQe42DN2UKVPcsEHz4EbwbMTj9NtvvxnnWjjl8/7C4HjFCZvfdfOGdba9Buixxx7TFwT8Pa+//rrycMp95ZVXOlEKzC8NUAYupdcA4S2KGs/AgQNNnz59TK1atdyHqkKFCubPP//UHDk3Mm72pk2bqlFBTcj6AsxLL73kxkEtCG9F1C7q1aunxxs1aqQ6rL/APPHEE27YMWPGaPMID1VQHVDkNUDIz6hRo7TJgocETaITTjhB0zj00EON7UnSPCKM86DiQYJ4DRAMpWMIly9fbg4++GAND12OOE0P1P6Qd9REli1b5tb6wA5NuETy4Oj2/sbjhGvklMP6YMyqVauM9clorck5DsaOxLtuTpjoX68BcnRG/1500UVa1ui4ub5PA5SBK+g1QNE3lrOPtxwedEe8N7JTK3LO4WZEPDyY8Ek48uCDD7oPy2effaaHX331VfeYt8aQiA6vAerQoYOTnP7igXTKgGaKIzCUMKg417VrVz3sNUDdunVzgurvaaedpmFRw4KgaeO8+c8991w95vxDDc867w0MBySRPDg6on/z41StWjXNF5qKXv8TDL5T7iZNmrjq4l03N1DURhADhGuN2iFeHgVJ2Atm76JMim2eSPXq1bUnA+OArE9DTjzxRLFvWrE3e56sYCwIznvlq6++0l3bdJEzzzzTPYVeNUesk1hsDcHZzfObrI6TTjopQpejBwetgRPbFHLP24dFt5GXaAEDr9imlu5aZ7T+Io5tnuq2ba55g8rZZ58dsZ9sHiKUxNix/iexDnU9Yw2vWCPghqpRo4ZYn5fY2pusWLECL/KI3qlY182NHGdj6dKlYmuyqg/l37Bhg9x2220yefJksTVgsX43ue666+JoyK1TNEAZvl54eF5++eXAqWLgIG5mrzgPNh4InHcE25UqVdJddPPHk2R1RA+WdPQgLQya8+bHNqc0C4cddlierNjaTcQx78ONE7Zp5Z53jJJ7IGoj2TxEqcmza5uF7rHoa4ATTne48+sGthuxrpv3fH7beKngD2Kd/jr40DY/1bjDyMEQ0QDlR4/HU04AtaRoOeqoo8Q2O8Q2CcQ6pcU2dTSI7aYX21Mm3gcnOq6zn6yO6PxAjyPWISsY1esIxqzYJpWzG/Eb66H1BjjyyCPF9voIjE/02CjUBBYvXix16tSR/v37S7J58KYXa9v6tLS2ap3hOlIZRtExDmvXrhXrj9JoyEd0eaI5xdIf9NimTZu0RoTwv/zyS9BoORGO34Jl+WWKrhkgu506ddJcwwCNGDFCH1I8HDiON2+DBg0ExiieJKsjOj9oHjpNx4kTJ6phRLpoiuEBRo1p9OjR8bIS8xwe9DZt2ug56MIASjumRj8nsUMRxI5pEjtkQcubrjwg8ebNm2seYAT69u0rMDxoJqHJ7EivXr2cTfc3mpN7wmcDZbX+KP1Dbcf2irnXG1Ftr6KPhhw7XZAcWtlaFq8TOnogYn55dpyZtvmSJwgG7Vkj4zpB0eXr7R4fN26cGyc/52oiOrxOaPRGRcsLL7wQ0Q2P3ilrQDR/2HYGSXqd0EOGDIlQg+5s++gYjB9yBOFts84tp32o3W1b4zDegZVB8+Dojv7Nj5MdRW5OOeUUN11vHpDf9u3bR6iKd90iAnp2gjihkZb9Fswdd+WJntObrAHl2AsD2UX13hoFwbB9OwZI8HbGnx2Xo45KO6DPt1Sp0OEkcvHFF4sd3ChwFqMpguYhnOu2p0t9FvANJSP4bAVNrWbNmqlex6+F5tnzzz8f8YFvuvKAGhw+l8FnLFWrVtVPaFAWlA+1MjvEIJmi+cYBRzSnHT+QHbYh9ltBKVeunG/cXAqwH8xnLmWYeY0kgMtnu4S1qeP4giJD+O+lQoeTCnwU69ev1y/8HX+Jc25ffm0tSmyNSL+TglM72ufi1Z2uPCAN+INQLqfZ6U2X24kToAFKnBljkAAJpIgAm2ApAkk1JEACiROgAUqcGWOQAAmkiAANUIpAUg0JkEDiBGiAEmfGGCRAAikiQAOUIpBUQwIkkDgBGqDEmTEGCZBAigjQAKUIJNWQAAkkToAGKHFmjEECJJAiAjRAKQJJNSRAAokToAFKnBljkAAJpIgADVCKQFINCZBA4gRogBJnxhgkQAIpIkADlCKQVEMCJJA4ARqgxJkxBgmQQIoI0AClCCTVkAAJJE6ABihxZoxBAiSQIgI0QCkCSTUkQAKJE6ABSpwZY5AACaSIAA1QikBSDQmQQOIEaIASZ8YYJEACKSJAA5QikFRDAiSQOAEaoMSZMQYJkECKCNAApQgk1ZAACSROgAYocWaMQQIkkCICNEApAkk1JEACiROgAUqcGWOQAAmkiAANUIpAUg0JkEDiBGiAEmfGGCRAAikiQAOUIpBUQwIkkDgBGqDEmWV1jL1798q2bduyOo/xMvf555/LW2+9JVu3bo0XLKvP/f7777Jr166szmO2ZI4GKFuuRIrysW7dOmnQoEGKtGVezZAhQ6Rt27Yya9aszCeeohRHjx4tgwYNSpG2gq2GBqhgX1+WjgSymgANUFZfHmaOBAo2ARqggn19WToSyGoCRbM6d8xcwgTKly8vQ4cOTTheNkd4/vnnE8repZdemlD4VAdu1aqV/Prrr6lWWyD10QAVsMtaqlQpCfsBLGBIEy5OvXr1Eo5TWCOwCVZYrzzLTQJZQIAGKAsuArNAAoWVAA1QYb3yLDcJZAEB+oCy4CKkMgsYhfvJJ59IkyZNUqk2rq5EncTxlK1Zs0ZPz507V/bs2RMvaNaeW79+vfzxxx9Ss2bNrM1jtmSMNaBsuRIpysePP/4o3bp1S5E2qkmGwKRJk+TRRx9NJmqhi5PzBsgYI5s2bcrpb4cK3V3HApPA3wRy0gB9//33MnDgQKlevboccMABUqlSJcH4lzJlykj9+vXlpptu4jgM3uIkkAMEcs4HtHbtWmnWrJnst99+0qVLF6lRo4aUK1dO9/EF9erVq2Xy5MkyZcoUeffdd9kOz4GbkFksvARyzgCNGDFCaz7vvPOOHHjggTGv3LBhw6Rdu3Yyfvx4wdfVFBIggewkkHNNMPTwdO/ePV/jA8zFihWTnj17ysyZM7OTehpzVa1aNVm6dGkaU6BqPwJ9+vSRu+++2y8Yz1sCOWeAmjZtKh999JHvxXvvvfekSpUqvuEKWoAiRYrIIYccUtCKlVPlOeigg6REiRI5leewMptzTbBLLrlEYIQ2btyo3zxhrAUc0Hjw4APCOBKMS5kxY4agmUYhARLIXgK+BujDDz+UqVOnapMGPUxhy4knnijLli2Tq6++Wnr06CGYgjRa8DUyZtRr0aJF9CnukwAJZBEBXwNUtmxZnaN31KhR2sUN3wq+tq5YsWJoxTj66KO1h+vPP/8UTEGKWs/u3bvl8MMPl6pVq2qNKLTMMWESIIHABHx9QHXr1pXPPvtMPv74Y2nevLnONQPfynnnnSevvPKKPviBU0txQIwBgjFCjQe1nb/++kubYbk6hD8VeNAMveOOO1KhijqSJAD/I4aBUPwJ+BogR0Xjxo3loYcekg0bNmiTDH4X1IZQ6+jXr59gNYNMyP3335/nAbv33nt1LBDyeOyxx8pxxx2X05Oa7wvHn3/+WSZMmLAvKhh3HwmgpzZIR8k+JlMgogc2QE5p0eRBN++SJUsENzsGAi5YsEBq166dxzA4cVL5C//P4sWLXZXjxo2T2267TQcnPvXUUzJy5EhtgqGGhhuBQgIkkL0EfH1AyPrmzZvlpZdekueee07mz58vlStXlssvv1z369Spo6V7+eWXpWvXrtK5c2dp2LBhxkr8xBNP6Jff3h4v1MiQB/itnnnmmYTygtHTDz74oG8cNEsffvhhHfDoG5gBSIAEYhLwNUBz5sxRHwtid+zYUV5//XV96IoWjYzavn17TQDd45mU7du3S9++ffMkeeWVV8rYsWPzHPc7gCYc4vrJgAEDJJqBXxyeJwESiCQQaUUiz+le8eLFZfjw4b49X/gsAlNBoHaUbvntt9/U+Y0Rz506dRJ8nBotaKZVqFAh+rDv/hFHHCH48xN87oH5l7NNDj74YDn11FOzLVuFKj+4f3AdKP4EfH1AcOzecMMN8vbbb8u3337rasRwc6/jGbWBTBgffIQ6e/ZsffhPPvlk9QfBOe58foDJoK699lp1xKJJWNgE1+DFF18sbMXOqvLiI2ncgxR/Ar41IKho06aNPvR48OF0xhw8n376qZxwwgly3333aS+Yf1KpCYFmFXw8cDDD6OAXxu+bb77RJYmnT58ujz/+uDbLevXqlZpEqSWnCCQ6QyNXEQnv8voaoK+++ko++OADWbFihXZxI6uohaCbETWPf/3rX4LaEMbkZEKQDkZD4w/DABzBGCDIOeecIx06dNABic45/pIACWQnAd8mGHqFnPE10UW47LLLZNeuXToaOfpcpvcdh7AzGjrT6TM9EiCBxAn4GqDjjz9eFi5cKN99910e7dOmTZP999+/UH51ngcGD5AACSRMwNcAYXWFQw89VMf9YIg5nLxff/21OjoHDRqksxLS458w97RFwIuiUaNGadNPxf4E4IMcPHiwf0CGEF8fEOY2mTdvnvpVWrZsGYHswgsvlNGjR0ccS/cOmoMYBBhEzj//fB0sGSRsQQmD7+C2bNlSUIqTk+WAW4Jrwwe7dL4GCGowrgGfQKD2g14nNLvQNDvqqKOCpZLCUPjU4oILLhB8CY8aGOYByk8wqJBCAiSQvQQCGSAn+5jqAn9hyumnny5YtA69YHjb33zzzWFmh2mTAAnsA4FABghNMHxXhRoQah7RsmjRouhDad1HzQZz7t51113Su3dvzv+TVtpUTgLpI+BrgODUbNu2ra65hSH+pUuXTl9uEtDsfHBamOf+iYUL06QMHTo01ikeyxABzE9FH1Aw2L4GCOOA4GeBDyibJjuHH+rMM88MVspCFArfp3Fkb7gXvF69euFmIIdSz9+D+3ch8NkF/D7ZZHxyiC+zSgIkEIeArwFCLQMfoa5cuTKOGp4iARIggcQJ+DbBMA4IS+FgzuWLLrpIa0No/njllltu8e5ymwRIgAQCEfA1QPjqHbMdQjAjYiyhAYpFJZxjv//+u47Vwgh2SjgE0Fv8xx9/CNaso8Qn4GuAMBXHjh074mvh2awhgEnhunXrpksVZU2mCllGJk2apENWHnjggUJW8sSL6+sD8qrcuXOn9oahixEWnkICJEAC+0IgkAFClRLffZUsWVIXJ8S3WAMHDtRRyPjuhUICJEACyRDwbYJh5PO5556ri/7hO6zbb79d08EnEZhxEEvzJDP5ezKZZZzMEEh0RsHM5IqpFEQCvgYIy91g0vcvv/xSR0PfeeedygEfhGJUNNZnx1ghzJJIIQESIIFECPgaIEzJirW/ypQpk0fvSSedJD/88IM6PMP4Mj5PhnhAqlWr5k7QTxzhEMAUxfxEKBh7Xx8QDAvmf960aVMejRMnTtQJ4bFWPCU7COCzGY5aD/daYOxciRIlws1EjqTuWwM666yzdPAhFh7s37+/7N27V1AreuONN3T1CQxSzNSE9DnClNkkARIISMDXAKHn65VXXpGePXvqiGjoxWT0EKy/HmQZYw3MfyRAAiQQRcDXACE8vu7FnD9YbRS1H9R44BeqVatWlLrc33322Wd1jiG/kmAZIHwfxxHHfqR4ngTyJxDIACE6fAtwOuOvIEv37t0DTWfRrFkzNcLZxmLr1q2C4RKYsI0SDgEs3rBt2zadOjicHOROqr4GaP78+XLrrbfGLRFWTC0oguEEzhpj8cqUrcMOMC5rwoQJNEDxLl6az2HedAzexVAVSnwCvgbowAMPlMMOOyxCC25yrAsPyJiZkEICJEACyRDwNUANGjTQNcCilWPw4fXXX58Vq6JG5437JEACuUHAdxxQfsVAEwQrUmCqDs5/mx8lHicBEohHwLcGFC/yxo0bdcQnputAdz0lfAJYpRaLB1DCI4B19LhacDD+vgZozZo18vrrr0dowzBz9LY88cQTukAhR0JH4Al1p3LlyjGbzKFmqpAl3qVLl0JW4uSL62uAMPXGgAED8qSAoeZYHJADEfOg4QESIIGABHwNED7B4ORjAWkyGAmQQEIEknZCJ5QKA5MACZBADAK+NaAgAxG9ep9++mmpUaOG9xC3SYAESCAmAd8aEKZ2gEd/zpw5UqxYMZ2StWLFioLVMj744AP9RAMDFZ2/IKOIY+aEB1NCAEtpN2rUKCW6qCQ5Ao8//rgMHjw4uciFLJZvDQjd6xhajpkRMTWHI+gJu+6662T58uXsdXGgZMEvrsuWLVuyICeFNwuYJ51j44Jdf18DNGvWLMFoaK/xgWosTjho0CCdKwizIkZ/rhEseYYiARIozAR8m2CY93nhwoUxLTrWoMJX8hyEWJhvIZadBJIn4GuAWrVqJVht84orrlC/D6qWP/30k0ydOlUw4Kpr165SqlSp5HPAmCRAAoWWgG8TDJPRY07oc845RwceeknB+IwZM8Z7iNshEyhfvrwMHTo05FwU7uTx0qYPKNg94GuAoKZ+/fryzTff6AyAcEhjFHTdunX1M4xgyTBUpgigNnrppZdmKjmmE4MAZhClBCMQyABBFbrgjznmGPX5YJwP9ikkUBAIJLoQIw186q66rw8ISWXz0syYlwhLBuHjWAoJkEBuEfA1QM7SzFgZFXMNFy9eXEuIpZmffPLJUGZExEqtWJu+evXqOkF+pUqVBL4P+KvQXLzpppvYBs+t+5C5LaQEfJtg2bY089q1awUTwmNCNPTCoTlYrlw53UctaPXq1TJ58mSZMmWKvPvuu1KzZs1CdWnRYwk/HVfrCO+yo8WAD7gL272XDHFfA5RtSzOPGDFCaz4wjJivOpYMGzZM2rVrJ+PHj5chQ4bEClJgj2FsVrdu3XS57AJbyCwv2KRJk9Rt8cADD2R5TsPPnm8TLNuWZsbbHUvn5Gd8gBQOciykOHPmzPAJMwckQAL5EvA1QN6lmV988UV3aeY77rhD8JfppZmbNm2q45LyLdHfJ7A2E2dq9KPE8yQQLgHfJli2Lc0MgwcjhPmo0R2KdjYc0PgkBD4gTCGLbtUZM2boB7Th4mXqJEAC8Qj4GqDt27frAESMhl6xYkXoSzNjGthly5bJ1VdfLT169NAaWXQBMRIVH9G2aNEi+pTvPj6sxTS0foKJ+Hfu3OkXLCvOJzrOJSsyzUwUCgK+Bmj69Oly2WWXCeaZyZalmY8++mjt4cIQgXXr1mmtZ/fu3XL44Yfr1/moESUrMG5wdPsJhgJg/FG2SbVq1WTp0qXZlq1ClZ8+ffroajGFqtBJFtbXAFWoUEFV441ftWrVJJNJT7QDDjhAYIzwhzlYVq1apc2wsmXL6nQhyaTapk0bwZ+fYOkbjEPKNkFTFJPIUcIjcNBBB4WXeI6l7GuA0OSB36V58+bSuXNn9blE90D1798/Y8W+//77Bc3Cu+++203z3nvv1e52Z/J8+IVGjx4trVu3dsNwgwRIIPsI+BogVOdfffVVzTl6wWJJJg0Qmkjeps+4cePktttuk5YtW6qhxLr1yOd5550n8+bNy/MFf6z88xgJkEA4BHwNUNu2bbPa2YrFETHqFwMTHenXr580bNhQRo0aJc8884xzmL8kQAJZRiDmOKC//vpLJyHLsrzGzA6aY3CSR8uVV16pE6hFHy/o+xiKgPFZlPAIYAwaPgWi+BOIaYCw2mn00jr4vgVO3myQ3377TdDrBenUqZOgRypaFi9eLI4DPfpcQd5HE3TChAkFuYhZXzaM1sewFYo/gZgGKFY0fAl/+eWXxzqV0WP4CHX27Nk6DezJJ58sMDQPPfSQ2/UMQ3nttdfqQ4gZGykkQALZSyCwAcqWIowdO1aNDdZeQlc4er6wFhlmbIRg3BLOoQnWq1evbMk280ECJBCDgK8TOkacUA9h7A+GBuAPH5w6Ar8VBHNXd+jQIevGLDn55G/uE/AbWY5aOXxxTjjOoJj/Nc85A5RfUZwVWTEaujALVrFFzZASHgGMxMeLkuJPoMAYIP+iFo4QlStX5kq1IV/qU045JeQc5E7y+RogDPY74YQT3JJgoit8fOk95pzER6oUEiABEkiUQEwDhG+rOnbsGKELK2JQSIAESCCVBGIaIIytwR8lOwk4zs3szB1zRQLBCeRcN3zwojEkCZBAthOgAcr2K5Rg/rZs2SKDBg1KMBaDp5IAVmPByiwUfwIxm2D+0RgiWwns3btXfvnll2zNXqHIFwbHYnkkRxJtMhemcUOsATl3CX9JgAQyToAGKOPImSAJkIBDgAbIIcFfEiCBjBOgDyjjyNObIJZR4iwA6WXspx2Ddb0+IL/whfk8DVABu/r4Fuy0004rYKXKreJgZRJKMAJsggXjxFAkQAJpIJDzBsgYo5PUY/oDCgmQQG4RyMkmGKZgxSyIL730kk7H6swFVLp0aV2rCyujDhkyROAPyQVJdJxILpSJeSSBIARyzgCtXbtWmjVrJpiatUuXLjp3dbly5XQftaDVq1frKFRMCo4RqVgjrDAJVosFI348HN5Vx2h0vBQxNQolPoGcM0BYNrm6XZEUy/BEL5DoFHXYsGHSrl07GT9+vNaEnONBfhctWqQ1K7+weMgxRUksCbNGgxVsH3nkEcHCApRwCCxYsEBnRIy1WkuQHCV6/+TyyOmcM0BYcaBHjx75Gh9c4GLFiul0rQ8//HDCBgjL6gZ5c51xxhlqCGPdUGHeEGvWrNH10MLMQywmQY9NmzZN5s+fr7Xcbt26BY2WVeE2btwoWBwhV69BJmHmnAFq2rSpLnmCSefjCdZmqlKlSrwgMc9hDEesSddiBuZBEiCBfSKQcwYI69TDCOEtgzcMfDyYg7dIkSJa7UUNAFXYGTNmRKyWuk+UGJkESCAtBHLOAGE1DKwPf/XVV2tTDF9/Rwt6wWbNmiUtWrSIPpV1+/BTLV++PN/mXKIZhgMUixM++uijGnXDhg2ya9cuwSyX6ZCffvpJkGatWrVSov6rr75SPW+99ZbqRccCej3r1q2bEv3RSjBzwNdffy0NGjSIPpX0/ty5cwUr9uIaYBFNTFl80kknJa0vXkQs0Am3RK4uQb6fHUdj4hUwm8+hx2fdunWCWg8uBFbEqFq1qtaIsjnf3rzBn/XZZ59JKicyxyVFLyEExg0PcbqM8Zdffqn8W7du7S1W0tvoPcIfVpVArRbOfrxwsNxSOgQ16Tlz5miPajr0wxDBr5WsQ9ovT/jkAzV+GLqcFBggSngE7r77bmMnEEtbBmyPmOnbt2/a9L/wwgvm4osvTpv+N99807Rt2zZt+m2PlbEr7KZNvzXQxg6JSJv+zZs3G+uCSJv+dCvO+ZHQOWn1mWkSIAElQAPEG4EESCA0AjRAoaFnwiRAAjRAvAdIgARCI0ADFBp6JkwCJEADxHuABEggNAI0QKGhZ8IkQAI0QLwHSIAEQiOQ0yOhQ6OWwoTxKYMd7BXoC/xkksVnElgoD6PE0yEY6YvPGY444oh0qJdff/1VZ7w86qij0qIfI4jxqUe6PlXBaH3MUXXcccelJf979uyRL774QmrXrp0W/elWSgOUbsLUTwIkkC8BNsHyRcMTJEAC6SZAA5RuwtRPAiSQLwEaoHzR8AQJkEC6CdAApZsw9ZMACeRLgAYoXzQ8QQIkkG4CNEDpJkz9JEAC+RKgAcoXDU+QAAmkmwANULoJUz8JkEC+BGiA8kXDEyRAAukmQAOUbsLUTwIkkC8BGqB80aTmBL7zSpfgOyN8C5TLkgyfIHFiLdeUi5yClDW6XMnEidaRqX0aoDSRfvbZZwXLNxcvXlzsqgsye/bshFK6/vrr436gig8c8YHpzJkzE9KbLYGT4YO1wrp06SKlSpWSOnXqyMiRI/MU5+233xasHXfggQdKxYoV5YYbbtB10fIEzPIDS5cu1YU3DznkEKlRo4bY1VN8c/ztt99K//79pVq1anLooYfK5ZdfrmvEeSNOmTJFjj/++Dx/OB6KWGtJSTGBDz74wNh1rYxdm97YG8lcc801xq45bz799NNAKdllpY1d18tUqlQpZnh7oxm7fDSqVsauORUzTDYfTIaPXR/MWKNi7IKUxq4db+677z5TtGhRc//997tFteuraZhOnTqZDz/80NiFAY01VqZ3795umFzY2Llzp7FGR5c7WrJkiRk3bpyxLzJzzz33xM1+8+bNjV0A0UyfPt1Yg2KsoTFNmjSJiGMNlLEzF5g77rgj4m/x4sUR4TK1g6kgKCkmYFcJNXbZ6AitMBj/+Mc/Io7F2rGrmprq1asb+4aPaYBGjx5tSpQoYez0DjlrgJLhAyNil+E2tsnpYjv//PNN/fr13f17771XDdCmTZvcY9ddd53y8sZzT2bpxp133mlKly5t7FQhbg6HDBliKlSoYOxChO4x74atHer98PHHH7uHX3zxRT2GtckcadmypenZs6ezG/ovm2AprneuX79eVq1aJZ07d47QbN/Kul59xMEYOzfffLNYYyVdu3aNcVZk6NChgubZjBkzYp7P9oPJ8rG1SbG1Gl0t1Snjjz/+KMWKFXN2pVy5crpCrneVUKyyWrJkSXelWDdwFm+gqdmuXTuxtWY3l7h/7CKEYg2Me8y7cfbZZ+u8Q40bN3YPY9VXiJcRVpl1lqHOBj8ZDZB7uVKz4axtXqVKlQiF2LdvZol30bGe/aRJk+Txxx+PiOvdgW/Avul16WLv8VzZTpYPHsbDDjtMi4m13P/973+LbYqpMXbKfu6554qtPYqtaQoeYts802WL//nPf+aUAUL5Yt0/KCeMbizBUtwoO2Tbtm0yceJEGT58uDhMcBxxcQ9+/vnnctZZZ6mBa9iwoRp2nA9DioaRaEFOE7MDQuxyuRHFhDMRPVaYoRDO0WjZsWOH9OrVS0aMGJHn5vOGjRXXez7bt5Pl45QLzne71LHuduzYUS655BLnlDpesU766aefLu+8844e79Chg9x6661umFzYsM3wPPdP2bJlNetOrSZeOVAbsj4dwT2HmqMj1gepmx999JH06dNHjdCYMWP0F+Hr1avnBM3YL2tA+4AaU3muWbPG/cP0pNYxqhrxRool6DqPJf369dOeCevriHU6J4+lko8DAAYYTVw8OCtWrBDrZBU0syCvvPKKnHnmmVoDWrRokYwdO1bf9q1bt87K4Qq4X7z3D3hB0GSKvn+c/d27d2uYeP8mTJgg6A085ZRTxPrIZOHChRrc+t6Uie0EEOvMl0GDBgk4oVaOGmUoEroXKocz4PRE2Qunzj44CtFrgX2vMxBFhPMYx+FkjhY4EO0NZp5++mnz/vvv6x8chfatp9vr1q2LjmK+++471ZfNvWCp4pOn8H8feOGFF5QBerwgrVq1Mta/8ffZ//9jjZKGmTt3bsTxbNixD73mzbl/wAsCZ/uAAQMismj9Pxr2ySefjDgebwf3GnrPrM8wXjBjh4sYOyd23DDpOskmmL36yQrGZqC97Qice5UrV9bdH374wTmsv2h/214MHcMSccLufPLJJzoxPXwX0YI3Otry9oaMPpX1+6nig4LOmTNHmxR169Z1y42mBgRv+GbNmsm8efPycGrfvr3WKKwBktNOO82Nmw0b8M9UrVrVzQqaTBCM4Yn29Tj3kzVObnjvBnxraJ6itucIxkthDJpTA7IvMtmwYYPWGp0w+IV/zeuo9p5L+3a6LFth1osudIxX8Yp19hnbs+U95G5b/4/BOBfv34033qjdrjiG89GSCzWg6Dw7+4nyQTxrePLUbp555hmtFbz22muqGuNeLrzwQicZ/cWYKvsQmeeeey7ieDbvoBveNjWNbVq62fzPf/6jXfOxatAIZI29KVKkiPF2uW/dulWHJVx22WWqB3pR07ZNWFevfYEaa3yMHbToHsvkBscBpYH2Y489pgMPJ0+erMZj2LBhuv/NN9+4qdl2uhqkXbt2uce8G2jO5TcQEeFy2QAlw+eRRx5RQ4IHzdYODJpfGKxnHafG+tUUnRPGOvKNdfabN9980zRq1EgH3uX34HqZZ8u2raWoUejbt6+BEbGj6I0dYmBGjRrlZtHWePT+gYGF4N6yww1MmzZtzPLly9UFcM455+hgTbgFIHb5HnPwwQcbW3M0ttZtrEPbwDhhgOfKlSs1TKb/0QClgTjeXBhxijcL3r4YeGc/PYhIyY730XP5PRgF2QAlywejd/GwgCn+bHPDWCeuy9U6U3W0MB4yJwxGBtuxL26YXNmAbw8DD1EOGJ+rrroqoka0YMECPef1CcGXeOSRR7plt59k5BkpD7047vCB78c2XUPDwnXB7JVIl9hRq4KFB/FtDiUvgWT4YJFF5zs4O1o4r1J7BMMdEAYDE/GXq2KtgtgmuPqJnN5Vv7IgDnw9+++/f4R/yRsPYTAg1H4u5PosveczuU0DlEnaTIsESCCCAMcBReDgDgmQQCYJ0ABlkjbTIgESiCBAAxSBgzskQAKZJEADlEnaTIsESCCCAA1QBA7ukAAJZJIADVAmaTMtEiCBCAI0QBE4uEMCJJBJAjRAmaTNtEiABCII0ABF4OAOCZBAJgnQAGWSNtMiARKIIEADFIGDOyRAApkkQAOUSdpMiwRIIIIADVAEDu6QAAlkkgANUCZpMy0SIIEIAjRAETi4QwIkkEkCNECZpM20SIAEIgjQAEXg4A4JkEAmCdAAZZI20yIBEoggQAMUgYM7JEACmSRAA5RJ2kyLBEggggANUAQO7pAACWSSAA1QJmkzLRIggQgCNEAROLhDAiSQSQI0QJmkzbRIgAQiCNAAReDgTmEnsHv3brHrrMsvv/xS2FFkpPw0QD6Y69atK/vtt5/7Z9cmlxNOOEGuvvpq2blzp0/sxE/b9c3lscceEyxbHFQ2b94sdu35oMGzNtzUqVOV85YtW0LJ43PPPSdlypQRu1663H777aHkobAlyqWZfa44DBCMTr9+/TTkr7/+qut1P/zww9KwYUOZM2eOj4bETk+cOFEuvvhiQTolSpQIFPmCCy6Q3377TWbMmBEofLYGggFCWWBQy5cvn/FsHnfccVKpUiUZM2aMrpkeRh4yXuiQEywacvo5kfwRRxwhl112WURejzzySLn22mtl9erVctRRR0Wc25cd1IASFcRBLY2ybwR++ukn6dGjh9SuXXvfFDF2YAJsggVGFRmwQoUKemD//fd3T2zcuFF69uwpVapUkYoVK0qnTp3Un+AGsBvxwrz++usyaNAgDd6sWTOZMGGCbi9cuFBatGghpUqVUmOHNLZu3arnrr/+epk9e7bMnTtXGjduLD/88IPcddddMnjwYLnlllv0jT5w4EAN++2338r5558v1atXV12NGjWS559/Xs/h3//+9z/V8fHHH8upp54qKONFF10kS5YsccPcfffdAn0PPvigNlVq1qwp/fv3z9McXb58ubRu3VrKlSsnxxxzjNx2220C/4pXpk2bJm3atNHaDliBTTwJkr/8yo4aZZ8+fbTsKNd5552nNVmk99VXX2m5f/75Zxk9erS7jXN+5cB1wnUDL7B48803Ec03HjiimQf+DRo0kLJly0q7du1kzZo1Gt/5h2uBa4Ca2ZlnnimPPvqoeF9SfvmLd+84aYT6ayhxCVh/jznnnHOMbeLon33wzYIFC8yJJ55o2rZt68b9888/TZ06dYx9uM0zzzxjJk2aZE466SRjfQrm+++/13B+YeyDYK655hpjbwjzxBNPmJUrVxr74BjbFDDt27c3L7/8snnkkUfMYYcd5qb9zjvvGNsUNPXq1TPjx4/X8PYtbg4//HBTrVo1c/nll5sRI0aY7du3G3uTG/vAaP6gv2nTppqWvYk1fzNnztR9e7ObkSNHGmvYNN2qVasa2yzSML169dIyWaNirAExkydP1rSQjiOfffaZOfjgg401iGbcuHHGNmmMfehNt27dnCBm6dKlxjZtzRVXXGHeeOMNY2uT5oADDtD0nbTcwH9vBMlfrLLbB9acfPLJBuWyhtnYpp5p3ry5sS8JY/1NBtcU7JBn5BHbf/zxhwlSjuLFi2v5oQ/3ySeffBIoHjjiOtqatBkyZIjyxvUBM0fsC0O5dezY0bz22mvGGldlZF9MGsQvf373jpNOmL8SZuK5kDYMEAxC9F/lypXNtm3b3CI89NBDxtaGzJdffukew82NY1dddZUeCxLGvhE1Ldw8EBg7pG1rIbqPf3iAbrrpJoMHC2Lf5moodMf+w0Nom2Tmiy++cA4Z+2Y2xx57rFm7dq17DAYPuseOHavHnAf8nnvuccPA8KKsN998sx7Dg1OkSBGzatUqN8ysWbNUj33b6jFby1KjaR3pbhgYKqS1ePFiPdayZUs1Am4Au9G5c2cN42eA4uUvVtlffPFF1Wt9ZG5yu3btMtbHZv71r3+5x0qXLq3G2jkQpBwwQHgB7Nmzx4lmgsQDR9wbX3/9tRvvgQce0HzCIEKsL9Acf/zxEbptLc7l5pdOkHvHTTykDfqA7FPhJ/Xr11d/D8LZN6OsW7dO7E0ttgYh06dP12YRqsq2xqPNDUcfmh/2zSvz5s3TQ0HCOHGdX1urkpIlS2rTqXfv3nLuueeKfVD1zwkT6/fQQw8Va3DcU7a2JtYg6T66mK0BEXuDijUmYh9GNxw24AR35KCDDhLEtTUW55A2GeyD4e6fffbZYh9EbaqBARzzp512mlhj44ZB7xLSmj9/vjrvbU0hT08TmhqvvPKKGye/Db/8RZf9ww8/1N4tNGGdawHduK7e/ej0gpQDcdAMQ9kcCRrP1iy12ebEQ+8bBNfnkEMOETDCtfbqRhPMEb90rDFO6t5x9GfilwYoAGU4mdHt7hVbAxE4op966imxb2R1RsP3Ey3w3aBXBQKHtV+Y6PjoCXv33Xelb9++6h+Cjwg+lfvuu0+NUXR4Zx83d7TAb4PeO/iCbHNDDah98aEWHBE0Oi78D/AzORKrDAhj3+ZqzNCLBb8I/qIFPg74r9DVbmtWEadtszFiP78dv/xFn0eaO3bskNNPPz2PSvjDYgmMsl85nHjopHAkkXjwE3oFva0QW5tSP4+tTQuMaSwJkk6y906s9NJ17P/MdrpSKKB6bftdUDvB2xWCmwkPVrTgbea82YKEiY6PfdSiUHvCg2R9QDosAG9GGLT8xOscRxgYyhtvvFGdrx999JFYn5BYv4JGjzZAeFi9AmPh7emLPo+weFhr1Kihhg3GDQYaztLov+HDh+vbHbWRaF5BB/9Fpx+dv+iyozsdLwvnwfbmCcY4lqAMfuVw4nnTSySeEz/WL2o9tkmo18l7HmPP0NEQNJ1k7h1veunepgFKkvCmTZu098Rp5sAYLVq0SKxfyNWIGx21F+sg1mNBwjjd6YgLgbE444wz9AHHQ4SakHWS6oO9bNkyDYM4eLjiCcYIoXsZNSf02BQrVkyr+DA+0XEdowp9yMcHH3wg1unuqv/0008jRgqj3OhlQpMGeUE61gmv4bGPP7zNrd9He9qwj7DvvfeeqxMbKGsQ8ctftA5wt74vbXI6+UG5L7nkEuURHR77COdXjlTGi6UL6aN30yvWT6RNWLgC/PIX5N7x6g5l214IShwCcEKjxwuOWvw9/vjjBk5QOHTtQ2xszURj2y5kY9/qpkOHDsbWVIwdU2KssTBwUtru48Bh0CNkbwRju4ON/STA2DeeOoHtw6KOX1vrMdYXZIoWLappQHH37t2NbXYYOJHhvIYjtkmTJpqm82/UqFHq9LQPvUFvnPV9aI8d0kIvDMRxQtuajLFGRXuHbDe/sb4szQvCwHmKOOgtQhnhjHZ6BK0hQxB1kiMM8oGePDjQ0eMGlrYrXsOgBw1OWPTq2e5vY5tryg/x/JzQ8fIXq+zoLEBPou3u1t4k9EoOGDBA0/c696Od0HD2+5UD1xfOY68EiQeO3h4vxH/rrbc0PfR+QdDzhfTRI4leTDCzTTJjXyJ63i+dIPeOKgrxH3vBfOBH94LZqrF2n1rHq0EXuFdgaOxbSW8a+wbVbnk81F7xC4OHBQ80bjz0eEBeffVVY526anRwHMbm7bffdtUiH3h4cO7999/XBz/aAEEvests1V314Ea2nx4YOw7H2PEnqssxQHZcj7HOZ9WH8thanJsWHpxatWoZO25He9pghDEcAfq9AmMNw4U8IU0YZhg1r1h/lHZhIwy66f/73/9qeD8DFC9/sQwQ0kS3PwwQ0sK1Qc8VXiZeiTZAOOdXjlgGKEi8IAYIevAicq4t8o5r6PSQBknH796BjjCFBigN9G3zzK2d5KfeLwweaKe24Oiwvg+zYcMGZzfi1zaVdExLxMEYO3grfvfddzHO/F8NCLUsdKGvX78+Tzjvg4MyoPYST9Dtbx2m+QZBvlFjxK+fOAYyXv78dICr9aH4Bctz3q8ceSL8fSDZeF59YINaUbSR94bxSyfevePVk+lt9oLZ10qqBSNt/cQvDEbGRot9E6pjMvo49uGzQLe/n6C7HH9+gh6ZWL1d3nh+ZUBYOxjSGyXPNvIN31aiEiR/sXTG4horXPQxv3JEh3f2k43nxMcvGHk7AbznnG2/dOLdO46OMH7phA6DOtMkARJQAvwanjeCSwA9WbZ5IxhkiF6yWGKbZTpViDO0IFaYdB0Lkr90pU296SFAA5QertRKAiQQgACbYAEgMQgJkEB6CNAApYcrtZIACQQgQAMUABKDkAAJpIcADVB6uFIrCZBAAAI0QAEgMQgJkEB6CNAApYcrtZIACQQgQAMUABKDkAAJpIcADVB6uFIrCZBAAAI0QAEgMQgJkEB6CNAApYcrtZIACQQgQAMUABKDkAAJpIcADVB6uFIrCZBAAAI0QAEgMQgJkEB6CNAApYcrtZIACQQgQAMUABKDkAAJpIcADVB6uFIrCZBAAAI0QAEgMQgJkEB6CNAApYcrtZIACQQgQAMUABKDkAAJpIfA/wMejGUb+LdPMwAAAABJRU5ErkJggg=="
 /><!-- --></p>
+</section>
+</section>
+<section id="references" class="level1">
+<h1>References</h1>
+<ul>
+<li>Chapelle, O., Joachims, T., Radlinski, F., &amp; Yue, Y. (2012). 
Large-scale validation and analysis of interleaved search evaluation. <em>ACM 
Transactions on Information Systems</em>, <strong>30</strong>(1), 1-41. <a 
href="https://doi.org/10.1145/2094072.2094078";>doi:10.1145/2094072.2094078</a></li>
+<li>Radlinski, F. and Craswell, N. (2013). <a 
href="https://www.microsoft.com/en-us/research/publication/optimized-interleaving-for-online-retrieval-evaluation/";>Optimized
 interleaving for online retrieval evaluation</a>. <em>ACM International 
Conference on Web Search and Data Mining (WSDM)</em>. <a 
href="https://doi.org/10.1145/2433396.2433429";>doi:10.1145/2433396.2433429</a></li>
+</ul>
+</section>
+
+
+
+<!-- dynamically load mathjax for compatibility with self-contained -->
+<script>
+  (function () {
+    var script = document.createElement("script");
+    script.type = "text/javascript";
+    script.src  = 
"https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML";;
+    document.getElementsByTagName("head")[0].appendChild(script);
+  })();
+</script>
+
+</body>
+</html>

-- 
To view, visit https://gerrit.wikimedia.org/r/391063
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I81f0f93a97f7467e1fcf30e20c252fc044bbbd31
Gerrit-PatchSet: 1
Gerrit-Project: wikimedia/discovery/wmf
Gerrit-Branch: master
Gerrit-Owner: Bearloga <mpo...@wikimedia.org>

_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] wikimedia...wmf[master]: db1047 => db1108

Reply via email to