An update for folks on this - I've extracted a few statistics below from the main dashboard. We said it would be worth testing this out for 1 release to see what we thought. I haven't had time to do a deep dive, but overall, I do think it is providing value for users, so we should keep it there. Happy to discuss though.
852 questions over the past month, averaging around 191 per week. People aren't using the feedback mechanisms much, but there were 19 thumbs up and 3 thumbs down. In 90% of cases, the bot was able to give a confident response. Languages: * English - 88% * Chinese - 8% * Russian - 3% * Japanese - 1% Topics: * Unsupported features - 14% * Troubleshooting - 9% * Comparison with similar technologies - 2% * Off topic questions - 2% If there are any other PMC members or committers who'd like access to the dashboard, please do let me know. On Mon, 12 May 2025 at 15:02, Matt Topol <zotthewiz...@gmail.com> wrote: > Thanks! > > On Mon, May 12, 2025, 9:47 AM Nic Crane <thisis...@gmail.com> wrote: > > > Absolutely Matt - I'm adding them now and I'll add you as an admin so > > you'll be able to review Go questions in the UI and make any changes > you'd > > like to what we include. > > > > On Wed, 7 May 2025 at 12:59, Matt Topol <zotthewiz...@gmail.com> wrote: > > > > > Would we be able to feed it the arrow-go documentation too? > > > > > > On Wed, May 7, 2025, 12:52 PM Nic Crane <thisis...@gmail.com> wrote: > > > > > > > A brief update on this from me - I've now added the Python, R, C++, > and > > > > Java cookbooks as a source for the kapa.ai question bot, so code > > > examples > > > > from those should be on there now. > > > > > > > > At a later point, I'll take a look at potentially adding GitHub PRs > and > > > > issues, but I'd like to test what we have so far before adding more > in > > > > there. > > > > > > > > > > > > On Mon, 28 Apr 2025 at 04:56, Nic Crane <thisis...@gmail.com> wrote: > > > > > > > > > We've current had 64 questions, though many of these are from folks > > > > > testing it out. I've personally found it useful for some of my dev > > > work. > > > > > > > > > > It's hard to tell yet if it's useful for users but we'll have more > > data > > > > > once it's live on the latest docs rather than the dev docs, so my > > plan > > > is > > > > > to keeping checking in on the questions and answers between the > > > upcoming > > > > > release and the next one so we can decide whether it's a useful > tool > > > for > > > > us > > > > > to have. > > > > > > > > > > If anyone would like a login for kapa.ai so you can see the > > > > conversations > > > > > and answers (and add in any feedback you'd like to add yourself for > > > > > questions), then please email me and I'll add you as a user. > > > > > > > > > > > > > > > On Wed, 23 Apr 2025 at 06:11, Nic Crane <thisis...@gmail.com> > wrote: > > > > > > > > > >> > nit: any chance the popup can be made to respect the light/dark > > mode > > > > >> toggle > > > > >> in the docs? It seems to always use a light color scheme. > > > > >> > > > > >> From the looks of it, the colour can be configured via the > > > > >> 'data-project-color' parameter[1] so I imagine this could be done > > via > > > > >> updating the widget script with some custom JS on our side. > > > > >> > > > > >> [1] > https://docs.kapa.ai/integrations/website-widget/configuration > > > > >> > > > > >> On Mon, 31 Mar 2025 at 17:15, Brian Hulette <bhule...@apache.org> > > > > wrote: > > > > >> > > > > >>> Very cool, I tried a few queries and it provided good answers. > > > > >>> > > > > >>> nit: any chance the popup can be made to respect the light/dark > > mode > > > > >>> toggle > > > > >>> in the docs? It seems to always use a light color scheme. > > > > >>> > > > > >>> On Fri, Mar 28, 2025 at 5:05 PM Nic Crane <thisis...@gmail.com> > > > wrote: > > > > >>> > > > > >>> > Cookbooks aren't part of the sources but we can certainly add > > them > > > > >>> later. > > > > >>> > > > > > >>> > The kapa folk recommend trying with an initial sunset of > sources > > > and > > > > >>> adding > > > > >>> > more in gradually, so I'll check back in on this next week and > > > think > > > > >>> about > > > > >>> > what we might want to add next. > > > > >>> > > > > > >>> > On Fri, 28 Mar 2025, 21:45 Aldrin, <octalene....@pm.me.invalid > > > > > > wrote: > > > > >>> > > > > > >>> > > Ooh, yeah it's looking fairly effective. I asked some > questions > > > > and I > > > > >>> > like > > > > >>> > > that the answers address differences in language > > implementations > > > > >>> (e.g. > > > > >>> > > python bindings vs cpp) and that there are relatively good > code > > > > >>> > suggestions. > > > > >>> > > > > > > >>> > > I assume this means the tests and benchmarks are helping the > > RAG > > > a > > > > >>> lot, > > > > >>> > so > > > > >>> > > general well done to all contributors. Out of curiosity, are > > the > > > > >>> > cookbooks > > > > >>> > > part of the learning/retrieval sources? > > > > >>> > > > > > > >>> > > Sent from Proton Mail <https://proton.me/mail/home> for iOS > > > > >>> > > > > > > >>> > > > > > > >>> > > On Fri, Mar 28, 2025 at 13:58, Neal Richardson < > > > > >>> > > neal.p.richard...@gmail.com > > > > >>> > > > <On+Fri,+Mar+28,+2025+at+13:58,+Neal+Richardson+%3C%3Ca+href=>> > > > > >>> wrote: > > > > >>> > > > > > > >>> > > Awesome, thanks for doing this! I tried out a few queries and > > was > > > > >>> > > impressed > > > > >>> > > with what I got back. > > > > >>> > > > > > > >>> > > Neal > > > > >>> > > > > > > >>> > > On Fri, Mar 28, 2025 at 2:01 PM Nic Crane < > thisis...@gmail.com > > > > > > > >>> wrote: > > > > >>> > > > > > > >>> > > > Hi folks, > > > > >>> > > > > > > > >>> > > > The kapa.ai bot is now live on the dev docs! We're using > it > > to > > > > >>> test > > > > >>> > > > whether > > > > >>> > > > we can enhance user experience of our documentation via > > > retrieval > > > > >>> > > augmented > > > > >>> > > > generation which generates answers to user questions by > > > combining > > > > >>> > > > similarity searches of the docs and LLM output. > > > > >>> > > > > > > > >>> > > > You can use it by clicking the "Ask AI" button in the > bottom > > > > right > > > > >>> > > corner > > > > >>> > > > of https://arrow.apache.org/docs/dev/index.html > > > > >>> > > > > > > > >>> > > > Currently we've just ingested the docs, but as we > experiment > > > > more, > > > > >>> we > > > > >>> > > can > > > > >>> > > > add other sources like Zulip and the mailing lists if we > > think > > > > >>> these > > > > >>> > > will > > > > >>> > > > provide useful additional source material. > > > > >>> > > > > > > > >>> > > > Please give it a go and use the "good answer" and "bad > > answer" > > > > >>> buttons > > > > >>> > > so > > > > >>> > > > we can assess how useful this is likely to be for our end > > > users; > > > > >>> bonus > > > > >>> > > > points for adding a better answer when you're not happy > with > > > the > > > > >>> one > > > > >>> > > given. > > > > >>> > > > > > > > >>> > > > I'll be checking in on it every so often and will report > back > > > > here > > > > >>> with > > > > >>> > > the > > > > >>> > > > results. > > > > >>> > > > > > > > >>> > > > Thanks, > > > > >>> > > > > > > > >>> > > > Nic > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > >> > > > > > > > > > >
