Yea sorry about that code snippet. I don't know why I added that, even I don't understand it. I just wanted to let you know that I can figure out the part of integrating the bot into the activity. That's why I added that random snippet I've removed it.
I'm attaching my final proposal, that I'm submitting, as a reference. Thank you for your assistance throughout the proposal review process On Tue, Apr 2, 2024 at 12:04 AM Chihurumnaya Ibiam <ib...@sugarlabs.org> wrote: > Please use the reply-all button to keep the conversation in the list. > > *"Please excuse me if I’m wrong, I’m not very familiar with the > codebase.:"* > > You should remove this line, it serves no purpose, I also don't understand > the code snippet > you added and why. > > -- > > Ibiam Chihurumnaya > ib...@sugarlabs.org > > > > On Sun, Mar 31, 2024 at 6:10 PM Sujay R <sujay1...@gmail.com> wrote: > >> I've taken your advice and added more details about the implementation. >> I'm sorry if sharing PDFs again and again is irritating. I'm sure you'll >> let me know of a better method. >> >> I have asked you to review multiple versions of my draft proposal. I'm >> very grateful for you take out your time and help me applying. Can you >> please review it one last time? I'll submit once you give the green light. >> >> Thank you again for your guidance and support throughout this process >> >> On Fri, Mar 29, 2024 at 8:10 PM Chihurumnaya Ibiam <ib...@sugarlabs.org> >> wrote: >> >>> Your timeline is terse, it'll be great if you add some implementation >>> detail to your proposal as there's >>> scarcely any. >>> >>> -- >>> >>> Ibiam Chihurumnaya >>> ib...@sugarlabs.org >>> >>> >>> >>> On Thu, Mar 28, 2024 at 7:54 PM Sujay R <sujay1...@gmail.com> wrote: >>> >>>> Ok, the cloud sounds good. >>>> >>>> And thank you for patiently discussing the requirements with me and >>>> answering my questions. Now, I think I can come up with a tentative >>>> timeline. >>>> I've attached v2 of my proposal with the timeline. Please let me know >>>> if there are any more changes. If none, I'll submit it on the GSoC website. >>>> The deadline is right around the corner >>>> >>>> On Wed, Mar 27, 2024 at 11:37 PM Chihurumnaya Ibiam < >>>> ib...@sugarlabs.org> wrote: >>>> >>>>> We were thinking of a cloud option, but we haven't decided yet. >>>>> >>>>> -- >>>>> >>>>> Ibiam Chihurumnaya >>>>> ib...@sugarlabs.org >>>>> >>>>> >>>>> >>>>> On Tue, Mar 26, 2024 at 6:40 PM Sujay R <sujay1...@gmail.com> wrote: >>>>> >>>>>> Thanks for explaining that. But it's still not clear to me where the >>>>>> FOSS LLM should be run. Not on the devices running Sugar, so is cloud the >>>>>> option you're looking for? >>>>>> >>>>>> On Tue, Mar 26, 2024 at 11:02 PM Chihurumnaya Ibiam < >>>>>> ib...@sugarlabs.org> wrote: >>>>>> >>>>>>> The plan was never really to run an LLM on Sugar as that'll >>>>>>> drastically increase the size of the activity >>>>>>> and Sugar itself as Chat is a fructose >>>>>>> <https://wiki.sugarlabs.org/go/Development_Team/Release/Modules> >>>>>>> activity. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Ibiam Chihurumnaya >>>>>>> ib...@sugarlabs.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Mar 26, 2024 at 10:54 AM Sujay R <sujay1...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Sugar runs on a lot of devices including low end devices - 2GB ram >>>>>>>>> - and we intend to keep it that way, the chat activity is >>>>>>>>> typically used by more than one Sugar instance, the chatbot should >>>>>>>>> also be able to run on just one instance. >>>>>>>> >>>>>>>> >>>>>>>> Running LLMs on just 2GB ram is atleast a few years away. So the >>>>>>>> bot has to be hosted on an API. Cloud is a good option, there is >>>>>>>> serverless >>>>>>>> GPU inference and provisioned ones. One that I like is RunPod >>>>>>>> (serverless >>>>>>>> pricing <https://www.runpod.io/serverless-gpu> and provisioned >>>>>>>> pricing <https://www.runpod.io/gpu-instance/pricing>). Local >>>>>>>> hosting is also an option, for a 7B model, a moderately new (4-5 year >>>>>>>> old) >>>>>>>> GPU with 16GB VRAM. Running with lower RAM is possible but with >>>>>>>> excessive >>>>>>>> quantisation (rounding off) at the cost of quality and speed. >>>>>>>> >>>>>>>> >>>>>>>>> You can leverage the sugar-datastore if you need to store activity >>>>>>>>> related data. >>>>>>>>> >>>>>>>> >>>>>>>> Storing the chat history is not an issue. Time complexity for >>>>>>>> generation(inference) of a transfomer is O(n^2) where n is the number >>>>>>>> of >>>>>>>> tokens. So we need to be mindful of how much history do we actually >>>>>>>> need >>>>>>>> >>>>>>>>>
Sujay_R-Sugar_labs_GSoC_Chat_Activity_v4.pdf
Description: Adobe PDF document
_______________________________________________ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel