Please use the reply-all button to keep the conversation in the list. *"Please excuse me if I’m wrong, I’m not very familiar with the codebase.:"*
You should remove this line, it serves no purpose, I also don't understand the code snippet you added and why. -- Ibiam Chihurumnaya ib...@sugarlabs.org On Sun, Mar 31, 2024 at 6:10 PM Sujay R <sujay1...@gmail.com> wrote: > I've taken your advice and added more details about the implementation. > I'm sorry if sharing PDFs again and again is irritating. I'm sure you'll > let me know of a better method. > > I have asked you to review multiple versions of my draft proposal. I'm > very grateful for you take out your time and help me applying. Can you > please review it one last time? I'll submit once you give the green light. > > Thank you again for your guidance and support throughout this process > > On Fri, Mar 29, 2024 at 8:10 PM Chihurumnaya Ibiam <ib...@sugarlabs.org> > wrote: > >> Your timeline is terse, it'll be great if you add some implementation >> detail to your proposal as there's >> scarcely any. >> >> -- >> >> Ibiam Chihurumnaya >> ib...@sugarlabs.org >> >> >> >> On Thu, Mar 28, 2024 at 7:54 PM Sujay R <sujay1...@gmail.com> wrote: >> >>> Ok, the cloud sounds good. >>> >>> And thank you for patiently discussing the requirements with me and >>> answering my questions. Now, I think I can come up with a tentative >>> timeline. >>> I've attached v2 of my proposal with the timeline. Please let me know if >>> there are any more changes. If none, I'll submit it on the GSoC website. >>> The deadline is right around the corner >>> >>> On Wed, Mar 27, 2024 at 11:37 PM Chihurumnaya Ibiam <ib...@sugarlabs.org> >>> wrote: >>> >>>> We were thinking of a cloud option, but we haven't decided yet. >>>> >>>> -- >>>> >>>> Ibiam Chihurumnaya >>>> ib...@sugarlabs.org >>>> >>>> >>>> >>>> On Tue, Mar 26, 2024 at 6:40 PM Sujay R <sujay1...@gmail.com> wrote: >>>> >>>>> Thanks for explaining that. But it's still not clear to me where the >>>>> FOSS LLM should be run. Not on the devices running Sugar, so is cloud the >>>>> option you're looking for? >>>>> >>>>> On Tue, Mar 26, 2024 at 11:02 PM Chihurumnaya Ibiam < >>>>> ib...@sugarlabs.org> wrote: >>>>> >>>>>> The plan was never really to run an LLM on Sugar as that'll >>>>>> drastically increase the size of the activity >>>>>> and Sugar itself as Chat is a fructose >>>>>> <https://wiki.sugarlabs.org/go/Development_Team/Release/Modules> >>>>>> activity. >>>>>> >>>>>> -- >>>>>> >>>>>> Ibiam Chihurumnaya >>>>>> ib...@sugarlabs.org >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Mar 26, 2024 at 10:54 AM Sujay R <sujay1...@gmail.com> wrote: >>>>>> >>>>>>> Sugar runs on a lot of devices including low end devices - 2GB ram - >>>>>>>> and we intend to keep it that way, the chat activity is >>>>>>>> typically used by more than one Sugar instance, the chatbot should >>>>>>>> also be able to run on just one instance. >>>>>>> >>>>>>> >>>>>>> Running LLMs on just 2GB ram is atleast a few years away. So the bot >>>>>>> has to be hosted on an API. Cloud is a good option, there is serverless >>>>>>> GPU >>>>>>> inference and provisioned ones. One that I like is RunPod (serverless >>>>>>> pricing <https://www.runpod.io/serverless-gpu> and provisioned >>>>>>> pricing <https://www.runpod.io/gpu-instance/pricing>). Local >>>>>>> hosting is also an option, for a 7B model, a moderately new (4-5 year >>>>>>> old) >>>>>>> GPU with 16GB VRAM. Running with lower RAM is possible but with >>>>>>> excessive >>>>>>> quantisation (rounding off) at the cost of quality and speed. >>>>>>> >>>>>>> >>>>>>>> You can leverage the sugar-datastore if you need to store activity >>>>>>>> related data. >>>>>>>> >>>>>>> >>>>>>> Storing the chat history is not an issue. Time complexity for >>>>>>> generation(inference) of a transfomer is O(n^2) where n is the number of >>>>>>> tokens. So we need to be mindful of how much history do we actually need >>>>>>> >>>>>>>>
_______________________________________________ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel