On 14/7/24 12:56, Emanuel Berg wrote:
You can, but how much?
So this is the context? You mean include it in the prompt?
Then it is more easy to find in the llamafile(1) man page, it
is probably this
-c N, --ctx-size N
Set the size of the prompt context. A larger
context size helps the model to better comprehend
and generate responses for longer input or
conversations. The LLaMA models were built with
a context of 2048, which yields the best results
on longer input / inference.
If the unit is bytes that means 2048 bytes.
Okay, that I can try it right now just by inserting all the
data from a file into the query (prompt) and ask. And everyone
can try that, actually.
the 2048 is tokens which is approximately the number of words in a
prompt, so not character count.
The context explains how you want it to respond and the prompt is the
actual question.
You can massively increase the size of strings to match using localdocs.
I know how to do this in GPT4all. I assume there is a localdocs
equivalent equivalent in llamafile?
