Re: [ot][spam][crazy][random] non-canon traffick/mc boss spinoffs #93

[Undescribed Horrific Abuse, One Victim & Survivor of Many]

> ok it doesn't download all the files, looks like the total size is
> about 12g, 3 files.
> 1607
>
> now it fails to run because it tries to allocate 8GB on my 2GB gpu.
> maybe i can make it do cpu-only
> 1608
> but i worry i'm out of posts for today
>
> i seem to be okay so far for today, no bounce messages yet :s might
> have only a few left
> 1608
> 1609
> hrm and https://health.petals.dev shows no available providers or
> working models for me :s
> 1610
> 1624
> ok i debugged it and it's not trying to put it on my gpu, it's trying
> to put in my normal ram, and it fails to allocate 8GB of normal ram.
> it looks like the 8GB weight is the 'lm_head' which i think is the
> linear layer at the bottom of the model that converts the word
> property logits to logits that predict the next token.
comment on linear layer, which is used a lot
"linear layer" seems to be a machine-learning-neural-networks term for "matrix".
it's just a plain matrix that takes N inputs and produces M outputs
(or vice versa) and has N*M floats in it.
so it's incredibly simple and changing it is very easy because all it
is doing is taking a dot product with a constant vector to produce
each output.
> 1625
> considering this a little, i realize this shows another bug in the use
> of safetensors: it shouldn't need to reallocate data that is mmap'd.
> so i guess i'd better check why this is happening.
> 1626
> the code is trying to convert the bfloat16 weights into float32. this
> could lose a lot of information imo bfloat16 is way better at
> representing log ranges. but that's why it allocates all the ram. i
> wonder why it's float32ing it.
> 1627
> srcline is 
> /home/user/.local/lib/python3.12/site-packages/transformers/modeling_utils.py(899)_load_state_dict_into_meta_model()
> transformers==4.43.1
> here's the backtrace:
>   
> /home/user/.local/lib/python3.12/site-packages/petals/utils/auto_config.py(79)from_pretrained()
> -> return super().from_pretrained(model_name_or_path, *args,
> revision=revision, **kwargs)
>   
> /home/user/.local/lib/python3.12/site-packages/petals/utils/auto_config.py(52)from_pretrained()
> -> return proper_cls.from_pretrained(model_name_or_path, *args, **kwargs)
>   
> /home/user/.local/lib/python3.12/site-packages/petals/client/from_pretrained.py(31)from_pretrained()
> -> return super().from_pretrained(model_name_or_path, *args,
> low_cpu_mem_usage=low_cpu_mem_usage, **kwargs)
>   
> /home/user/.local/lib/python3.12/site-packages/transformers/modeling_utils.py(3903)from_pretrained()
> -> ) = cls._load_pretrained_model(
>   
> /home/user/.local/lib/python3.12/site-packages/transformers/modeling_utils.py(4377)_load_pretrained_model()
> -> new_error_msgs, offload_index, state_dict_index =
> _load_state_dict_into_meta_model(
> > /home/user/.local/lib/python3.12/site-packages/transformers/modeling_utils.py(899)_load_state_dict_into_meta_model()->None
> -> param = param.to(old_param.dtype)