cypherpunks list not loading for me.
also many emails from it missing to inbox at the moment.
here is last spam that i tried to send during connectivity issues:
troubleshooting deepseek inference failure [on remote hardware]
transformers/modeling_utils.py 4788
`p` is an mlp weight, "model.layers.61.self_attn.q_a_proj.weight"
`param_device_map[p]` does not exist
`p` is enumerated from `weight_map`
transformers modeling_utils.py 4785:
- `weight_map` has mlp weights and `param_device_map` does not
- an mlp weight is "model.layers.61.self_attn.q_a_proj.weight"
- this is in PreTrainedModel._load_pretrained_model
0352
what conditions cause this block to execute?
where do weight_map and param_device_map come from?
`weight_map` is constructed in previous block.
`else` block in line 4783 indent depth 3
which weight map is constructed?
go up file. indent depth 3 is weight map condition
indent depth 2 is offload code condition
weightmap condition is `if sharded_metadata is None`
(Pdb) p sharded_metadata is None
False
so we have `weight_map = {p: os.path.join(folder, f) for p, f in
sharded_metadata["weight_map"].items()}`
-> weight map is constructed from `sharded_metadata`. if sharded_metadata were
None, it would be constructed from `original_loaded_keys` and would still
contain mlp weights.
it looks like a good avenue would either be to figure out why
`param_device_map` does not have mlp keys, or why the larger block is being
executed.
0358
line 4773 indent depth 2: `if device_map is not None and is_safetensors`
so basically this block is only run if there is both a device map, and
is_safetensors is set.
i think i'm manually setting is_safetensors; maybe i'll try disabling it and
see if i can generate the data then.
0359
0400 ok while that is loading lets see if we can figure out where
param_device_map comes from
0402: removing `use_safetensors` did not resolve the crash. param_device_map is
set on line 4774:
4773 if device_map is not None and is_safetensors:
4774 param_device_map = expand_device_map(device_map,
original_loaded_keys, start_prefix)
basically, `device_map` is expanded to `model.layers.[i]` but does not have an
entry for layer 61 which is the mlp layer.
so when it is expanded it doesn't have any of the weights in that layer.
this probably happens when the device map is autogenerated, which happened
outside this function.
0405
but rather in the calling function: .from_pretrained()
likely line 4259 device_map = infer_auto_device_map(...)
right now:
(Pdb) p device_map_kwargs
{'no_split_module_classes': ['DeepseekV3DecoderLayer'], 'special_dtypes':
{'lm_head.weight': torch.bfloat16}, 'max_memory': {'cpu': 85212960085}}
0407
so basically it sounds like these weights are not present in the model
enumeration
but are present on disk
i have run the model before as have many other so there's some way to make it
work.
it looks like the easiest way is to disable device_map which may mean fitting
the entire model on one device, or it may mean manually calling offload code
after construction.
i could maybe put it on cpu, then set the dtype and offloading after
or maybe i can set the offloading for the whole model without using a device
map somehow .... maybe not
- set a breakpint on infer_auto_device_map ? (i confirmed the layer is not in
the model)
- look at the model source code again to see if the layer can be enabled for
this step
- try calling without a device map
some confusion. it looks like the model has _62_ layers, whereas ....
uhhh ...
so num_hidden_layers is 61 and num_nextn_predict_layers is 1.
the ModuleList .layers is constructed with num_hidden_layers
and it has names that range from 0 to 60.
so the layer that is named "61" is the mlp layer. and it's #62.
confusing because there are 61 hidden layers
and it seemed like the kind of community that might use 1-based numbering
but nope! layer 61 is the 62nd layer, the mlp layer, and it's not in the list
of layers
so i don't see any way for layer 61 to be instantiated here :/ which is strange
cause i've thought i've seen it eval'd
maybe i can look at my logits and see what happened !
0417
0424
no, the log doesn't show layer 61 ever used. it does show expert 61 used a lot,
maybe i missaw that
ok hmm
so the huggingface device_map code assumes that what's on-disk matches what's
in the model ...
but i know elsewhere in the code they often handle that mismatching, so maybe
something just needs to be set for something to mismatch ...?
0425
0427
looks like the mismatched key code might be after this code; the present
assumption might be that sharded device mapped models are kind of tuned for use
hmm there's an unused function _load_pretrained_model_low_mem that looks
intended for people like me to try out
the keys come from the state_dict parameter. so i could either look into the
function for loading that, or preload a custom state dict, or not use a device
map
it looks like it might work to call transformers.modeling_utils.load_state_dict
in advance and filter the unused keys.
oh no that function is only used if it's not sharded
the keylist comes from get_checkpoint_shard_files
hrm >(
ok options:
- likely a way by passing a custom state dict
- likely a way by not using a device map
- likely a way by engaging internals, one option is get_checkpoint_shard_files
- likely a way by modifying the model to add the unused layers in
that last option might be _easiest and quickest_ here while it's kind of a
unique quirk just for generating test data
i'd just list all the keys in the weights that are on layer 61 and patch them
in i guess
when i run without a device map the warning says i'm supposed to use
"device_map = 'cuda'".
it seems happy to load on cpu
hmm device_map='cuda' seems to work. why is this?
ok i'll try on an H100 again. last time i checked i had $6 on vast.ai . an H100
is maybe $2.50/hr .
0516
ok device_map='cuda' works fine but then i run out of gpu memory ...
0526
so i stepped into device_map='cuda' and i'm around line 4586 and it did
actually enumerate missing_keys and unexpected_keys way back on line 4582 ...
there is also a list of unexpected keys to accept:
4620 # Some models may have keys that are not in the state by
design, removing them before needlessly warning
4621 # the user.
4622 -> if cls._keys_to_ignore_on_load_missing is not None:
4623 for pat in cls._keys_to_ignore_on_load_missing:
4624 missing_keys = [k for k in missing_keys if
re.search(pat, k) is None]
4625
4626 if cls._keys_to_ignore_on_load_unexpected is not None:
4627 for pat in cls._keys_to_ignore_on_load_unexpected:
4628 unexpected_keys = [k for k in unexpected_keys if
re.search(pat, k) is None]
4629 if hf_quantizer is not None:
4630 missing_keys = hf_quantizer.update_missing_keys(model,
missing_keys, prefix)
however, layer 61 is still in loaded_keys after, despite being detected as
unexpected
ok so on line 4773 is_safetensors is _false_ and the failing block isn't
executed. that's basically why it worked.
so why is is_safetensors false?
looks like on line 4534 that is_safetensors is only set if device_map contains
"disk".
it sounds like deepseek will run if i offload to cpu and not to disk.
maybe if i can get a VM running i can use swap. i haven't gotten VMs working on
vast.ai, it won't let me connect to them. hrm
maybe i'll just patch those lines to run the model! i can add a check for the
key to be present. lemme see how that works. line 4788 of modeling_utils.py 0535
0556 well now i get an error in get_disk_only_shard_files
i might want to just capture some weights manually at this point
- initially config.quantization_config = {'activation_scheme': 'dynamic',
'fmt': 'e4m3', 'quant_method': 'fp8', 'weight_block_size': [128, 128]}
- then config.quantization_config =
AutoHfQuantizer.merge_quantization_configs(config.quantization_config,
quantization_config=None) =
FineGrainedFP8Config(quant_method=<QuantizationMethod.FP8: 'fp8'>)
- then
3691 -> hf_quantizer = AutoHfQuantizer.from_config(
3692 config.quantization_config,
3693 pre_quantized=pre_quantized, // = True
3694 )
3699 hf_quantizer.validate_environment(
3700 torch_dtype=torch_dtype,
3701 from_tf=from_tf,
3702 -> from_flax=from_flax,
3703 device_map=device_map,
3704 weights_only=weights_only,
3705 )
3706 torch_dtype = hf_quantizer.update_torch_dtype(torch_dtype)
3707 device_map = hf_quantizer.update_device_map(device_map)
(... the model is constructed with empty weights ...)
4200 -> hf_quantizer.preprocess_model(
4201 model=model, device_map=device_map,
keep_in_fp32_modules=keep_in_fp32_modules
4202 )
it looks like preprocess_model is replacing Linear modules with FP8Linear
modules, before weights are loaded.
so that's likely a really important step my code was missing
... now it's doing the weight loading code i engaged so much ...
[hey one thing i could do is forward with saving weights, but only save them
for e.g. the first layer]
it looked like some of the param quantization initialization could have been in
_load_state_dict_into_meta_model or somesuch
so here's this, but it doesn't look properly initialized:
(Pdb) p model_to_load.model.layers[0].self_attn.q_a_proj.weight.cpu()
tensor([[ -22.0000, -72.0000, 88.0000, ..., -9.0000, -208.0000,
-28.0000],
[ 128.0000, 14.0000, 16.0000, ..., 104.0000, -64.0000,
26.0000],
[ 72.0000, -36.0000, 64.0000, ..., -120.0000, 80.0000,
-72.0000],
...,
[-144.0000, 80.0000, 48.0000, ..., -72.0000, -96.0000,
72.0000],
[ -80.0000, 120.0000, 72.0000, ..., -44.0000, 112.0000,
112.0000],
[ 224.0000, 4.5000, -56.0000, ..., 160.0000, -64.0000,
36.0000]], dtype=torch.float8_e4m3fn)
these are much higher magnitude numbers than i'd expect, i don't think they've
been scaled here
ok it's in weight_scale_inv:
(Pdb) p model_to_load.model.layers[0].self_attn.q_a_proj.weight_scale_inv.cpu()
tensor([[0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0003, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0001, 0.0002, 0.0002, 0.0002, 0.0001, 0.0001, 0.0001, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0004,
0.0002, 0.0002],
[0.0002, 0.0001, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0001,
0.0001, 0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001, 0.0001, 0.0002,
0.0001, 0.0002, 0.0003, 0.0001, 0.0003, 0.0003, 0.0002, 0.0002, 0.0001,
0.0002, 0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0003, 0.0002,
0.0002, 0.0001],
[0.0003, 0.0001, 0.0002, 0.0003, 0.0001, 0.0003, 0.0005, 0.0002, 0.0002,
0.0002, 0.0003, 0.0003, 0.0003, 0.0002, 0.0004, 0.0004, 0.0002, 0.0004,
0.0003, 0.0002, 0.0002, 0.0005, 0.0002, 0.0003, 0.0005, 0.0001, 0.0002,
0.0002, 0.0004, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0003, 0.0003, 0.0005, 0.0002, 0.0005, 0.0004, 0.0002, 0.0001, 0.0001,
0.0002, 0.0003, 0.0003, 0.0002, 0.0002, 0.0003, 0.0002, 0.0005, 0.0004,
0.0002, 0.0002],
[0.0004, 0.0002, 0.0002, 0.0003, 0.0001, 0.0003, 0.0005, 0.0002, 0.0003,
0.0002, 0.0003, 0.0003, 0.0004, 0.0002, 0.0004, 0.0004, 0.0003, 0.0004,
0.0003, 0.0001, 0.0002, 0.0005, 0.0002, 0.0003, 0.0005, 0.0002, 0.0003,
0.0002, 0.0004, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001,
0.0003, 0.0003, 0.0005, 0.0002, 0.0005, 0.0005, 0.0002, 0.0002, 0.0001,
0.0001, 0.0003, 0.0003, 0.0002, 0.0002, 0.0003, 0.0002, 0.0005, 0.0004,
0.0002, 0.0001],
[0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0003, 0.0002, 0.0003,
0.0002, 0.0001, 0.0002, 0.0003, 0.0002, 0.0002, 0.0003, 0.0002, 0.0002,
0.0001, 0.0003, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001,
0.0002, 0.0002, 0.0003, 0.0002, 0.0004, 0.0003, 0.0002, 0.0002, 0.0002,
0.0001, 0.0002, 0.0002, 0.0002, 0.0001, 0.0003, 0.0002, 0.0004, 0.0004,
0.0002, 0.0002],
[0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001,
0.0002, 0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0001, 0.0001, 0.0002,
0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0001, 0.0002, 0.0001, 0.0002,
0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0001, 0.0002, 0.0002, 0.0001,
0.0002, 0.0001, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0001, 0.0001, 0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002,
0.0002, 0.0001],
[0.0003, 0.0001, 0.0002, 0.0003, 0.0002, 0.0002, 0.0004, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0003, 0.0001, 0.0003, 0.0004, 0.0002, 0.0003,
0.0003, 0.0002, 0.0002, 0.0004, 0.0002, 0.0002, 0.0004, 0.0002, 0.0002,
0.0002, 0.0003, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001, 0.0002,
0.0002, 0.0003, 0.0004, 0.0002, 0.0004, 0.0004, 0.0001, 0.0002, 0.0002,
0.0001, 0.0003, 0.0002, 0.0002, 0.0002, 0.0003, 0.0002, 0.0004, 0.0003,
0.0002, 0.0002],
[0.0003, 0.0001, 0.0002, 0.0003, 0.0002, 0.0002, 0.0003, 0.0002,
0.0002,
[3/1806]
0.0002, 0.0003, 0.0002, 0.0002, 0.0001, 0.0004, 0.0003, 0.0002, 0.0003,
0.0003, 0.0002, 0.0002, 0.0004, 0.0002, 0.0002, 0.0003, 0.0002, 0.0002,
0.0002, 0.0003, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0003, 0.0004, 0.0002, 0.0004, 0.0004, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0003, 0.0002, 0.0001, 0.0003, 0.0002, 0.0005, 0.0004,
0.0002, 0.0002],
[0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0003, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0003,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0003, 0.0002,
0.0002, 0.0002],
[0.0003, 0.0002, 0.0002, 0.0003, 0.0002, 0.0003, 0.0005, 0.0002, 0.0002,
0.0002, 0.0003, 0.0003, 0.0003, 0.0002, 0.0004, 0.0004, 0.0002, 0.0004,
0.0003, 0.0002, 0.0002, 0.0004, 0.0002, 0.0003, 0.0005, 0.0002, 0.0002,
0.0002, 0.0004, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001,
0.0003, 0.0003, 0.0005, 0.0002, 0.0005, 0.0004, 0.0002, 0.0002, 0.0002,
0.0001, 0.0003, 0.0003, 0.0002, 0.0002, 0.0003, 0.0002, 0.0005, 0.0004,
0.0002, 0.0002],
[0.0001, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001,
0.0002, 0.0001, 0.0001, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0001, 0.0002,
0.0002, 0.0001, 0.0002, 0.0002, 0.0001, 0.0001, 0.0002, 0.0002, 0.0001,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002,
0.0002, 0.0001],
[0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0001, 0.0002, 0.0002, 0.0002, 0.0001, 0.0001, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0002, 0.0002,
0.0001, 0.0002, 0.0002, 0.0001, 0.0002, 0.0002, 0.0001, 0.0002, 0.0001,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0002,
0.0002, 0.0002, 0.0002, 0.0002, 0.0002, 0.0001, 0.0001, 0.0002, 0.0001,
0.0001, 0.0002]])
and of course i could have made mistakes copying that by hand from pdb
