Qwen/Qwen3.5-9B

Qwen3_5ForConditionalGeneration · meta device, no weights downloaded

architectureQwen3_5ForConditionalGeneration
model_typeqwen3_5
attentionGQA 4x · heads 16 · kv heads 4 · head_dim 256
positionRoPE · θ=1e+07 · partial 25%
normRMSNorm · eps 1e-06
ffndense · inter 12288 · silu
embeddingvocab 248,320 · tie False
contextmax_position_embeddings 262144

layer schedule · green = linear/sliding, blue = full attention

Dimensions

hidden4,096
layers32
heads16
kv_heads4
head_dim256
intermediate12,288
vocab248,320
ctx262,144

Parameters

≈8.95B

8,953,803,264 instantiated · Hub safetensors 9.65B (delta 699.3M: checkpoint holds modules this architecture does not instantiate, e.g. an MTP head)

model.layers6.92B77.3%
model.embed_tokens1.02B11.4%
lm_head1.02B11.4%
model.norm4.1K0.0%
model.rotary_emb00.0%

Memory (bf16 unless noted)

params_total8,953,803,264
weights_bf16_gib16.68
weights_int8_gib8.34
weights_int4_gib4.17
kv_cache_bytes_per_token131,072
kv_cache_gib_at_ctx4.00
ctx_used32,768

KV cache = 2 · layers · kv_heads · head_dim · 2 bytes per token

One decoder layer · [0..23] ×24

hidden_states [B, S, 4096]Qwen3_5RMSNorminput_layernorm · 4.1KQwen3_5GatedDeltaNetlinear_attn · 67.4Mout_proj 4096→4096 · in_proj_qkv 4096→8192 · in_proj_z 4096→4096 · in_proj_b 4096→32Qwen3_5RMSNormpost_attention_layernorm · 4.1KQwen3_5MLPmlp · 151.0Mgate_proj 4096→12288 · up_proj 4096→12288 · down_proj 12288→4096+ residual+ residual

Module tree (click to expand)

  • model Qwen3_5ForCausalLM 8.95B 100.0%
    • model Qwen3_5TextModel 7.94B 88.6%
      • embed_tokens Embedding 1.02B 11.4% 248320×4096
      • layers ModuleList 6.92B 77.3%
        • [0..23] ×24 Qwen3_5DecoderLayer 218.4M 2.4%
          • linear_attn Qwen3_5GatedDeltaNet 67.4M 0.8% 32, 32
            • conv1d Conv1d 32.8K 0.0% 8192×1×4
            • norm Qwen3_5RMSNormGated 128 0.0% 128
            • out_proj Linear 16.8M 0.2% 4096×4096
            • in_proj_qkv Linear 33.6M 0.4% 8192×4096
            • in_proj_z Linear 16.8M 0.2% 4096×4096
            • in_proj_b Linear 131.1K 0.0% 32×4096
            • in_proj_a Linear 131.1K 0.0% 32×4096
          • mlp Qwen3_5MLP 151.0M 1.7%
            • gate_proj Linear 50.3M 0.6% 12288×4096
            • up_proj Linear 50.3M 0.6% 12288×4096
            • down_proj Linear 50.3M 0.6% 4096×12288
            • act_fn SiLUActivation 0 0.0%
          • input_layernorm Qwen3_5RMSNorm 4.1K 0.0% 4096
          • post_attention_layernorm Qwen3_5RMSNorm 4.1K 0.0% 4096
        • [0..7] ×8 Qwen3_5DecoderLayer 209.7M 2.3%
          • self_attn Qwen3_5Attention 58.7M 0.7%
            • q_proj Linear 33.6M 0.4% 8192×4096
            • k_proj Linear 4.2M 0.0% 1024×4096
            • v_proj Linear 4.2M 0.0% 1024×4096
            • o_proj Linear 16.8M 0.2% 4096×4096
            • q_norm Qwen3_5RMSNorm 256 0.0% 256
            • k_norm Qwen3_5RMSNorm 256 0.0% 256
          • mlp Qwen3_5MLP 151.0M 1.7%
            • gate_proj Linear 50.3M 0.6% 12288×4096
            • up_proj Linear 50.3M 0.6% 12288×4096
            • down_proj Linear 50.3M 0.6% 4096×12288
            • act_fn SiLUActivation 0 0.0%
          • input_layernorm Qwen3_5RMSNorm 4.1K 0.0% 4096
          • post_attention_layernorm Qwen3_5RMSNorm 4.1K 0.0% 4096
      • norm Qwen3_5RMSNorm 4.1K 0.0% 4096
      • rotary_emb Qwen3_5TextRotaryEmbedding 0 0.0%
    • lm_head Linear 1.02B 11.4% 248320×4096