Qwen/Qwen3.8-27B

Qwen3_5ForConditionalGeneration · meta device, no weights downloaded

architectureQwen3_5ForConditionalGeneration
model_typeqwen3_5
attentionGQA 6x · heads 24 · kv heads 4 · head_dim 256
positionRoPE · θ=1e+07 · partial 25%
normRMSNorm · eps 1e-06
ffndense · inter 17408 · silu
embeddingvocab 248,320 · tie False
contextmax_position_embeddings 262144

layer schedule · green = linear/sliding, blue = full attention

Dimensions

hidden5,120
layers64
heads24
kv_heads4
head_dim256
intermediate17,408
vocab248,320
ctx262,144

Parameters

≈26.90B

26,895,998,464 instantiated · Hub safetensors 27.78B (delta 885.4M: checkpoint holds modules this architecture does not instantiate, e.g. an MTP head)

model.layers24.35B90.5%
model.embed_tokens1.27B4.7%
lm_head1.27B4.7%
model.norm5.1K0.0%
model.rotary_emb00.0%

Memory (bf16 unless noted)

params_total26,895,998,464
weights_bf16_gib50.10
weights_int8_gib25.05
weights_int4_gib12.52
kv_cache_bytes_per_token262,144
kv_cache_gib_at_ctx8.00
ctx_used32,768

KV cache = 2 · layers · kv_heads · head_dim · 2 bytes per token

One decoder layer · [0..47] ×48

hidden_states [B, S, 5120]Qwen3_5RMSNorminput_layernorm · 5.1KQwen3_5GatedDeltaNetlinear_attn · 115.9Mout_proj 6144→5120 · in_proj_qkv 5120→10240 · in_proj_z 5120→6144 · in_proj_b 5120→48Qwen3_5RMSNormpost_attention_layernorm · 5.1KQwen3_5MLPmlp · 267.4Mgate_proj 5120→17408 · up_proj 5120→17408 · down_proj 17408→5120+ residual+ residual

Module tree (click to expand)

  • model Qwen3_5ForCausalLM 26.90B 100.0%
    • model Qwen3_5TextModel 25.62B 95.3%
      • embed_tokens Embedding 1.27B 4.7% 248320×5120
      • layers ModuleList 24.35B 90.5%
        • [0..47] ×48 Qwen3_5DecoderLayer 383.3M 1.4%
          • linear_attn Qwen3_5GatedDeltaNet 115.9M 0.4% 48, 48
            • conv1d Conv1d 41.0K 0.0% 10240×1×4
            • norm Qwen3_5RMSNormGated 128 0.0% 128
            • out_proj Linear 31.5M 0.1% 5120×6144
            • in_proj_qkv Linear 52.4M 0.2% 10240×5120
            • in_proj_z Linear 31.5M 0.1% 6144×5120
            • in_proj_b Linear 245.8K 0.0% 48×5120
            • in_proj_a Linear 245.8K 0.0% 48×5120
          • mlp Qwen3_5MLP 267.4M 1.0%
            • gate_proj Linear 89.1M 0.3% 17408×5120
            • up_proj Linear 89.1M 0.3% 17408×5120
            • down_proj Linear 89.1M 0.3% 5120×17408
            • act_fn SiLUActivation 0 0.0%
          • input_layernorm Qwen3_5RMSNorm 5.1K 0.0% 5120
          • post_attention_layernorm Qwen3_5RMSNorm 5.1K 0.0% 5120
        • [0..15] ×16 Qwen3_5DecoderLayer 372.3M 1.4%
          • self_attn Qwen3_5Attention 104.9M 0.4%
            • q_proj Linear 62.9M 0.2% 12288×5120
            • k_proj Linear 5.2M 0.0% 1024×5120
            • v_proj Linear 5.2M 0.0% 1024×5120
            • o_proj Linear 31.5M 0.1% 5120×6144
            • q_norm Qwen3_5RMSNorm 256 0.0% 256
            • k_norm Qwen3_5RMSNorm 256 0.0% 256
          • mlp Qwen3_5MLP 267.4M 1.0%
            • gate_proj Linear 89.1M 0.3% 17408×5120
            • up_proj Linear 89.1M 0.3% 17408×5120
            • down_proj Linear 89.1M 0.3% 5120×17408
            • act_fn SiLUActivation 0 0.0%
          • input_layernorm Qwen3_5RMSNorm 5.1K 0.0% 5120
          • post_attention_layernorm Qwen3_5RMSNorm 5.1K 0.0% 5120
      • norm Qwen3_5RMSNorm 5.1K 0.0% 5120
      • rotary_emb Qwen3_5TextRotaryEmbedding 0 0.0%
    • lm_head Linear 1.27B 4.7% 248320×5120