Qwen/Qwen2.5-7B-Instruct

Qwen2ForCausalLM · meta device, no weights downloaded

architectureQwen2ForCausalLM
model_typeqwen2
attentionGQA 7x · heads 28 · kv heads 4 · head_dim 128
positionRoPE · θ=1e+06
normRMSNorm · eps 1e-06
ffndense · inter 18944 · silu
embeddingvocab 152,064 · tie False
contextmax_position_embeddings 32768

Dimensions

hidden3,584
layers28
heads28
kv_heads4
head_dim128
intermediate18,944
vocab152,064
ctx32,768

Parameters

≈7.62B

7,615,616,512 instantiated · Hub safetensors 7.62B (matches)

model.layers6.53B85.7%
model.embed_tokens545.0M7.2%
lm_head545.0M7.2%
model.norm3.6K0.0%
model.rotary_emb00.0%

Memory (bf16 unless noted)

params_total7,615,616,512
weights_bf16_gib14.19
weights_int8_gib7.09
weights_int4_gib3.55
kv_cache_bytes_per_token57,344
kv_cache_gib_at_ctx1.75
ctx_used32,768

KV cache = 2 · layers · kv_heads · head_dim · 2 bytes per token

One decoder layer · [0..27] ×28

hidden_states [B, S, 3584]Qwen2RMSNorminput_layernorm · 3.6KQwen2Attentionself_attn · 29.4Mq_proj 3584→3584 · k_proj 3584→512 · v_proj 3584→512 · o_proj 3584→3584Qwen2RMSNormpost_attention_layernorm · 3.6KQwen2MLPmlp · 203.7Mgate_proj 3584→18944 · up_proj 3584→18944 · down_proj 18944→3584+ residual+ residual

Module tree (click to expand)

  • model Qwen2ForCausalLM 7.62B 100.0%
    • model Qwen2Model 7.07B 92.8%
      • embed_tokens Embedding 545.0M 7.2% 152064×3584
      • layers ModuleList 6.53B 85.7%
        • [0..27] ×28 Qwen2DecoderLayer 233.1M 3.1%
          • self_attn Qwen2Attention 29.4M 0.4%
            • q_proj Linear 12.8M 0.2% 3584×3584, 3584
            • k_proj Linear 1.8M 0.0% 512×3584, 512
            • v_proj Linear 1.8M 0.0% 512×3584, 512
            • o_proj Linear 12.8M 0.2% 3584×3584
          • mlp Qwen2MLP 203.7M 2.7%
            • gate_proj Linear 67.9M 0.9% 18944×3584
            • up_proj Linear 67.9M 0.9% 18944×3584
            • down_proj Linear 67.9M 0.9% 3584×18944
            • act_fn SiLUActivation 0 0.0%
          • input_layernorm Qwen2RMSNorm 3.6K 0.0% 3584
          • post_attention_layernorm Qwen2RMSNorm 3.6K 0.0% 3584
      • norm Qwen2RMSNorm 3.6K 0.0% 3584
      • rotary_emb Qwen2RotaryEmbedding 0 0.0%
    • lm_head Linear 545.0M 7.2% 152064×3584