Qwen/Qwen2.5-7B-Instruct
← Booking Agent Demo · Qwen2ForCausalLM · meta device, no weights downloaded
| architecture | Qwen2ForCausalLM |
|---|
| model_type | qwen2 |
|---|
| attention | GQA 7x · heads 28 · kv heads 4 · head_dim 128 |
|---|
| position | RoPE · θ=1e+06 |
|---|
| norm | RMSNorm · eps 1e-06 |
|---|
| ffn | dense · inter 18944 · silu |
|---|
| embedding | vocab 152,064 · tie False |
|---|
| context | max_position_embeddings 32768 |
|---|
Dimensions
| hidden | 3,584 |
|---|
| layers | 28 |
|---|
| heads | 28 |
|---|
| kv_heads | 4 |
|---|
| head_dim | 128 |
|---|
| intermediate | 18,944 |
|---|
| vocab | 152,064 |
|---|
| ctx | 32,768 |
|---|
Parameters
≈7.62B
7,615,616,512 instantiated · Hub safetensors 7.62B (matches)
| model.layers | 6.53B | 85.7% |
|---|
| model.embed_tokens | 545.0M | 7.2% |
|---|
| lm_head | 545.0M | 7.2% |
|---|
| model.norm | 3.6K | 0.0% |
|---|
| model.rotary_emb | 0 | 0.0% |
|---|
Memory (bf16 unless noted)
| params_total | 7,615,616,512 |
|---|
| weights_bf16_gib | 14.19 |
|---|
| weights_int8_gib | 7.09 |
|---|
| weights_int4_gib | 3.55 |
|---|
| kv_cache_bytes_per_token | 57,344 |
|---|
| kv_cache_gib_at_ctx | 1.75 |
|---|
| ctx_used | 32,768 |
|---|
KV cache = 2 · layers · kv_heads · head_dim · 2 bytes per token
One decoder layer · [0..27] ×28
Module tree (click to expand)
model Qwen2ForCausalLM 7.62B 100.0%
model Qwen2Model 7.07B 92.8%
- embed_tokens Embedding 545.0M 7.2% 152064×3584
layers ModuleList 6.53B 85.7%
[0..27] ×28 Qwen2DecoderLayer 233.1M 3.1%
self_attn Qwen2Attention 29.4M 0.4%
- q_proj Linear 12.8M 0.2% 3584×3584, 3584
- k_proj Linear 1.8M 0.0% 512×3584, 512
- v_proj Linear 1.8M 0.0% 512×3584, 512
- o_proj Linear 12.8M 0.2% 3584×3584
mlp Qwen2MLP 203.7M 2.7%
- gate_proj Linear 67.9M 0.9% 18944×3584
- up_proj Linear 67.9M 0.9% 18944×3584
- down_proj Linear 67.9M 0.9% 3584×18944
- act_fn SiLUActivation 0 0.0%
- input_layernorm Qwen2RMSNorm 3.6K 0.0% 3584
- post_attention_layernorm Qwen2RMSNorm 3.6K 0.0% 3584
- norm Qwen2RMSNorm 3.6K 0.0% 3584
- rotary_emb Qwen2RotaryEmbedding 0 0.0%
- lm_head Linear 545.0M 7.2% 152064×3584
How this page was made: the model is instantiated from its Hugging Face config on torch's meta device (shapes only, no weights downloaded), parameters are counted per module, and the total is cross-checked against the safetensors metadata on the Hub. Memory figures are arithmetic, not measurements.