ibm-granite/granite-embedding-278m-multilingual

XLMRobertaModel · meta device, no weights downloaded

architecture	XLMRobertaModel
model_type	xlm-roberta
attention	MHA · heads 12 · kv heads 12 · head_dim 64
position	absolute
norm	LayerNorm · eps 1e-05
ffn	dense · inter 3072 · gelu
embedding	vocab 250,002 · tie True
context	max_position_embeddings 514

Dimensions

hidden	768
layers	12
heads	12
kv_heads	12
head_dim	64
intermediate	3,072
vocab	250,002
ctx	514

Parameters

≈278.0M

278,043,648 instantiated · Hub safetensors 278.0M (matches)

embeddings.word_embeddings	192.0M	69.1%
encoder	85.1M	30.6%
pooler	590.6K	0.2%
embeddings.position_embeddings	394.8K	0.1%
embeddings.LayerNorm	1.5K	0.0%
embeddings.token_type_embeddings	768	0.0%
embeddings.dropout	0	0.0%

Memory (bf16 unless noted)

params_total	278,043,648
weights_bf16_gib	0.52
weights_int8_gib	0.26
weights_int4_gib	0.13
kv_cache_bytes_per_token	36,864
kv_cache_gib_at_ctx	0.02
ctx_used	514

KV cache = 2 · layers · kv_heads · head_dim · 2 bytes per token

One decoder layer · [0..11] ×12

Module tree (click to expand)

model XLMRobertaModel 278.0M 100.0%
- embeddings XLMRobertaEmbeddings 192.4M 69.2%
  - word_embeddings Embedding 192.0M 69.1% 250002×768
  - token_type_embeddings Embedding 768 0.0% 1×768
  - LayerNorm LayerNorm 1.5K 0.0% 768, 768
  - dropout Dropout 0 0.0%
  - position_embeddings Embedding 394.8K 0.1% 514×768
- encoder XLMRobertaEncoder 85.1M 30.6%
  - layer ModuleList 85.1M 30.6%
    [0..11] ×12 XLMRobertaLayer 7.1M 2.5%
    attention XLMRobertaAttention 2.4M 0.9%
    self XLMRobertaSelfAttention 1.8M 0.6%
    query Linear 590.6K 0.2% 768×768, 768
    key Linear 590.6K 0.2% 768×768, 768
    value Linear 590.6K 0.2% 768×768, 768
    dropout Dropout 0 0.0%
    output XLMRobertaSelfOutput 592.1K 0.2%
    dense Linear 590.6K 0.2% 768×768, 768
    LayerNorm LayerNorm 1.5K 0.0% 768, 768
    dropout Dropout 0 0.0%
    intermediate XLMRobertaIntermediate 2.4M 0.8%
    dense Linear 2.4M 0.8% 3072×768, 3072
    intermediate_act_fn GELUActivation 0 0.0%
    output XLMRobertaOutput 2.4M 0.8%
    dense Linear 2.4M 0.8% 768×3072, 768
    LayerNorm LayerNorm 1.5K 0.0% 768, 768
    dropout Dropout 0 0.0%
- pooler XLMRobertaPooler 590.6K 0.2%
  - dense Linear 590.6K 0.2% 768×768, 768
  - activation Tanh 0 0.0%