openai/privacy-filter

OpenAIPrivacyFilterForTokenClassification · meta device, no weights downloaded

architectureOpenAIPrivacyFilterForTokenClassification
model_typeopenai_privacy_filter
attentionGQA 7x · heads 14 · kv heads 2 · head_dim 64
positionRoPE · θ=150000 · scaling yarn
normRMSNorm · eps 1e-05
ffnMoE 128 experts · top-4 · inter 640 · silu
embeddingvocab 200,064 · tie False
contextmax_position_embeddings 131072 · sliding window 128

Dimensions

hidden640
layers8
heads14
kv_heads2
head_dim64
intermediate640
vocab200,064
ctx131,072

Parameters

≈1.40B

1,399,486,865 instantiated · Hub safetensors 1.40B (matches)

model.layers1.27B90.8%
model.embed_tokens128.0M9.1%
score21.2K0.0%
model.norm6400.0%
model.rotary_emb00.0%
dropout00.0%

Memory (bf16 unless noted)

params_total1,399,486,865
weights_bf16_gib2.61
weights_int8_gib1.30
weights_int4_gib0.65
kv_cache_bytes_per_token4,096
kv_cache_gib_at_ctx0.12
ctx_used32,768

KV cache = 2 · layers · kv_heads · head_dim · 2 bytes per token

One decoder layer · [0..7] ×8

hidden_states [B, S, 640]OpenAIPrivacyFilterRMSNorminput_layernorm · 640OpenAIPrivacyFilterAttentionself_attn · 1.3Mq_proj 640→896 · k_proj 640→128 · v_proj 640→128 · o_proj 896→640OpenAIPrivacyFilterRMSNormpost_attention_layernorm · 640OpenAIPrivacyFilterMLPmlp · 157.6Mrouter 640→128+ residual+ residual

Module tree (click to expand)

  • model OpenAIPrivacyFilterForTokenClassification 1.40B 100.0%
    • model OpenAIPrivacyFilterModel 1.40B 100.0%
      • embed_tokens Embedding 128.0M 9.1% 200064×640
      • layers ModuleList 1.27B 90.8%
        • [0..7] ×8 OpenAIPrivacyFilterEncoderLayer 158.9M 11.4%
          • self_attn OpenAIPrivacyFilterAttention 1.3M 0.1% 14
            • q_proj Linear 574.3K 0.0% 896×640, 896
            • k_proj Linear 82.0K 0.0% 128×640, 128
            • v_proj Linear 82.0K 0.0% 128×640, 128
            • o_proj Linear 574.1K 0.0% 640×896, 640
          • mlp OpenAIPrivacyFilterMLP 157.6M 11.3%
            • router OpenAIPrivacyFilterTopKRouter 82.0K 0.0% 128×640, 128
            • experts OpenAIPrivacyFilterExperts 157.5M 11.3% 128×640×1280, 128×1280
          • input_layernorm OpenAIPrivacyFilterRMSNorm 640 0.0% 640
          • post_attention_layernorm OpenAIPrivacyFilterRMSNorm 640 0.0% 640
      • norm OpenAIPrivacyFilterRMSNorm 640 0.0% 640
      • rotary_emb OpenAIPrivacyFilterRotaryEmbedding 0 0.0%
    • dropout Dropout 0 0.0%
    • score Linear 21.2K 0.0% 33×640, 33