DIVE: Embedding Compression via Self-Limiting Gradient Updates

arXiv:2605.20689v2 Announce Type: replace Abstract: High-dimensional language-model embeddings increase storage and search costs, while supervised compressors can overfit when relevance labels are scarce. We present DIVE (Dimensionality reduction with Implicit View Ensembles), a residual compression adapter codesigned with a self-limiting hinge loss, geometry distillation, and head-wise NT-Xent over implicit coordinate views. The hinge stops updating satisfied ranking constraints, while the dense objectives stabilize the compressed representation; only the first head is retained at inference. Under query-disjoint evaluation with two LLM2Vec backbones, five BEIR benchmarks, 128d and 256d outputs, and six baselines, DIVE is the strongest adapter on all five primary benchmarks. It also outperforms PCA and an autoencoder in comparisons against unsupervised compressors.