Tag: distillation
-
When Transformers Sing: Adapting SpectralKD for Text-Based Knowledge Distillation
When Transformers Sing: Adapting SpectralKD for Text-Based Knowledge Distillation Exploring the frequency fingerprints of Transformers to guide smarter knowledge distillation The post When Transformers Sing: Adapting SpectralKD for Text-Based Knowledge Distillation appeared first on Towards Data Science. Ankit Singh Chauhan Go to original source
-
Generalized Kernel Inducing Points by Duality Gap for Dataset Distillation
Generalized Kernel Inducing Points by Duality Gap for Dataset Distillation arXiv:2502.12607v1 Announce Type: new Abstract: We propose Duality Gap KIP (DGKIP), an extension of the Kernel Inducing Points (KIP) method for dataset distillation. While existing dataset distillation methods often rely on bi-level optimization, DGKIP eliminates the need for such optimization by leveraging duality theory in…