Feature Kernel Distillation
We study the significance of Feature Learning in Neural Networks (NNs) for Knowledge Distillation (KD), a popular technique to improve an NN model’s generalisation performance using a teacher NN model. We propose a principled framework Feature Kernel Distillation (FKD), which performs distillation directly in the feature space of NNs and is therefore able to transfer knowledge across different datasets.
Zero-Cost Neural Architecture Search
Neural architecture search (NAS) is quickly becoming the standard methodology for designing deep neural networks (DNNs). NAS replaces human-designed DNNs by automatically searching for a DNN topology and size that maximize performance.