Russian Researchers Develop Neural Networks That Can Choose Which Layers to Fine-Tune
Researchers at the Moscow Institute of Physics and Technology (MIPT) have developed a method for fine-tuning large language models that automatically identifies the most important neural network layers and allocates more resources to fine-tuning them.
Developers are often reluctant to retrain these models from scratch because doing so requires enormous amounts of time and computing resources. Instead, they use small add-on modules called adapters, which were previously assigned the same size across all layers. But layers contribute differently to a model’s performance: some need more resources than others to learn a new task. The new approach adjusts adapter size based on what the model learns during the early stages of training.
The method was tested on four language models across 12 task categories. In every case, AI performance improved.
