Introduction
Large Language Models (LLMs) have shown to deliver ground breaking performance but at the expense of huge power costs.
Both during LLM pre-training (e.g. GPT-5 is rumoured to be trained on 25,000 GPU's for many days) and LLM fine-tuning (e.g. the modest LLaMa3 90B model fine tuning needs a minimum of 2 x A100 GPU cards and several days; these days 500B+ open source LLMs being released which are very GPU hungry). The upward curve of LLM GPU training resource requirements is not sustainable environmentally.
This inter-disciplinary PhD will explore combining statistical approaches from the discipline of Optimal Experimental Design with LLM deep active learning algorithms from the discipline of Computer Science. Deep active learning aims to use LLM patterns within data encodings and model output distributions to identify the best unlabelled training data instance which, if labelled and trained with, would train the LLM much more efficiently than a randomly chosen training data instance. OED uses statistical information stored in unlabelled data to choose the subsample to train the model on. OED is well studied for parametric models, while its use in LLMs has not been explored before. This is due to the need for the derivatives of the model with respect to model parameters.
Hypothesis
Our hypothesis is that even though LLMs are nonparametric, the necessary gradient information can be approximated using automated differentiation. This will identify the best subsamples to be used for non-linear deep active learning algorithms for LLM training cycles and ultimately train LLMs, given a target accuracy, with less training data & epochs compared to traditional methods. If successful this project has the potential to discover new ways to substantially reduce the GPU footprint of companies fine-tuning LLMs for downstream applications, and if adopted by a major LLM vendor reduce the massive LLM pre-training footprint.