Introduction
Recently, large language models (LLMs), such as OpenAI’s ChatGPT, have caught enormous attention of the public.
They can generate remarkably realistic, coherent text based on a user's input and have the potential to be general-purpose tools used throughout society, e.g. for customer service, summarizing texts, answering questions, code generation and solving math problems. These LLMs are typically built on Transformer architectures with tens of billions of parameters and trained on trillions of tokens.
However, the large model size and high computational complexity of LLMs results in significant computational power and storage requirements far surpassing what is currently available with standard consumer hardware. For instance, even the relatively modest-sized model LLaMA-65B needs 130GB of GPU RAM for inference and more than 780 GB for fine-tuning with new data. Therefore, the aim of this PhD project is to develop sustainable architectures and fast training approaches for LLMs such that they can be accessible and deployed, given limited computational resources. This will promote sustainable language model usage in more scenarios in terms of power and hardware consumption.
In this project, the student will explore various pathways to achieve sustainable LLM architecture and training, including:
Developing alternative architectures to Transformers to reduce the size of language models, through model compression or fundamentally new networks;
Data selection during fine-tuning, where selection mechanisms of data will be developed instead of using entire training dataset.
Students will work in a collaborative team with both experts in machine learning and natural language processing involved. We strongly suggest that students with strong motivation dive into the research of sustainable large language models to make GPT-type models accessible even with limited computational resources.