Introduction
This project aims to pioneer advancements in the energy-efficient Generative AI models (GenAI), focusing on achieving faster inference times and reduced model sizes without compromising performance and increasing carbon emissions.
As GenAI becomes increasingly central to a wide range of applications, from generating images to generating videos and music, their computational demand and the time required for training and inference have escalated. This research seeks to address these challenges by developing innovative techniques for efficiency, including architectural innovations, compression strategies, algorithmic improvements, and system level optimizations.
The goal is to enable the deployment of state-of-the-art GenAI models across broader scenarios of computing environments, from high-end servers to consumer-level machines. This project will contribute to making GenAI more democratic, efficient, and scalable, paving the way for their application in real-time and resource-constrained scenarios.
Objectives
To develop cutting-edge techniques for model compression, such as pruning, quantization, and knowledge distillation, tailored for GenAI models.
To design and experiment with new GenAI architectures that are more efficient, requiring less computational power and memory.
To create new algorithms and system wide optimizations to accelerate both training and inference processes for GenAI, making them more suitable for deployment across a variety of computing environments.
To develop and utilize benchmarks and metrics specifically designed to evaluate the energy-aware efficiency and performance of GenAI under various computational constraints.
Methodology
Via a thorough literature review and benchmarking of existing models, we aim to identify key limitations and research gaps ripe for advancement. Possible solutions may relate to tailored pruning techniques, which selectively remove less critical parameters, alongside quantization methods that lower the precision of numerical values to significantly decrease model sizes. These optimization methods are still under-explored in the area of GenAI. Moreover, knowledge distillation will be utilized to investigate the potential for subgraph optimization. We also aim to investigate potential of distributed computing along with memory reuse and caching. These strategic directions are anticipated to reduce energy consumption, inference times and computational demands, without sacrificing the performance of the models.
You might be also asked to explore new model architectures designed to balance computational efficiency with robust performance, focusing on the potential of subgraph decomposition to break down complex models into simpler, more manageable components. This strategy aims to facilitate more efficient processing and parallelization. Enhancing the underlying algorithms of GenAI models, such as optimizing attention mechanisms and streamlining data processing, will be another crucial aspect for improving overall model responsiveness and efficiency. Additionally, the project will investigate adaptive deployment strategies, enabling models to dynamically adjust their computational complexity in real-time, ensuring optimal performance across diverse hardware environments. We will conduct comprehensive experimental validation and testing, this approach seeks to push the boundaries of what's possible with GenAI, setting new standards for efficiency and accessibility in AI technologies.
The Project Team
This project will require expertise from the domain of ML, Generative AI and Systems. The main supervisor Dr Zhiwu will provide his expertise on ML and GenAI. He has published numerous works in top conferences such as CVPR, NeurIPS and AAAI. The external supervisor Dr Jagmohan from UCL will provide his expertise on efficient ML systems, edge computing and on device learning crucial to implement developed techniques on devices and understand the tradeoffs between accuracy and efficiency. He has a track record in publishing papers in top conferences such as ICML, NeurIPS, SenSys and IPSN.
Together, Dr Zhiwu and Dr Jagmohan will be able to provide in depth expertise to lead the student to successful completion of their PhD. More details about the supervisors are present on their Google Scholar pages. Links here: Dr Jagmohan and Dr Zhiwu
PhD desirable skills
Strong ML background
Good Programming Skills
Background in Systems will be an additional advantage but is not necessary
References:
References
[1] R. Verdecchia, J. Sallou, L. Cruz, A systematic review of Green AI, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 13 (2023).
[2] Li, Yanyu, et al. "Snapfusion: Text-to-image diffusion model on mobile devices within two seconds." Advances in Neural Information Processing Systems 36 (2024).
[3] Li, Yanjing, et al. "Q-dm: An efficient low-bit quantized diffusion model." Advances in Neural Information Processing Systems 36 (2024)
[4] Wang, Yunke, et al. "Learning to schedule in diffusion probabilistic models." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2023.