Multi-PrefDrive: Advancing LLM-Based Autonomous Driving via Multi-Preference Learning

[2023 – 2024]

Research on developing safer and more human-like autonomous driving systems using large language models that learn human driving preferences.

As autonomous driving technology evolves, the next major challenge is to create systems that not only follow rules but also drive smoothly, safely, and comfortably, much like a skilled human. This research project addresses this challenge by developing the “PrefDrive” framework, which integrates nuanced human driving preferences—such as maintaining safe distances or ensuring smooth acceleration—into autonomous driving models using Large Language Models (LLMs). The goal is to create a system that can align with a wide range of requirements, from basic operational needs like traffic rule compliance to more human-like driving behaviors.

The core of PrefDrive is its pioneering use of Direct Preference Optimization (DPO), a preference learning technique, in the autonomous driving domain. This approach trains the model by having it learn from pairs of “chosen” (desirable) and “rejected” (undesirable) driving actions for a given scenario, allowing it to discern the optimal human choice. For this research, we built and publicly released a comprehensive dataset of 74,040 driving preference sequences. By implementing memory-efficient techniques like LoRA and 4-bit quantization, we have also made advanced LLM fine-tuning accessible on consumer-grade hardware, broadening research opportunities in the field.

However, real-world driving decisions are rarely a simple binary choice. For a single correct action, there are often multiple potential incorrect actions, each carrying a different degree of risk. To capture this complex decision-making landscape, we evolved the project to create “Multi-PrefDrive”. This advanced framework trains the model by pairing one “chosen” action with

multiple “rejected” alternatives, such as actions that are “aggressive,” “inattentive,” or “overcautious”. This enables the model to develop a more nuanced understanding of the spectrum of possible driving errors.

Multi-PrefDrive implements the sophisticated Plackett-Luce preference model to handle the ranked list of multiple choices. Experiments in the CARLA simulator demonstrated that this multi-preference approach dramatically improves performance over the standard DPO, especially in safety. The framework achieved an 83.6% reduction in infrastructure collisions and, in certain environments, eliminated traffic light violations entirely. This work validates that teaching AI to understand complex human judgment is a critical step toward creating safer and more reliable autonomous vehicles.

Publication

• Multi-PrefDrive: Optimizing Large Language Models for Autonomous Driving Through Multi-Preference Tuning
  Li, Yun, Javanmardi, Ehsan, Thompson, Simon, Katsumata, Kai, Orsholits, Alex, Tsukada, Manabu. 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hangzhou, China, 2025 [Proceedings Article]BibTeX
• PrefDrive: Enhancing Autonomous Driving through Preference-Guided Large Language Models
  Li, Yun, Javanmardi, Ehsan, Thompson, Simon, Katsumata, Kai, Orsholits, Alex, Tsukada, Manabu. 36th IEEE Intelligent Vehicles Symposium (IV2025), Cluj-Napoca, Romania, 2025 [Proceedings Article]BibTeX