Xiangyu Qi

I am a Member of Technical Staff at OpenAI, where I work to make LLMs safe and robust. Previously, I obtained my Ph.D. from Princeton University, advised by Prof. Prateek Mittal and Prof. Peter Henderson.

Selected Research

Safety Alignment Should Be Made More Than Just a Few Tokens Deep

ICLR, 2025. Outstanding Paper Award (3 Out Of 11,672 Submissions)

Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, Peter Henderson
Paper Code

Coverage: Yannic Kilcher’s Paper Explainer Center for Democracy and Technology letter to NIST

AI Risk Management Should Incorporate Both Safety and Security

Preprint.

Xiangyu Qi, Yangsibo Huang, Yi Zeng, Edoardo Debenedetti, Jonas Geiping, Luxi He, Kaixuan Huang, Udari Madhushani, Vikash Sehwag, Weijia Shi, Boyi Wei, Tinghao Xie, Danqi Chen, Pin-Yu Chen, Jeffrey Ding, Ruoxi Jia, Jiaqi Ma, Arvind Narayanan, Weijie J Su, Mengdi Wang, Chaowei Xiao, Bo Li, Dawn Song, Peter Henderson, Prateek Mittal
Paper

Fine-tuning Aligned Language Models Compromises Safety,
Even When Users Do Not Intend To!

ICLR, 2024. Oral Presentation, 1.2%
Covered by The New York Times

Xiangyu Qi $^*$ , Yi Zeng $^*$ , Tinghao Xie $^*$ , Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, Peter Henderson
Paper Policy Brief Code Website

Coverage: The New York Times PCMag The Register VentureBeat

Visual Adversarial Examples Jailbreak Aligned Large Language Models

AAAI, 2024. Oral Presentation, 4.6%
GPT-4V(ision) system card cited this paper to underscore the emerging threat vector of multimodal jailbreaking.

Xiangyu Qi $^*$ , Kaixuan Huang $^*$ , Ashwinee Panda, Peter Henderson, Mengdi Wang, Prateek Mittal
Paper Code

Towards A Proactive ML Approach for Detecting Backdoor Poison Samples

USENIX Security, 2023.

Xiangyu Qi, Tinghao Xie, Jiachen T. Wang, Tong Wu, Saeed Mahloujifar, Prateek Mittal
Paper & An Oral Presentation Code

Towards Practical Deployment-Stage Backdoor Attack on Deep Neural Networks

CVPR, 2022. Oral Presentation, 4.2%

Xiangyu Qi $^*$ , Tinghao Xie $^*$ , Ruizhe Pan, Jifeng Zhu, Yong Yang, Kai Bu
Paper Code

Xiangyu Qi (漆翔宇)

Selected Research