Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
Haoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu, Yizhou Sun, Yian Ma, Yeyun Gong
ICLR 2026 | October 2025
Xiao Liu
Senior Researcher
Publications
-
-
-
LayerNorm Induces Recency Bias in Transformer Decoders
Junu Kim, Xiao Liu, Zheng-Wen Lin, Lei Ji, Yeyun Gong, Edward Choi
September 2025
-
DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
Haebin Shin, Lei Ji, Xiao Liu, Zhiwei Yu, Qi Chen, Yeyun Gong
August 2025
-
Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training
Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Qi Chen, Peng Cheng
ACL | August 2025
-
Enhancing Large Language Model Performance with Gradient-Based Parameter Selection
Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Qi Chen, Peng Cheng
AAAI Conference on Artificial Intelligence, 2025 | April 2025
Oral Presentation
-
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
Haebin Shin, Lei Ji, Xiao Liu, Yeyun Gong
ICML 2025 | March 2025
-
Optimizing Large Language Model Training Using FP4 Quantization
Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, Peng Cheng
ICML 2025 | January 2025
-
EpiCoder: Encompassing Diversity and Complexity in Code Generation
Yaoxiang Wang, Haoling Li, Xin Zhang, Jie Wu, Xiao Liu, Wenxiang Hu, Zhongxin Guo, Yangyu Huang, Ying Xin, Yujiu Yang, Jinsong Su, Qi Chen, Scarlett Li
ICML 2025 | January 2025
-
-
-
Not All Tokens Are What You Need for Pretraining
Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Jian Jiao, Nan Duan, Weizhu Chen
2024 Neural Information Processing Systems | October 2024
Best Paper Runner Up
-
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, Weizhu Chen
March 2024
-
Using Left and Right Brains Together: Towards Vision and Language Planning
Jun Cen, Chenfei Wu, Xiao Liu, Sheng-Siang Yin, Yixuan Pei, Jinglong Yang, Qifeng Chen, Nan Duan, Jianguo Zhang
ICML 2024 | February 2024
-
-
-
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, zhongyu wei, Jian Guo, Nan Duan, Weizhu Chen
NeurIPS 2023 | December 2023
-
-
-
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Majumder, Ji-Rong Wen, Nan Duan, Weizhu Chen
EMNLP 2022 | October 2022
-
-
-
Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
Haoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu, Yizhou Sun, Yian Ma, Yeyun Gong
ICLR 2026 | October 2025
-
LayerNorm Induces Recency Bias in Transformer Decoders
Junu Kim, Xiao Liu, Zheng-Wen Lin, Lei Ji, Yeyun Gong, Edward Choi
September 2025
-
DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
Haebin Shin, Lei Ji, Xiao Liu, Zhiwei Yu, Qi Chen, Yeyun Gong
August 2025
-
Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training
Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Qi Chen, Peng Cheng
ACL | August 2025
-
Enhancing Large Language Model Performance with Gradient-Based Parameter Selection
Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Qi Chen, Peng Cheng
AAAI Conference on Artificial Intelligence, 2025 | April 2025
Oral Presentation
-
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
Haebin Shin, Lei Ji, Xiao Liu, Yeyun Gong
ICML 2025 | March 2025
-
Optimizing Large Language Model Training Using FP4 Quantization
Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, Peng Cheng
ICML 2025 | January 2025
-
EpiCoder: Encompassing Diversity and Complexity in Code Generation
Yaoxiang Wang, Haoling Li, Xin Zhang, Jie Wu, Xiao Liu, Wenxiang Hu, Zhongxin Guo, Yangyu Huang, Ying Xin, Yujiu Yang, Jinsong Su, Qi Chen, Scarlett Li
ICML 2025 | January 2025
-
Not All Tokens Are What You Need for Pretraining
Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Jian Jiao, Nan Duan, Weizhu Chen
2024 Neural Information Processing Systems | October 2024
Best Paper Runner Up
-
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, Weizhu Chen
March 2024
-
Using Left and Right Brains Together: Towards Vision and Language Planning
Jun Cen, Chenfei Wu, Xiao Liu, Sheng-Siang Yin, Yixuan Pei, Jinglong Yang, Qifeng Chen, Nan Duan, Jianguo Zhang
ICML 2024 | February 2024
-
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, zhongyu wei, Jian Guo, Nan Duan, Weizhu Chen
NeurIPS 2023 | December 2023
-
-
-
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Majumder, Ji-Rong Wen, Nan Duan, Weizhu Chen
EMNLP 2022 | October 2022
-
-
-
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Majumder, Ji-Rong Wen, Nan Duan, Weizhu Chen
EMNLP 2022 | October 2022
-
-
-
LayerNorm Induces Recency Bias in Transformer Decoders
Junu Kim, Xiao Liu, Zheng-Wen Lin, Lei Ji, Yeyun Gong, Edward Choi
September 2025
-
DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections
Haebin Shin, Lei Ji, Xiao Liu, Zhiwei Yu, Qi Chen, Yeyun Gong
August 2025
-
Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning
Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, Weizhu Chen
March 2024
-
-
-
Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
Haoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu, Yizhou Sun, Yian Ma, Yeyun Gong
ICLR 2026 | October 2025
-
Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training
Zheheng Luo, Xin Zhang, Xiao Liu, Haoling Li, Yeyun Gong, Qi Chen, Peng Cheng
ACL | August 2025
-
Enhancing Large Language Model Performance with Gradient-Based Parameter Selection
Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Qi Chen, Peng Cheng
AAAI Conference on Artificial Intelligence, 2025 | April 2025
Oral Presentation
-
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
Haebin Shin, Lei Ji, Xiao Liu, Yeyun Gong
ICML 2025 | March 2025
-
Optimizing Large Language Model Training Using FP4 Quantization
Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, Peng Cheng
ICML 2025 | January 2025
-
EpiCoder: Encompassing Diversity and Complexity in Code Generation
Yaoxiang Wang, Haoling Li, Xin Zhang, Jie Wu, Xiao Liu, Wenxiang Hu, Zhongxin Guo, Yangyu Huang, Ying Xin, Yujiu Yang, Jinsong Su, Qi Chen, Scarlett Li
ICML 2025 | January 2025
-
Not All Tokens Are What You Need for Pretraining
Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Jian Jiao, Nan Duan, Weizhu Chen
2024 Neural Information Processing Systems | October 2024
Best Paper Runner Up
-
Using Left and Right Brains Together: Towards Vision and Language Planning
Jun Cen, Chenfei Wu, Xiao Liu, Sheng-Siang Yin, Yixuan Pei, Jinglong Yang, Qifeng Chen, Nan Duan, Jianguo Zhang
ICML 2024 | February 2024
-
AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
Tong Wu, Zhihao Fan, Xiao Liu, Yeyun Gong, Yelong Shen, Jian Jiao, Hai-Tao Zheng, Juntao Li, zhongyu wei, Jian Guo, Nan Duan, Weizhu Chen
NeurIPS 2023 | December 2023
-
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Majumder, Ji-Rong Wen, Nan Duan, Weizhu Chen
EMNLP 2022 | October 2022
-