Hi, my name is Xinyu Lyu (吕新昱). I’m an Associate Professor in School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics. Before that, I obtained my Ph.D. degree from College of Computer Science, University of Electronic Science and Technology of China, advised by Prof.Lianli Gao, Prof.Jingkuan Song and Prof.Jie Shao.
My research interests mainly focus are Multimedia Learning and AI Safety, icluding Hallucination Mitigation, Jailbreak Defense, Reasoning Vulnerabilities and Spatial Intelligence of Mutli-modal Large Lanuage Model.
🔥 News
- 2025.06: I was recognized as an outstanding reviewer (Top 5%) for CVPR 2025.
- 2025.05: One paper was accepted by IEEE Transactions on Image Processing (TIP 2025).
- 2025.02: One paper was accepted by IEEE Transactions on Image Processing (TIP 2025).
- 2025.01: One paper was accepted by International Journal of Computer Vision (IJCV 2025).
- 2024.09: One paper was accepted by Conference on Neural Information Processing Systems (NeurIPS 2024).
📝 Selected Publications
Adaptive Fine-Grained Predicates Learning for Scene Graph Generation
Xinyu Lyu, Lianli Gao, Pengpeng Zeng, Heng Tao Shen, Jingkuan Song
Code
Ensuring balanced and efficient learning process for fine-grained SGG.
Informative Scene Graph Generation via Debiasing
Lianli Gao, Xinyu Lyu(Corresponding Author), Yuyu Guo, Yuxuan Hu, Yuan-Fang Li, Lu Xu, Heng Tao Shen, Jingkuan Song
Code
Making balanced and informative predicate prediction for unbiased SGG.
Multi-Concept Learning for Scene Graph Generation
Xinyu Lyu, Lianli Gao, Junlin Xie, Pengpeng Zeng, Yulu Tian, Jie Shao, Heng Tao Shen
Code
Addressing both predicate-level and concept-level imbalance issues for generalized unbiased SGG.
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization.
Xinyu Lyu*, Beitao Chen*(Equal Contribution), Lianli Gao, Jingkuan Song and Heng Tao Shen Code
Alleviating Hallucinations in LVLMs via Weak-to-Strong Generalization.
Fine-Grained Predicates Learning for Scene Graph Generation
Xinyu Lyu, Lianli Gao, Yuyu Guo, Zhou Zhao, Hao Huang, Heng Tao Shen, Jingkuan Song
Video
|
Code
Aims at differentiating hard-to-distinguish predicates for fine-grained SGG.
(†: equal contribution, *: corresponding author, ✧: Project Leader)
Arxiv 2025FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models, Shengming Yuan, Xinyu Lyu✧, Shuailong Wang, Jingkuan Song, Lianli Gao.Arxiv 2025SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism, Beitao Chen, Xinyu Lyu✧, Shengming Yuan, Jingkuan Song, Heng Tao Shen, Lianli Gao.Arxiv 2025Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation, Beitao Chen, Xinyu Lyu✧, Lianli Gao, Jingkuan Song, Heng Tao Shen.TIP 2025Text-Video Retrieval with Global-Local Semantic Consistent Learning, Haonan Zhang, Pengpeng Zeng, Lianli Gao, Jingkuan Song, Yihang Duan, Xinyu Lyu, Haonan Zhang, Heng Tao Shen.Arxiv 2024ALF: Adaptive Label Finetuning for Scene Graph Generation, Qishen Chen, Jianzhi Liu, Xinyu Lyu✧, Lianli Gao, Jingkuan Song.Arxiv 2024A Sentence-Semantic-Based Watermark for Large Language Models Against Paraphrasing Attacks, Han Wang, Lei Zhao, Xinyu Lyu, Lianli Gao, Jingkuan Song, Heng Tao Shen.ISAIR 2013Local-global Information Interaction Debiasing for Dynamic Scene Graph Generation, Xinyu Lyu*, Jingwei Liu, Yuyu Guo.CVPR 2023Prototype-based Embedding Network for Scene Graph Generation, Chaofan Zheng, Xinyu Lyu†, Lianli Gao, Bo Dai, Jingkuan Song.TCSVT 2023SPT: Spatial Pyramid Transformer for Image Captioning, Haonan Zhang, Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Jingkuan Song, Heng Tao Shen.TCVST 2023Dual-branch hybrid learning network for unbiased scene graph generation, Chaofan Zheng, Lianli Gao, Xinyu Lyu, Pengpeng Zeng, Abdulmotaleb El Saddik, Heng Tao Shen.ICME 2022Learning to Generate Scene Graph from Head to Tail, Chaofan Zheng, Xinyu Lyu, Yuyu Guo, Jingkuan Song, Lianli Gao.ICME 2022Multi-Scale Graph Attention Network for Scene Graph Generation, Min Chen, Xinyu Lyu, Yuyu Guo, Jingkuan Song, Lianli Gao.ACM MM 2022Dynamic Scene Graph Generation via Temporal Prior Inference, Shuang Wang, Lianli Gao, Xinyu Lyu, Yuyu Guo, Jingkuan Song.ACM MM 2021(oral) Conceptual and Syntactical Cross-modal Alignment with Cross-level Consistency for Image-Text Matching, Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Shuaiqi Jing, Jingkuan Song.PR 2021GuessWhich? Visual dialog with attentive memory network, Lei Zhao, Xinyu Lyu✧, Jingkuan Song, Lianli Gao.ACM MM 2019(oral) Mutual correlation attentive factors in dyadic fusion networks for speech emotion recognition, Yue Gu, Xinyu Lyu, Weijia Sun, Weitian Li, Shuhong Chen, Xinyu Li, Ivan Marsic.
🎖 Honors and Awards
- 2025.06 Outstanding Reviewers for CVPR 2025.(Webpage)
- 2024.05 Academic Newcomer Honor of UESTC. (Webpage)
- 2024.01 SCF Annual Outstanding Student Paper. (Webpage)
📚 Course Teaching
- Graduate: Multimodal Large Models (Semester 1, 2025–2026)
- Graduate: Principles and Practice of Multimodal Large Models (Semester 1, 2025–2026)
- Graduate: Frontiers in Multimodal Large Models (Semester 2, 2025–2026)
- Undergraduate: Artificial Intelligence and Modern Technology (Semesters 1–2, 2024–2026)
🙋 Service
- Journal Reviewer: IEEE TPAMI, IJCV, IEEE TIP, IEEE TCSVT, IEEE TMM, IEEE TGRS, ACM TOMM, etc.
- Conference Reviewer: CVPR, ICCV, ECCV, NeurIPS, ICLR, ACM MM, AAAI, EMNLP, ICME, etc.
- Competition Reviewer: Guangdong-Hong Kong-Macao Greater Bay Area (Whampoa) International Algorithm Calculation Competition 2022/2023.
💻 Main Projects
- 2025.07 - 2027.07, Teaching Practice Research on an Inclusive Finance Risk-Control Dialogue System Driven by Multimodal Large-Scale Models, Sichuan Provincial Department of Education Innovative Experimental Project of Undergraduate, Project Leader.
- 2025.01 - 2026.12, Research on Visual Relationship Detection Algorithms and Applications in Open World Settings, Sichuan Provincial Natural Science Foundation Project (No. 2025ZNSFSC1463), Project Leader.
- 2023.02 - 2024.06, 2030 Science and Technology Innovation Major Project for Next-Generation Artificial Intelligence by the Ministry of Science and Technology, National Key Research and Development Program of China (No. 2018AAA0102200), Main Researcher.
- 2021.01 - 2025.12, Research on Key Technologies for Visual Natural Cognition through Collaborative Vision and Language Processing, National Natural Science Foundation of China under Grant (No. 62020106008), Participant.
- 2020.01 - 2022.07, Research on Scene Graph Generation for E-commerce Live Streaming/Video, Kuaishou (No. 202109FKY00302), Main Researcher.
- 2019.01 - 2022.12, Research on Key Technology Research on Collaborative Deep Video Understanding, Description, and Visual Question Answering, National Natural Science Foundation of China (No. 61872064), Participant.
-
2018.01 - 2021.12, Research on Key Technology for Deep Visual Understanding Integrated with Natural Language Processing, National Natural Science Foundation of China (No. 61772116), Participant.
visitors since Jun 2024.