Hi, my name is Xinyu Lyu (吕新昱). I’m an Associate Professor in School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics. Before that, I obtained my Ph.D. degree from College of Computer Science, University of Electronic Science and Technology of China, advised by Prof.Lianli Gao, Prof.Jingkuan Song and Prof.Jie Shao.

My research interests mainly focus are Multimedia Learning and AI Safety, icluding Hallucination Mitigation, Safety Defense, and Jailbreak in Mutli-modal Large Lanuage Model.

🔥 News

• 2025.02:   One paper was accepted by IEEE Transactions on Image Processing (TIP 2025).
• 2025.01:   One paper was accepted by International Journal of Computer Vision (IJCV 2025).
• 2024.09:   One paper was accepted by Conference on Neural Information Processing Systems (NeurIPS 2024).
• 2023.11:   One paper was accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT 2023).
• 2023.07:   One paper was accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2023).
• 2023.07:   One paper was accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT 2023).
• 2023.02:   One paper was accepted by IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023).
• 2022.10:   One paper was accepted by ACM International Conference on Multimedia (ACM MM 2022).
• 2022.07:   Two papers were accepted by IEEE International Conference on Multimedia and Expo (ICME 2022).
• 2022.06:   One paper was accepted by IEEE/CVF Conference on Computer Vision and Pattern Recognitionn (CVPR 2022).
• 2021.10:   One paper was accepted by ACM International Conference on Multimedia (ACM MM 2021).
• 2021.06:   One paper was accepted by Pattern Recognition (PR 2021).
• 2019.10:   One paper was accepted by ACM International Conference on Multimedia (ACM MM 2019).

📝 Selected Publications

TPAMI 2023

Adaptive Fine-Grained Predicates Learning for Scene Graph Generation
Xinyu Lyu, Lianli Gao, Pengpeng Zeng, Heng Tao Shen, Jingkuan Song Code

Ensuring balanced and efficient learning process for fine-grained SGG.

IJCV 2025

Informative Scene Graph Generation via Debiasing
Lianli Gao, Xinyu Lyu(Corresponding Author), Yuyu Guo, Yuxuan Hu, Yuan-Fang Li, Lu Xu, Heng Tao Shen, Jingkuan Song Code

Making balanced and informative predicate prediction for unbiased SGG.

TIP 2025

Multi-Concept Learning for Scene Graph Generation
Xinyu Lyu, Lianli Gao, Junlin Xie, Pengpeng Zeng, Yulu Tian, Jie Shao, Heng Tao Shen Code

Addressing both predicate-level and concept-level imbalance issues for generalized unbiased SGG.

NeurIPS 2024

Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization.
Xinyu Lyu*, Beitao Chen*(Equal Contribution), Lianli Gao, Jingkuan Song and Heng Tao Shen Code

Alleviating Hallucinations in LVLMs via Weak-to-Strong Generalization.

CVPR 2022

Fine-Grained Predicates Learning for Scene Graph Generation
Xinyu Lyu, Lianli Gao, Yuyu Guo, Zhou Zhao, Hao Huang, Heng Tao Shen, Jingkuan Song Video | Code

Aims at differentiating hard-to-distinguish predicates for fine-grained SGG.

• Arxiv 2025 Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation, Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, Heng Tao Shen.
• Arxiv 2024 ALF: Adaptive Label Finetuning for Scene Graph Generation, Qishen Chen, Jianzhi Liu, Xinyu Lyu, Lianli Gao, Jingkuan Song.
• Arxiv 2024 Text-Video Retrieval with Global-Local Semantic Consistent Learning, Haonan Zhang, Pengpeng Zeng, Lianli Gao, Jingkuan Song, Yihang Duan, Xinyu Lyu, Haonan Zhang, Heng Tao Shen.
• Arxiv 2024 A Sentence-Semantic-Based Watermark for Large Language Models Against Paraphrasing Attacks, Han Wang, Lei Zhao, Xinyu Lyu, Lianli Gao, Jingkuan Song, Heng Tao Shen.
• CVPR 2023 Prototype-based Embedding Network for Scene Graph Generation, Chaofan Zheng*, Xinyu Lyu*(Equal Contribution), Lianli Gao, Bo Dai, Jingkuan Song.
• TCSVT 2023 SPT: Spatial Pyramid Transformer for Image Captioning, Haonan Zhang, Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Jingkuan Song, Heng Tao Shen.
• TCVST 2023 Dual-branch hybrid learning network for unbiased scene graph generation, Chaofan Zheng, Lianli Gao, Xinyu Lyu, Pengpeng Zeng, Abdulmotaleb El Saddik, Heng Tao Shen.
• ICME 2022 Learning to Generate Scene Graph from Head to Tail, Chaofan Zheng, Xinyu Lyu, Yuyu Guo, Jingkuan Song, Lianli Gao.
• ICME 2022 Multi-Scale Graph Attention Network for Scene Graph Generation, Min Chen, Xinyu Lyu, Yuyu Guo, Jingkuan Song, Lianli Gao.
• ACM MM 2022 Dynamic Scene Graph Generation via Temporal Prior Inference, Shuang Wang, Lianli Gao, Xinyu Lyu, Yuyu Guo, Jingkuan Song.
• ACM MM 2021 (oral) Conceptual and Syntactical Cross-modal Alignment with Cross-level Consistency for Image-Text Matching, Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Shuaiqi Jing, Jingkuan Song.
• PR 2021 GuessWhich? Visual dialog with attentive memory network, Lei Zhao, Xinyu Lyu, Jingkuan Song, Lianli Gao.
• ACM MM 2019 (oral) Mutual correlation attentive factors in dyadic fusion networks for speech emotion recognition, Yue Gu, Xinyu Lyu, Weijia Sun, Weitian Li, Shuhong Chen, Xinyu Li, Ivan Marsic.

🎖 Honors and Awards

• 2024.01 SCF Annual Outstanding Student Paper. (Webpage)
• 2024.05 Academic Newcomer Honor of UESTC. (Webpage)

📖 Educations

• 2020.09 - 2024.06, Ph.D. student, University of Electronic Science and Technology of China, Chengdu.
• 2017.09 - 2019.06, Master student, Rutgers University, New Jersey.
• 2014.09 - 2018.06, Undergraduate, University of Electronic Science and Technology of China, Chengdu.

💬 Invited Talks

• 2023.05, 2023 CVPR Southwest Region Pre-Sharing Session, Chengdu, Sichuan. (Video)
• 2022.05, 2022 CVPR Southwest Region Pre-Sharing Session, Chengdu, Sichuan. (Video)
• 2022.05, 2022 ChinaMM (Technical Forum: MM-CV High Impact Paper Appreciation Forum), Guiyang, Guizhou.
• 2019.10, 2019 ACM MM (Mutual correlation attentive factors in dyadic fusion networks for speech emotion recognition), Nice, France.

🙋 Service

• Journal Reviewer: IEEE TPAMI, IJCV, IEEE TIP, IEEE TCSVT, IEEE TMM, IEEE TGRS, etc.
• Conference Reviewer: CVPR, ECCV, NeurIPS, ACM MM, AAAI, EMNLP, ICPR, ICME, etc.
• Competition Reviewer: Guangdong-Hong Kong-Macao Greater Bay Area (Whampoa) International Algorithm Calculation Competition 2022 (Track: Panoptic Scene Graph Genetation)/ 2023(Track: FunQA) .

💻 Main Projects

• 2025.01 - 2026.12, Research on Visual Relationship Detection Algorithms and Applications in Open World Settings, Sichuan Provincial Natural Science Foundation Project (No. 2025ZNSFSC1463), Project Leader.
• 2023.02 - 2024.06, 2030 Science and Technology Innovation Major Project for Next-Generation Artificial Intelligence by the Ministry of Science and Technology, National Key Research and Development Program of China (No. 2018AAA0102200), Main Researcher.
• 2021.01 - 2025.12, Research on Key Technologies for Visual Natural Cognition through Collaborative Vision and Language Processing, National Natural Science Foundation of China under Grant (No. 62020106008), Participant.
• 2020.01 - 2022.07, Research on Scene Graph Generation for E-commerce Live Streaming/Video, Kuaishou (No. 202109FKY00302), Main Researcher.
• 2019.01 - 2022.12, Research on Key Technology Research on Collaborative Deep Video Understanding, Description, and Visual Question Answering, National Natural Science Foundation of China (No. 61872064), Participant.

• 2018.01 - 2021.12, Research on Key Technology for Deep Visual Understanding Integrated with Natural Language Processing, National Natural Science Foundation of China (No. 61772116), Participant.

  visitors since Jun 2024.