Hi, my name is Xinyu Lyu (吕新昱). I’m an Associate Professor in School of Computing and Artificial Intelligence, Southwestern University of Finance and Economics. Before that, I obtained my Ph.D. degree from College of Computer Science, University of Electronic Science and Technology of China, advised by Prof.Lianli Gao, Prof.Jingkuan Song and Prof.Jie Shao .

My research interests mainly focus are Multimedia Learning, including Scene Graph Generation, Visual Dialog and Image Captioning. I’m now especially interested in Hallucination in MLLM.

🔥 News

• 2024.09:   One paper was accepted by Conference on Neural Information Processing Systems(NeurIPS 2024).
• 2023.11:   One paper was accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT 2023).
• 2023.07:   One paper was accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2023).
• 2023.07:   One paper was accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT 2023).
• 2023.03:   One paper was accepted by International Symposium on Artificial Intelligence and Robotics (ISAIR 2023).
• 2023.02:   One paper was accepted by IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023).
• 2022.10:   One paper was accepted by ACM International Conference on Multimedia (ACM MM 2022).
• 2022.07:   Two papers were accepted by IEEE International Conference on Multimedia and Expo (ICME 2022).
• 2022.06:   One paper was accepted by IEEE/CVF Conference on Computer Vision and Pattern Recognitionn (CVPR 2022).
• 2021.10:   One paper was accepted by ACM International Conference on Multimedia (ACM MM 2021).
• 2021.06:   One paper was accepted by Pattern Recognition (PR 2021).
• 2019.10:   One paper was accepted by ACM International Conference on Multimedia (ACM MM 2019).

📝 Publications

TPAMI 2023

Adaptive Fine-Grained Predicates Learning for Scene Graph Generation
Xinyu Lyu, Lianli Gao, Pengpeng Zeng, Heng Tao Shen, Jingkuan Song

Code

Ensuring balanced and efficient learning process for fine-grained SGG.

CVPR 2023

Prototype-based Embedding Network for Scene Graph Generation
Chaofan Zheng*, Xinyu Lyu*(Equal Contribution), Lianli Gao, Bo Dai, Jingkuan Song Video | Code

Acquiring robust features for reliable relation prediction in SGG.

CVPR 2022

Fine-Grained Predicates Learning for Scene Graph Generation
Xinyu Lyu, Lianli Gao, Yuyu Guo, Zhou Zhao, Hao Huang, Heng Tao Shen, Jingkuan Song

Video | Code

Aims at differentiating hard-to-distinguish predicates for SGG.

IJCV 2023

Informative Scene Graph Generation via Debiasing (Minor Revision)
Lianli Gao, Xinyu Lyu, Yuyu Guo, Yuxuan Hu, Yuan-Fang Li, Lu Xu, Heng Tao Shen, Jingkuan Song

Code

Making balanced and informative predicate prediction for SGG.

• NeurIPS 2024 Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationg, Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, Heng Tao Shen.
• Arxiv 2024 ALF: Adaptive Label Finetuning for Scene Graph Generation, Qishen Chen, Jianzhi Liu, Xinyu Lyu, Lianli Gao, Jingkuan Song.
• Arxiv 2024 Text-Video Retrieval with Global-Local Semantic Consistent Learning, Haonan Zhang, Pengpeng Zeng, Lianli Gao, Jingkuan Song, Yihang Duan, Xinyu Lyu, Haonan Zhang, Heng Tao Shen.
• Arxiv 2024 A Sentence-Semantic-Based Watermark for Large Language Models Against Paraphrasing Attacks, Han Wang, Lei Zhao, Xinyu Lyu, Lianli Gao, Jingkuan Song, Heng Tao Shen.
• Arxiv 2023 Generalized Unbiased Scene Graph Generation, Xinyu Lyu, Lianli Gao, Junlin Xie, Pengpeng Zeng, Yulu Tian, Jie Shao, Heng Tao Shen.
• TCSVT 2023 SPT: Spatial Pyramid Transformer for Image Captioning, Haonan Zhang, Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Jingkuan Song, Heng Tao Shen.
• TCVST 2023 Dual-branch hybrid learning network for unbiased scene graph generation, Chaofan Zheng, Lianli Gao, Xinyu Lyu, Pengpeng Zeng, Abdulmotaleb El Saddik, Heng Tao Shen.
• ISAIR 2023 Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation, Xinyu Lyu, Jingwei Liu, Yuyu Guo, Lianli Gao.
• ICME 2022 Learning to Generate Scene Graph from Head to Tail, Chaofan Zheng, Xinyu Lyu, Yuyu Guo, Jingkuan Song, Lianli Gao.
• ICME 2022 Multi-Scale Graph Attention Network for Scene Graph Generation, Min Chen, Xinyu Lyu, Yuyu Guo, Jingkuan Song, Lianli Gao.
• ACM MM 2022 Dynamic Scene Graph Generation via Temporal Prior Inference, Shuang Wang, Lianli Gao, Xinyu Lyu, Yuyu Guo, Jingkuan Song.
• ACM MM 2021 (oral) Conceptual and Syntactical Cross-modal Alignment with Cross-level Consistency for Image-Text Matching, Pengpeng Zeng, Lianli Gao, Xinyu Lyu, Shuaiqi Jing, Jingkuan Song.
• PR 2021 GuessWhich? Visual dialog with attentive memory network, Lei Zhao, Xinyu Lyu, Jingkuan Song, Lianli Gao.
• ACM MM 2019 (oral) Mutual correlation attentive factors in dyadic fusion networks for speech emotion recognition, Yue Gu, Xinyu Lyu, Weijia Sun, Weitian Li, Shuhong Chen, Xinyu Li, Ivan Marsic.

🎖 Honors and Awards

• 2024.01 SCF Annual Outstanding Student Paper. (Webpage)
• 2024.05 Academic Newcomer Honor of UESTC. (Webpage)

📖 Educations

• 2020.09 - now, Ph.D. student, University of Electronic Science and Technology of China, Chengdu.
• 2017.09 - 2019.06, Master student, Rutgers University, New Jersey.
• 2014.09 - 2018.06, Undergraduate, University of Electronic Science and Technology of China, Chengdu.

💬 Invited Talks

• 2023.05, 2023 CVPR Southwest Region Pre-Sharing Session, Chengdu, Sichuan. (Video)
• 2022.05, 2022 CVPR Southwest Region Pre-Sharing Session, Chengdu, Sichuan. (Video)
• 2022.05, 2022 ChinaMM (Technical Forum: MM-CV High Impact Paper Appreciation Forum), Guiyang, Guizhou.
• 2019.10, 2019 ACM MM (Mutual correlation attentive factors in dyadic fusion networks for speech emotion recognition), Nice, France.

🙋 Service

• Journal Reviewer: IEEE TPAMI, IJCV, IEEE TIP, IEEE TCSVT, IEEE TMM, IEEE TGRS, etc.
• Conference Reviewer: CVPR, ECCV, NeurIPS, ACM MM, AAAI, EMNLP, ICPR, ICME, etc.
• Competition Reviewer: Guangdong-Hong Kong-Macao Greater Bay Area (Whampoa) International Algorithm Calculation Competition 2022 (Track: Panoptic Scene Graph Genetation)/ 2023(Track: FunQA) .

💻 Main Projects

• 2023.02 - 2024.06, 2030 Science and Technology Innovation Major Project for Next-Generation Artificial Intelligence by the Ministry of Science and Technology, National Key Research and Development Program of China (No. 2018AAA0102200), Main Researcher.
• 2021.01 - 2025.12, Research on Key Technologies for Visual Natural Cognition through Collaborative Vision and Language Processing, National Natural Science Foundation of China under Grant (No. 62020106008), Participant.
• 2020.01 - 2022.07, Research on Scene Graph Generation for E-commerce Live Streaming/Video, Kuaishou (No. 202109FKY00302), Main Researcher.
• 2019.01 - 2022.12, Research on Key Technology Research on Collaborative Deep Video Understanding, Description, and Visual Question Answering, National Natural Science Foundation of China (No. 61872064), Participant.

• 2018.01 - 2021.12, Research on Key Technology for Deep Visual Understanding Integrated with Natural Language Processing, National Natural Science Foundation of China (No. 61772116), Participant.

  visitors since Jun 2024.