Publications

• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective
  Yuchen Wen, Keping Bi, Wei Chen, Jiafeng Guo, Xueqi Cheng.
  arXiv, 2024
  [arXiv] [Abstract] [Bib] [Code]
  

  As Large Language Models (LLMs) become an important way of information seeking, there have been increasing concerns about the unethical content LLMs may generate. In this paper, we conduct a rigorous evaluation of LLMs' implicit bias towards certain groups by attacking them with carefully crafted instructions to elicit biased responses. Our attack methodology is inspired by psychometric principles in cognitive and social psychology. We propose three attack approaches, i.e., Disguise, Deception, and Teaching, based on which we built evaluation datasets for four common bias types. Each prompt attack has bilingual versions. Extensive evaluation of representative LLMs shows that 1) all three attack methods work effectively, especially the Deception attacks; 2) GLM-3 performs the best in defending our attacks, compared to GPT-3.5 and GPT-4; 3) LLMs could output content of other bias types when being taught with one type of bias. Our methodology provides a rigorous and effective way of evaluating LLMs' implicit bias and will benefit the assessments of LLMs' potential ethical risks. Code and datasets are available at https://github.com/wen112358/ImplicitBiasPsychometricEvaluation.

  @article{wen2024evaluating,
      title={Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective},
      author={Wen, Yuchen and Bi, Keping and Chen, Wei and Guo, Jiafeng and Cheng, Xueqi},
      journal={arXiv preprint arXiv:2406.14023},
      year={2024}
  }
                

Education & Research Experience

Blogs

Selected Honors & Awards

• National Scholarship(top 2%) - 2022.12

• Tianjin Government Scholarship(top 0.5%) - 2021.12

• Outstanding Undergraduate Thesis of NKU(top 3%) - 2023.06

• Outstanding Graduate(top 3%) - 2023.06

Academic Service

Social Service

• President of the students' union of CS, NKU - 2021.09~2022.09