Welcome to visit ACADEMIC MONTHLY,Today is

Volume 57 Issue 6
June 2025
Article Contents

Citation: GONG Weigang and HUANG Siyuan. Approximating Facts or Reproducing Bias——Evaluating LLMs in Social Research[J]. Academic Monthly, 2025, 57(6): 123-136. shu

Approximating Facts or Reproducing Bias——Evaluating LLMs in Social Research

  • The application of large language models (LLMs) in the social sciences is expanding rapidly, yet whether the data they generate accurately reflect real-world social phenomena remains contested. Using the 2021 Chinese General Social Survey (CGSS) as a benchmark, this study develops a multi-model comparative framework to systematically assess the representativeness and biases of "silicon-based samples" produced by various LLMs. The results show that mainstream models can reproduce statistical relationships among macro-level variables, but they exhibit representational biases—tending to reinforce dominant discourses while marginalizing alternative perspectives. Through the incorporation of Chain-of-Thought (CoT) analysis, we find that the models generate standardized causal reasoning structures when explaining their responses, revealing implicit pathways of social cognition embedded in their outputs. Furthermore, prompt design and fine-tuning mechanisms may inadvertently shape users' perceptions of public issues. This paper highlights both the potential and limitations of LLMs in social measurement and recommends enhancing data diversity, improving model interpretability, and developing domain-specific models tailored for the social sciences.
  • 加载中

Article Metrics

Article views: 184 Times PDF downloads: 5 Times Cited by: 0 Times

Metrics
  • PDF Downloads(5)
  • Abstract views(184)
  • HTML views(35)
  • Latest
  • Most Read
  • Most Cited
        通讯作者: 陈斌, bchen63@163.com
        • 1. 

          沈阳化工大学材料科学与工程学院 沈阳 110142

        1. 本站搜索
        2. 百度学术搜索
        3. 万方数据库搜索
        4. CNKI搜索

        Approximating Facts or Reproducing Bias——Evaluating LLMs in Social Research

        Abstract: The application of large language models (LLMs) in the social sciences is expanding rapidly, yet whether the data they generate accurately reflect real-world social phenomena remains contested. Using the 2021 Chinese General Social Survey (CGSS) as a benchmark, this study develops a multi-model comparative framework to systematically assess the representativeness and biases of "silicon-based samples" produced by various LLMs. The results show that mainstream models can reproduce statistical relationships among macro-level variables, but they exhibit representational biases—tending to reinforce dominant discourses while marginalizing alternative perspectives. Through the incorporation of Chain-of-Thought (CoT) analysis, we find that the models generate standardized causal reasoning structures when explaining their responses, revealing implicit pathways of social cognition embedded in their outputs. Furthermore, prompt design and fine-tuning mechanisms may inadvertently shape users' perceptions of public issues. This paper highlights both the potential and limitations of LLMs in social measurement and recommends enhancing data diversity, improving model interpretability, and developing domain-specific models tailored for the social sciences.

          HTML

        目录

        /

        DownLoad:  Full-Size Img  PowerPoint
        Return