AI Agents 相关度: 8/10

The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

Dusan Bosnjakovic
arXiv: 2602.17127v1 发布: 2026-02-19 更新: 2026-02-19

AI 摘要

论文提出一种新框架,利用心理测量理论审计LLM的潜在偏差,发现供应商级别的行为特征。

主要贡献

  • 提出一种基于心理测量理论的LLM潜在偏差审计框架
  • 使用强制选择排序小品和语义正交诱饵量化LLM的偏差
  • 发现LLM中存在显著的“实验室信号”,揭示潜在偏差的供应商级别行为特征

方法论

利用心理测量理论,通过强制选择排序小品和统计模型(MixedLM, ICC)量化LLM的优化偏差、谄媚和现状合法化等倾向。

原文摘要

As Large Language Models (LLMs) transition from standalone chat interfaces to foundational reasoning layers in multi-agent systems and recursive evaluation loops (LLM-as-a-judge), the detection of durable, provider-level behavioral signatures becomes a critical requirement for safety and governance. Traditional benchmarks measure transient task accuracy but fail to capture stable, latent response policies -- the ``prevailing mindsets'' embedded during training and alignment that outlive individual model versions. This paper introduces a novel auditing framework that utilizes psychometric measurement theory -- specifically latent trait estimation under ordinal uncertainty -- to quantify these tendencies without relying on ground-truth labels. Utilizing forced-choice ordinal vignettes masked by semantically orthogonal decoys and governed by cryptographic permutation-invariance, the research audits nine leading models across dimensions including Optimization Bias, Sycophancy, and Status-Quo Legitimization. Using Mixed Linear Models (MixedLM) and Intraclass Correlation Coefficient (ICC) analysis, the research identifies that while item-level framing drives high variance, a persistent ``lab signal'' accounts for significant behavioral clustering. These findings demonstrate that in ``locked-in'' provider ecosystems, latent biases are not merely static errors but compounding variables that risk creating recursive ideological echo chambers in multi-layered AI architectures.

标签

LLM Bias Alignment Psychometrics Auditing Generative AI

arXiv 分类

cs.CL