COGNITIVE DIVERGENCE AS A PREDICTOR OF LLM OUTPUT COMPREHENSIBILITY: EXTENDING RELATIVE-TO-HUMAN BENCHMARKS BEYOND TRANSLATION TASKS.

This research investigates the role of Cognitive Divergence as a critical factor in determining the comprehensibility of Large Language Models' (LLMs) outputs. Despite recent breakthroughs in artificial intelligence that have greatly enhanced the readability and grammar of machine-generated text, there is uncertainty about the comprehensibility of such language for human readers. Established evaluation measures such as BLEU, ROUGE, and other textual similarity measures primarily focus on surface-level textual similarity and may not reflect deeper cognitive compatibility between human and machine-generated language. To overcome this challenge, this study proposes a benchmarking framework that considers LLM outputs in relation to human-generated text under similar circumstances. Cognitive Divergence is defined as a multi-dimensional difference, including syntax, semantics, and discourse structure. The research uses a multi-pronged approach, integrating quantitative and qualitative discourse analysis in various natural language processing tasks such as translation, summarization and question answering. The results show that Cognitive Divergence is a better predictor of comprehensibility than conventional evaluation measures. Texts with lower divergence scores are perceived as more comprehensible than others, even when factors such as grammatical accuracy and vocabulary complexity are accounted for. Moreover, the findings show this approach is not only suitable for the task of translation but can also be applied to a range of text generation scenarios. This research adds to the debate about human-centered AI evaluation by introducing a holistic approach that combines cognitive science with relative benchmarking. It also underscores the need to align machine language with human cognitive processes to improve human-machine interaction. This study has implications for the development of future evaluation metrics, creation of more human-centred language models, and enhancement of AI communication technologies.