Header

Search

Kathrin presents 2 Papers at RecSys 2026

Kathrin presents 2 Papers at RecSys 2026

Full paper, main track: The Utility of LLMs in Recommender Systems Explanation Evaluation

The paper revolves around the problem that many explanation methods exist, but little guidance exists on which is best for which setting. Existing explanation generation methods often produce abstract outputs that require further formatting to become user-friendly, with a seemingly endless pool of options. Running user-based evaluations of all possible options is usually unfeasible, while automated evaluation metrics often either assess only the explainer’s abstract output or require comparison with a ground truth, which is generally unavailable. 
It explores how large language models (LLMs) can serve as “judges” for explanation evaluation to select an effective explanation method for a given application. We first explore their ability to generate explanation prototypes given varying information about the RS and the user. We compare the generated explanations against human ratings derived from a user study. Our results show that while LLMs exhibit human-like rating patterns and achieve moderate rank correlation with human raters, their absolute rating agreement is low and varies substantially by model size and evaluation construct. We derive four practical recommendations: keep explanation-generation prompts concise, prefer larger models for evaluation, pre-test evaluation constructs, and audit explanations for factual accuracy, as neither humans nor LLMs reliably detect non-factual content. 

ToRS Highlight Paper: Whom do Explanations Serve? A Systematic Literature Survey of User Characteristics in Explainable Recommender Systems Evaluation
Context: the ACM Transactions on Recommender Systems journal selected 3 papers to be presented in a special session during the main conference. This was one of them.

The paper is a large-scale literature survey looking at the participants of user studies that evaluate recommender systems explanations and how their characteristics impact the measured effects of explanations on users. Our findings suggest that the results from the surveyed studies predominantly cover specific users who do not necessarily represent the users of recommender systems in the evaluation domain. This may seriously hamper the generalizability of any insights we may gain from current studies on explanations in recommender systems. We further find inconsistencies in the data reporting, which impacts the reproducibility of the reported results. Hence, we recommend actions to move toward a more inclusive and reproducible evaluation.

Subpages