Evaluation of Language Model–Based Services on CAISE Platform: Methods, Metrics, and Applications

Abstract

Language models are increasingly used as core components of web services, providing functionality ranging from classification and information extraction to text completion and long-form generation. In these systems, the model’s behavior directly impacts service quality and reliability, making task-specific evaluation essential. However, designing effective evaluation strategies is non-trivial: discriminative and generative models require different approaches, and no single metric works well across all tasks, domains, or deployment scenarios. This paper provides a practical overview of evaluation methods for language model-based services, covering both discriminative and generative models and highlighting the strengths and limitations of each approach. For discriminative tasks, we summarize commonly used label-based metrics, including accuracy, precision, recall, and F1. For generative tasks, we describe evaluation approaches that restrict outputs (closed-ended questions and exact match), reference-based metrics (BLEU, ROUGE), semantic similarity metrics (BERTScore), and model-based evaluation using LLM-as-a-Judge. We demonstrate the applications of these evaluation methods through case studies from the CAISE platform, a cloud initiative that supports Polish SMEs in developing intelligent services. The presented examples span both general-language and domain-specific evaluation and use multiple complementary metrics. Overall, the paper provides a practical guideline for designing evaluation pipelines for language-model-based services in real-world settings.