TASK Quarterly

Evaluation of Language Model–Based Services on CAISE Platform: Methods, Metrics, and Applications

Mateusz Gierszewski — 2026-07-07

Language models are increasingly used as core components of web services, providing functionality ranging from classification and information extraction to text completion and long-form generation. In these systems, the model’s behavior directly impacts service quality and reliability, making task-specific evaluation essential. However, designing effective evaluation strategies is non-trivial: discriminative and generative models require different approaches, and no single metric works well across all tasks, domains, or deployment scenarios. This paper provides a practical overview of evaluation methods for language model-based services, covering both discriminative and generative models and highlighting the strengths and limitations of each approach. For discriminative tasks, we summarize commonly used label-based metrics, including accuracy, precision, recall, and F1. For generative tasks, we describe evaluation approaches that restrict outputs (closed-ended questions and exact match), reference-based metrics (BLEU, ROUGE), semantic similarity metrics (BERTScore), and model-based evaluation using LLM-as-a-Judge. We demonstrate the applications of these evaluation methods through case studies from the CAISE platform, a cloud initiative that supports Polish SMEs in developing intelligent services. The presented examples span both general-language and domain-specific evaluation and use multiple complementary metrics. Overall, the paper provides a practical guideline for designing evaluation pipelines for language-model-based services in real-world settings.

Digital Paper: How PDF Stalled the Evolution of Documents

Magdalena Godlewska — 2026-07-07

Despite its limited support for semantic representation, the PDF format remains the dominant medium for document publication due to its visual fidelity, portability, and widespread acceptance. As a result, contemporary documentprocessing workflows often treat PDFs as digital paper rather than structured data, creating a persistent gap between human-oriented presentation and machine-based processing requirements. PDF was designed primarily to preserve visual appearance, not document semantics or logical structure. Consequently, information extraction from PDF files is inherently complex and error-prone, as it requires reconstructing content hierarchy, relationships, and reading order from low-level layout cues. This limitation significantly constrains automated analysis, information retrieval, and AI-driven processing, particularly in domains such as law and public administration. This paper presents a PDF processing approach developed within the CAISE project that transforms arbitrary textbased PDF documents into a structured JSON representation. The solution preserves layout-related metadata, spatial localisation of elements, and embedded resources such as images and tables, while addressing character encoding corruption commonly found in real-world PDFs. The resulting structured data provide a foundation for reconstructing logical document structure and enabling downstream semantic processing. The proposed approach illustrates how presentation-oriented documents can be systematically transformed into machine-processable representations, mitigating the long-standing limitations of PDF-basedworkflows and supporting advanced document analysis and knowledge extraction.

Comparison of HPC and Cloud Environments for Developing Energy Efficient Services

Paweł Czarnul — 2026-07-07

The paper presents the characteristics and a comparison of HPC and cloud computing environments in the context of building and deploying services that support energy efficiency. The analysis covers typical hardware configurations, types of workloads (taking into account domain-specific features, application and software requirements), as well as the characteristics of use cases based on CPUs, GPUs, and their combinations. The scale of resource utilization and task execution performance in both environments is also discussed. The authors consider these aspects in the context of the potential migration and adaptation of the DEPO (Dynamic Energy-Performance Optimizer) software from an HPC environment to a cloud environment, with particular emphasis on performance-energy optimization. The paper also discusses the limitations resulting from differences between these environments and presents the hardware-software configurations in the cloud environment supported by the proposed cloud-based variant of the DEPO software.