Comparative Evaluation of Vanilla and Retrieval-Augmented Generation Approaches for a Domain-Specific Virtual Assistant in Maritime Vocational Education

Abstract

Deploying Large Language Models (LLMs) as vocational educational assistants poses risks, notably hallucination and inventing answers beyond local teaching materials. Literature often highlights Retrieval-Augmented Generation (RAG) as a mitigation, but empirical comparisons, especially in Indonesian maritime vocational institutions, are scarce. This research addresses that gap by testing two Virtual Boiler Assistant versions at Politeknik Pelayaran Surabaya: a Vanilla Gemini 2.5 Flash configuration and a RAG setup using the same model with semantic retrieval from 192 knowledge chunks (from four official boiler documents). Testing employed 20 validated ground-truth questions across procedure, calculation, and logic categories, with both models assessed under identical conditions using a three-point rubric and source cross-validation. The Vanilla model scored 50.0% (20/40), while RAG achieved 97.5% (39/40), demonstrating a consistent 47.5-point improvement. Hallucinations in Vanilla included numerical and procedural fabrications, and misinterpretation of the local "CBT" acronym. RAG responses cited sources in 85% of cases versus none from Vanilla. These findings show RAG substantially improves accuracy, hallucination resistance, and verifiability in LLM-based maritime vocational education, especially where local terminology diverges from broader industry usage.

Keywords       : retrieval-augmented generation; large language model; hallucination; maritime education; comparative evaluation