During the academic year 2024/2025, ReLU has collaborated with Scibsted Media on a project regarding image captioning and alt text generation at VG. The goal was to fine-tune LLM’s to generate image captions in VG-style, based on the article text and article image. In this way, it would be possible to automate some of the journalists' repetitive tasks. We also experimented with alt text generation doing prompt engineering, to ensure good quality alt texts for vision impaired readers.

Our task was to explore and identify the best models and different methods for generating image captions and alt text. We fine-tuned models like Gemma, Llama and Qwen to produce VG-style captions. During the project period, we tested both multimodal fine-tuning and pure text finetuning, and benchmarked this with GPT-4 Turbo. In addition to this, we experimented with an AWS-based face recognition system for identifying both international and VG celebrities.