Difficulty-aware adaptive reasoning for Vietnamese VQA with GPT-OSS

Most questions about a photo don't need a model to think hard. A dense caption from Gemini 2.5 describes the image, CLIP scores how well that caption matches the picture, and GPT-OSS gets low, medium or high reasoning effort accordingly.

Vietnamese VQA models either answer fast and shallow or reason at length on every question, including the ones where a good caption already contains the answer. The cost of always reasoning hard is real: on ViVQA-X it takes almost twelve times the wall-clock time of the low-effort setting.

I designed the framework end to end: dense captioning, a CLIP similarity router, multi-model inference across GPT-OSS, Qwen3 and DeepSeek with vLLM and Unsloth, batch processing with resumable jobs, and the automated evaluation.

On ViVQA-X the adaptive mode finished in 815 s instead of 3,317 s, a 75% cut, and scored best on BLEU-4, ROUGE-L and METEOR.

• Router: CLIP image–caption similarity below 0.63 sends a question to high effort, 0.63–0.64 to medium, above 0.64 to low.
• Inference across GPT-OSS, Qwen3 and DeepSeek on vLLM, with periodic checkpoints so long jobs resume after interruption.
• Evaluation on ViVQA-X and OpenViVQA with BLEU, ROUGE-L, METEOR, CIDEr, SPICE, BERTScore, FLOPs and wall-clock time.

• PyTorch
• Transformers
• vLLM
• Unsloth
• CLIP
• Gemini 2.5

@inproceedings{tran2026adaptive,
  title     = {Difficulty-Aware Adaptive Reasoning for Vietnamese VQA with GPT-OSS},
  author    = {Tran, Dai-Nhan and Nguyen, Phuc-Thinh and Vu, Tue-Anh and Do, Bao and Trinh, Hai-Au and Nguyen, Anh-Khoi},
  booktitle = {AICI 2026},
  year      = {2026},
  note      = {Accepted},
  url       = {https://github.com/hugebenevolence/ViVQA-GPT-OSS-DRA}
}