A Vietnamese vision-language model, LLaVA style
SigLIP2 and Llama 3.2 1B joined by an MLP projector, trained in two stages.
Overview
SigLIP2 sees the image, a small MLP translates what it sees into the language model's embedding space, and Llama 3.2 1B answers in Vietnamese. Training happens in two stages, and what each stage is allowed to change is the whole design.
Stage 1 aligns modalities: the vision encoder and the language model stay frozen and only the projector learns, on Vietnamese COCO captions and UIT-OpenViIC. Stage 2 unfreezes the language model for visual instruction tuning on Vietnamese VQA data, plus tourism Q&A we crawled and generated with the OpenAI Batch API.
Distributed training runs on Accelerate with token-weighted loss and checkpoint-and-resume, evaluated on KTVIC captioning and Vista conversation, with Streamlit demos for captioning and chat.
Training stages
- Photo
- SigLIP2 so400m576 image tokens
- MLP projectorimage to word space
- Llama 3.2 1Bwrites Vietnamese
- Answer in Vietnamese
Stage 1, alignment
- Learns
- the projector only
- Data
- Vietnamese COCO 2017 captions and UIT-OpenViIC, sampled so each source counts equally
- Optimizer
- AdamW, learning rate 1e-3, 2 epochs
Stage 2, instruction tuning
- Learns
- the projector and the language model
- Data
- Viet-ShareGPT-4o Text-VQA, Viet-Localization-VQA, and tourism Q&A we crawled and generated with the OpenAI Batch API
- Optimizer
- Adafactor, 1e-4 for the projector and 2e-5 for the LLM, 1 epoch