text_only_lr7e6_lora_vit_350

LoRA adapter checkpoints for Qwen/Qwen3-VL-Embedding-2B trained for image-screenshot contrastive retrieval.

Available checkpoints

• checkpoint-50
• checkpoint-100
• checkpoint-150
• checkpoint-200
• checkpoint-250
• checkpoint-300
• checkpoint-350

Eval metrics

Evaluated on test_miniv6 (200 queries / 5291 tiles) and test_miniv8 (400 queries / 7426 tiles), QA scored by GPT-4.1 grader against Qwen3-VL-4B-Instruct reader answers.

step	v6 R@1	v6 R@3	v6 QA	v8 R@1	v8 R@3	v8 QA
0	0.65	0.8	0.675	0.6875	0.8325	0.725
50	0.655	0.8	0.685	0.69	0.835	0.7325
100	0.65	0.755	0.685	0.6575	0.8	0.72
150	0.64	0.77	0.66	0.6525	0.795	0.7025
200	0.64	0.775	0.66	0.655	0.81	0.72
250	0.635	0.78	0.665	0.6425	0.805	0.7075
300	0.63	0.79	0.68	0.6425	0.815	0.71
350	0.635	0.785	0.675	0.635	0.8125	0.705

Usage

from peft import PeftModel
from transformers import AutoModel

base = AutoModel.from_pretrained("Qwen/Qwen3-VL-Embedding-2B", trust_remote_code=True)
model = PeftModel.from_pretrained(base, "Chrisyichuan/text_only_lr7e6_lora_vit_350",
                                  subfolder="checkpoint-200")

Run config

# Run: text_only_lr7e6_lora_vit_350

- **Ablation**: Train 100% on text-only QA pairs (no image contrastive) for 350 steps. Measure how much retrieval/QA can be learned from text alone — LoRA-tuned LLM still affects how the doc tower (image→ViT→merger→LLM) embeds image inputs at eval time, so the experiment isolates "language-side learning vs. image+language learning".
- **Date**: 2026-04-28
- **Machine**: hb-h1-01
- **GPUs**: 1× H100 (CUDA_VISIBLE_DEVICES=1)
- **Code change**: Added eval/save/test triggers inside `train_contrastors.py` text-warmup loop (line ~2413) so we can observe the curve over 350 text-only steps. Without this change, only step-0 baseline + final eval would emit data.
- **Key args**:
  - `--data-split-dir training/data/natural_filtered_4o_40k/split` (only used for eval-loss val split)
  - `--text-warmup-steps 350 --max-steps 350 --text-data-dir data/text-qa-pair` (14952 pairs, ~1.5 epochs)
  - `--test-data test_miniv6/test_miniv6.json test_miniv8/test_miniv8.json`
  - `--batch-size 64 --grad-cache-chunk 4 --num-hard-negatives 2`
  - `--lr 7e-6 --warmup-steps 20 --scheduler cosine`
  - `--max-num-visual-tokens 4096 --lora-vit --skip-image-verify`
  - `--simpleqa-max-examples 1000`
  - `--vllm-url http://localhost:8201/v1 --vllm-model Qwen/Qwen3-VL-4B-Instruct`
  - `--eval-steps 25 --test-eval-steps 50 --save-steps 50`
- **Baseline**: `v8r_4o40k_warmup50_lr7e6_lora_vit_350` (50 text + 300 image), `colin_v8r_warmup50_lr7e6_lora_vit_350` (v2 dataset).
- **Hypothesis**: Text-only training will lift QA modestly above base (text contrastive teaches the LLM to produce more retrieval-friendly representations) but plateau well below the image+text recipe; gap quantifies the value of image contrastive signal.