From Zero to RAGs. Balancing job-NER performance with Token Cost

Abstract

We investigate prompt-optimization strategies for domain-specific named entity recognition in job advertisements by balancing the extraction performance against the number of tokens. Using the SKILLSPAN corpus, we implement six pipelines combining three prompting methods, zero-shot, hard-coded few-shot, and dynamic RAG-based few-shot, with optional RAG-based semantic prefiltering. Each pipeline extracts skills and via GPT-4o-mini, measuring F1, precision, recall, and average tokens per advertisement. The results show that dynamic RAG-few-shot without prefiltering achieves the highest F1 (≈71% for knowledge, ≈60% for skills) and that prefiltering might reduce token usage by up to 70% while modestly lowering recall. Compared to zero-shot, few-shot prompting, especially with RAG retrieval, yields substantial recall gains of up to 28% at the cost of precision. Our findings demonstrate that RAG-augmented few-shot prompting offers an effective, token-efficient solution for specialized NER tasks.