GOJA: local LLMs for structured extraction

The question

GOJA is an annotation scheme and dataset for German job advertisements: eight entity types connected by eleven typed relations, annotated on 500 ads, with boundaries that split German compounds into their meaningful parts (Krüger et al., EMNLP 2025). A supervised model trained on it reaches an entity F1 of 0.801 on character spans and an end-to-end relation F1 of 0.580.

Supervised extraction works, but changing the scheme or the domain can mean new annotation and retraining. My study tested the alternative: send a prompt and one ad to a general-purpose model, get one JSON object back with entities and relations, validate it against the scheme, and score it against the human annotations.

Setup

• Hardware: a MacBook Pro with an Apple M1 Max and 64 GB of unified memory, models served through Ollama, one request at a time, temperature 0.
• Data: 500 ads; two serve as worked examples in the prompt, so 498 are evaluated.
• Pipeline: prompt, parse, schema check (labels and internal references), one repair attempt when the JSON or the schema breaks, then scoring. Hydra manages configuration; MLflow records parameters, prompts, metrics and predictions. The code is public: goja-llm-extraction.
• Metrics: entity F1 (surface text plus type), relation F1 (both endpoints and the type must be right), and, kept separate, the share of valid JSON and of schema-valid output.

Findings

Where it fails

The errors are mostly omissions, not confusions: when the model finds the intended words, it usually types them correctly. That limits what any label-correction layer can add. By entity type, Prozess scored best (F1 0.627) and Attribut worst (0.137, with recall 0.094), a type on which human annotators also show low agreement. Relations are weaker still: objects being transformed reach 0.338, most other relation types stay below 0.10.

Prompt lesson

Moving from short label descriptions to a detailed prompt clearly helped. Past that point, length explained neither quality nor reliability: as compound-splitting rules piled up, the share of valid JSON on first attempt fell from 0.88 to 0.48. Two complete annotated examples gave the best balance of entity F1, relation F1 and valid output. The full German annotation guidelines (37,727 bytes) scored 0.387, below the shorter English few-shot prompt.

Fine-tuning pilot

A deliberately small QLoRA adapter trained on the same laptop in 106.5 minutes, with 32.8 GB peak memory. Against a paired frozen-base control, entity F1 moved from 0.392 to 0.404 and relation F1 from 0.117 to 0.130, with confidence intervals that included zero. Local training works; a reliable gain needs more than this.

What it means in practice

• A prompted local model can produce GOJA-format output without task-specific training. It's good enough for controlled model comparisons and human-reviewed pre-annotation, not for replacing the supervised extractor.
• Choose models by measuring on your own data: in this study, neither size nor family reputation predicted the result.
• Keep structural checks in code and report valid output next to accuracy. They can catch broken JSON without reference answers; they can't catch a missing entity.
• Throughput matters: at 31 seconds per ad, 500 ads take about four hours on a laptop, and millions of ads need batching and server hardware whichever model you pick.

The next steps the report supports are validation-led model selection and GOJA-specific adaptation, each checked on held-out data. The evaluation rules behind this project are written up in how to evaluate LLM output.