Every model release is a marketing artifact first and an engineering artifact second. The launch post leads with one number, on one benchmark, under one prompt format — and the part you actually need (can I fine-tune it, can I serve it at my latency target, does it hold up on my distribution) lives in a config file, a license document, and an eval appendix nobody links. The tradeoff is blunt: an afternoon of verification now, or weeks of rework later when the thing you built on turns out to be API-only, non-commercially licensed, or benchmarked under a setup you cannot reproduce.

This is a standing checklist for spending that afternoon well. The methodology is evergreen; the specifics of any given launch are not. When you apply it to a real release, date your findings — licenses, runtime support, and “official” quant availability all change underneath you, often within the same week.

Access is the first fork

Open weights versus API-only is not a philosophical distinction. It is the concrete set of operations the release forecloses, and you should enumerate them before you read a single benchmark.

API-only forecloses, at minimum: fine-tuning beyond whatever hosted tuning the vendor sells you, private or air-gapped deployment, your own quantization, control over batching and the KV cache, a hard latency floor, custom logit processing, and reproducibility — the weights behind an endpoint can change silently, so a number you measured in March may not hold in June. Open weights hand you all of that, plus a maintenance bill: you now own the security patches and the serving stack.

Even inside “open,” grade the release:

  • Open weights, closed everything else — weights plus a license, no data, no training code. The common case. You can serve, tune, and quantize; you cannot reproduce the model or audit its provenance.
  • Open weights plus a technical report — a recipe sketch and an eval protocol. You can at least attempt to reproduce the headline evals, which is the difference between a measurement and a press release.
  • Fully open — data, code, weights, and logs. Rare; the OLMo (AI2) and Pythia (EleutherAI) releases are the reference points here. Treat this as the gold standard for verification, not the baseline you should expect.

The asymmetry worth internalizing: an API model can be deprecated or quietly updated under you, while an open-weights checkpoint is yours forever. That permanence is simultaneously the feature and the liability.

The license is a build constraint, not a footnote

You can describe license families without giving legal advice, and you should — because the family determines what you are allowed to ship. Three buckets cover most releases:

  • Permissive OSI licenses (Apache-2.0, MIT). Commercial use, redistribution, and derivatives are broadly allowed; Apache adds an explicit patent grant. As publicly documented, the DeepSeek-R1 weights shipped under MIT. These are the releases you do not have to lawyer before prototyping.
  • “Community” or open-ish licenses. Open weights with use restrictions and an Acceptable Use Policy attached. The Llama community license is the archetype: redistribution and commercial use are broadly permitted, but it carries an AUP and, as publicly documented, a clause requiring a separate license from Meta if your product exceeds 700 million monthly active users. Gemma ships under its own terms with a separate prohibited-use policy. Usable, but not OSI-open; read the AUP against your actual use case.
  • Research or non-commercial licenses. Some weights ship under CC-BY-NC or a vendor research license. Fine for a paper or an internal prototype, a non-starter for a product.

Two warnings that bite in practice. First, the license on the weights, the license on the code, and the terms governing the model’s outputs can all differ — some terms specifically restrict using outputs to train a competing model. Second, the license can change between versions of the “same” family; do not assume version n+1 inherits version n’s terms. The failure mode is reading a blog’s “open source” phrasing as a legal fact. Open the LICENSE file, read the AUP, and if revenue depends on it, route it to counsel. I am not giving legal advice here — I am telling you which file to open.

Read the card, not the launch post

The “card” is whatever passes for documentation: a model card in the sense of Mitchell et al.’s 2019 “Model Cards for Model Reporting,” a Hugging Face README plus a technical report for open weights, or a system card (the GPT-4 system card is the well-known template) for a frontier API model. Different artifacts, same job for you — extract three things.

  • Training-data disclosure. You will almost never get a manifest. You are looking for the data cutoff date, the languages and domain mix, and whether the eval sets were decontaminated. Treat “trained on a diverse mix of publicly available data” as exactly zero bits of information, and price contamination risk accordingly.
  • Eval methodology. A benchmark number without its setup is a claim, not a measurement. Which harness, how many shots, what prompt template, chain-of-thought or not, sampling or greedy, which split. If the card reports the score but not the protocol, log it as a rumor with a decimal point.
  • Known limitations and intended use. The honest cards name failure modes — weak languages, refusal behavior, hallucination on a stated domain. The absence of a limitations section is itself a signal, usually that the card was written by the marketing team.

A practical caveat on system cards from frontier labs: they are built around safety and dangerous-capability evaluations, not the throughput and serving facts you need. The engineering reality you will live with lives in the docs and the config, not in the card.

Two “same-benchmark” numbers are not comparable

This is the section practitioners most need and most often skip. “MMLU 84.2” from vendor A and “MMLU 85.1” from vendor B are not the same measurement unless the protocol matches, and the protocol almost never matches. The variables below routinely move a score by more than the gap separating two competing models:

  • Prompt formatting and answer extraction. Letter-choice versus free-form, regex parsing versus scoring the log-likelihood of each option, generate-and-parse versus loglikelihood ranking. The EleutherAI lm-evaluation-harness ships multiple MMLU variants precisely because these choices diverge by several points on identical weights.
  • Few-shot count and the exact exemplars. Zero-shot versus 5-shot, and which five examples — the specific shots matter, not just the count.
  • Chat template. Applying the wrong chat template (or none) to an instruct model can swing scores dramatically, and double-adding the BOS token or dropping the system prompt does it silently.
  • Sampling. Greedy versus a temperature; pass@1 versus pass@k on code, which are different questions.
  • Contamination. If the eval set or near-duplicates leaked into pretraining, the number is inflated. Detection is imperfect — n-gram overlap, canary strings, held-out completion probes — and labs rarely publish their decontamination procedure.

The practitioner move is to stop trusting the leaderboard cell and re-run the eval yourself, one harness, one protocol, all candidates. The absolute numbers will drift from the press release; the ranking under your protocol is what you actually wanted. And the only eval that fully counts is a held-out slice of your production distribution.

one_protocol.sh
# Same harness, same shots, same template — for every candidate model.
# Flags evolve; check `lm_eval --help` for your installed version.
lm_eval \
  --model vllm \
  --model_args "pretrained=org/new-model,dtype=bfloat16,gpu_memory_utilization=0.9" \
  --tasks mmlu,gsm8k \
  --num_fewshot 5 \
  --batch_size auto \
  --apply_chat_template \
  --output_path results/new-model

If lm-evaluation-harness does not cover your task, HELM (Stanford CRFM) is the other reference harness — but the principle is the same: one protocol, applied identically, or the comparison is noise.

Serving readiness: the gap between “released” and “runnable”

A model can be fully open and still cost you a week to put in front of traffic. Check these in order, because they fail in order.

  • Tokenizer. Is it actually shipped — a tokenizer.json or a known SentencePiece/BPE — and is the chat template present in tokenizer_config.json? How are the special tokens (BOS, EOS, pad) handled? A mismatched or missing chat template is the single most common day-one bug: the instruct model degrades quietly and your eval blames the weights.
  • Context length, nominal versus effective. config.json gives you max_position_embeddings and rope_scaling, which tell you the nominal window and how it was stretched (linear, NTK-aware, YaRN). Effective context — the length at which retrieval and reasoning actually hold — is usually shorter. Passing needle-in-a-haystack does not prove multi-hop or aggregation holds; RULER-style probes (NVIDIA’s “RULER” suite) are built to expose that gap. Treat “supports 1M tokens” as a nominal spec until you have measured it on your task.
  • Quantization support. Are official FP8/AWQ/GPTQ/GGUF weights released on day one, or are you waiting on the community? Does your runtime have efficient kernels for this architecture’s attention variant — grouped-query, sliding-window, MoE routing? A genuinely novel architecture usually means no fast kernel on launch day.
  • Day-one runtime support. New architectures typically land in transformers first, then in vLLM, SGLang, TGI, and llama.cpp over the following days to weeks. “Available on the Hub” is not “servable at your throughput and latency target.” Check the serving framework’s supported-models list and its open issues before you promise anyone a date.

A five-minute inspection catches most of these before you allocate a GPU:

inspect_release.py
from transformers import AutoConfig, AutoTokenizer


repo = "org/new-model"            # the Hub repo id
cfg = AutoConfig.from_pretrained(repo)
tok = AutoTokenizer.from_pretrained(repo)


print("arch:", cfg.architectures)
print("vocab:", cfg.vocab_size)
print("nominal ctx:", cfg.max_position_embeddings)
print("rope scaling:", getattr(cfg, "rope_scaling", None))  # None / linear / dynamic / yarn
print("has chat template:", tok.chat_template is not None)
print("special tokens:", tok.special_tokens_map)


# Does applying the chat template double-add BOS?
ids = tok.apply_chat_template(
    [{"role": "user", "content": "ping"}],
    add_generation_prompt=True,
)
print("starts with bos:", ids[0] == tok.bos_token_id, "first ids:", ids[:4])

If rope_scaling is populated, the long-context claim is an extension, not native — verify it. If the chat template is missing or double-adds BOS, fix that before you trust any eval you run.

The standing checklist

Apply this, in order, to the next release. Date every answer, because every answer expires.

  1. Access. Open weights or API-only? List exactly what that forecloses for your use — tuning, private deploy, quantization, reproducibility.
  2. License. Which file, and which family — OSI-permissive, community-plus-AUP, or non-commercial? Any MAU thresholds or output-use restrictions? Does it match your commercial plan? Open the LICENSE; if revenue depends on it, ask counsel.
  3. Card. Data cutoff, decontamination claim, eval protocol, stated limitations. Anything missing defaults to “assume the worst and verify.”
  4. Benchmarks. Is the full protocol published — harness, shots, template, sampling? Re-run under one fixed protocol across all candidates. Never compare cross-vendor cells.
  5. Serving. Tokenizer and chat template correct? Nominal versus effective context measured? Quant, kernel, and runtime support present on day one?
  6. Your eval. A held-out slice of your actual task, run before you commit anything. This is the only number that is yours.

The bottom line

A release is a hypothesis, not a result, until you have reproduced the part you intend to depend on. The headline benchmark is the least durable thing in the package; the license, the tokenizer, the effective context behavior, and your own held-out eval are what you will actually live with. The labs optimize the launch for the demo. Your job is to optimize for the Tuesday six weeks out when the thing is load-bearing — and that work is done with the LICENSE file, the config, and a fixed eval harness, not with the launch post.