# Training Data, Validation, and Test Sets – Why the Split Matters for Security When training a model, the goal is never to find out how well it memorizes examples it has already seen. The goal is to find out how well it behaves on new, unseen examples. That distinction requires deliberately splitting data into separate sets – and understanding exactly what each set is for. Without this discipline, you can prove almost anything, including complete nonsense. ### Overview | Split | Purpose | Security relevance | | -------------- | ----------------------------------- | --------------------------------------- | | **Training** | Model learns from this | Poisoning attacks target this | | **Validation** | Development decisions are made here | Tuning against val = biased development | | **Test** | Final, honest evaluation | Contamination here destroys all claims | ### The Three Data Splits #### A. Training Data The training set is what the model learns from. The model sees examples, adjusts its parameters, and tries to approximate the patterns in the data. **Purpose:** learn patterns, approximate relationships, adapt behavior to data. > **Training is the model's learning world.** #### B. Validation Data The validation set is not used for learning directly – it is used for development decisions during the process. With the validation set you check: * which model variant performs better * which hyperparameter settings make sense * when to stop training (early stopping) * whether the model is starting to overfit **Purpose:** model comparison, hyperparameter tuning, early stopping, development decisions. > **Validation is the workbench for model decisions.** **Critical point:** Even though the model does not train directly on validation data, your decisions are shaped by it. The validation set is therefore not neutral – it influences the final model indirectly. #### C. Test Data The test set remains untouched until the very end. It exists to provide the most honest possible measurement of how well the model performs on genuinely unseen data. **Purpose:** final evaluation, unbiased performance estimation, reality check after development. > **Test is the final exam.** The moment you use the test set to make further adjustments, it stops being a test set. It becomes validation with a more impressive name. ### A Concrete Example: 1,000 Samples A typical split for a dataset of 1,000 samples: ``` Training: 700 samples (70%) Validation: 150 samples (15%) Test: 150 samples (15%) ``` The exact ratio is not sacred. What matters: clear separation, meaningful representation in each split, and no unintended overlap between them. ### The Most Common Misconception **"Validation and test are both just data the model hasn't trained on – what's the difference?"** This is too coarse. The crucial distinction: * **Validation** shapes your development decisions * **Test** is supposed to be completely free of that influence As soon as you repeatedly consult the test set to tune and improve, you have turned it into a second validation set. The measured performance is no longer honest. ### Data Leakage – The Silent Killer Data leakage means that information from parts of the data that should remain unknown has crept into training or model decisions. When that happens, you are no longer measuring honest performance. #### Forms of Leakage **A. Direct overlap** Examples from training appear in the test set, or nearly identical variants do. The model effectively already knows the answers. **B. Derived overlap** Not the same data, but similar enough that the model has effectively seen the concept. Benchmark performance looks strong, real-world performance collapses. **C. Feature leakage** A feature indirectly reveals the target label, even though it would not be available or meaningful in a real-world deployment. Example: in a safety classifier, all harmful examples happen to be shorter than all harmless ones. The model learns "short = harmful" rather than learning the actual content patterns. Benchmark performance looks great. Deployment performance is unreliable. **D. Process leakage** Information from test data inadvertently influences preprocessing, feature selection, or normalization steps. Common when pipelines are built without strict data isolation. **E. Temporal leakage** Training and test data overlap in time, when they should not. In security systems especially, training on data from the same time window as test data means you are measuring pattern recognition of known threats, not generalization to new ones. Intrusion detection and malware classification are classic examples where temporal leakage produces misleadingly optimistic results. ### Benchmark Contamination in LLMs For large language models, leakage takes a specific and particularly insidious form: **benchmark contamination**. Public benchmarks like MMLU, HumanEval, TruthfulQA, or HellaSwag are used to evaluate LLM capabilities. But these benchmarks are also text on the internet – and pre-training corpora are scraped from the internet. The model may have seen the test questions during pre-training. This is not hypothetical. The GPT-4 Technical Report explicitly discusses contamination analysis. Several academic evaluations have found performance differences consistent with contamination. **The security-relevant consequence:** when a paper or vendor claims a model achieves X% on benchmark Y, you cannot automatically treat that as evidence of generalization or robustness. The benchmark may have been in the training data. This is especially critical when that benchmark is being used to argue for adversarial robustness or safety properties. The right question when reading any LLM paper with benchmark results: **was contamination analysis performed, and how?** ### Why Good Metrics Can Deceive A model can achieve high accuracy, precision, recall, or F1 and still be completely useless in practice. Four reasons: **1. The test set was too similar to training.** You measured memorization, not generalization. **2. There was leakage.** You measured the artifact, not the capability. **3. The model learned the wrong proxy.** You measured a spurious correlation, not the actual signal. **4. The test set does not represent the real attacker world.** You measured best-case behavior, not adversarial behavior. A model can shine on benchmarks and fall apart under adversarial conditions like a cheap garden chair in a storm. The distance between "works on the test set" and "resists a motivated attacker" is the gap where most AI security failures live. ### A Security Example: The Request Classifier You train a classifier to detect malicious web requests. Training data: * benign requests from source A * malicious requests from source B Source B happens to always use a specific header format that source A does not. The model does not learn "this request is malicious." It learns "this header format is suspicious." Benchmark performance looks excellent – as long as the test data comes from the same sources. In production: an attacker who does not use that header format bypasses the classifier entirely. A legitimate system that happens to use that format gets blocked. What happened here: * no genuine security understanding * no robust detection * dataset artifact + leakage/bias effect This pattern is extremely common in published security ML research. ### Cross-Validation A single train/val/test split has a weakness: it depends heavily on which examples ended up in which partition. With smaller datasets – common in security research, where labeled examples of specific attack types are rare – k-fold cross-validation is the standard approach. In k-fold CV, the data is divided into k equal parts. The model is trained k times, each time using k-1 parts for training and 1 part for validation. Performance is averaged across all k runs. This gives a more stable estimate of generalization performance and is less sensitive to the particular random split. For the final test evaluation, a held-out test set is still used separately – k-fold CV replaces only the validation phase, not the final test. ### Transfer to AI Security Research When evaluating any model, system, or paper, these questions are mandatory: * How were training, validation, and test split? * Were similar or related samples separated across splits? * Was there any tuning against the test set? * Are the data sources independent? * Is there any feature that indirectly reveals the label? * Does the test set measure actual generalization, or just proximity to training data? * Was contamination analysis performed for LLM benchmarks? * Are the time windows for training and test data separated appropriately? If these questions are not clearly answered, treat all performance claims with skepticism. #### LLM Evaluation If prompts, tasks, or benchmarks are too close to the model's training distribution, capability is overestimated. The InstructGPT paper (Ouyang et al. 2022) acknowledges this; the GPT-4 Technical Report discusses contamination analysis explicitly. #### RAG Evaluation If test questions are formulated too close to the indexed documents or derived from the same source patterns, you are measuring retrieval convenience, not robust performance. #### Agent Evaluation If tool-use tests only cover friendly, expected inputs, you are measuring best-case behavior, not security. Real adversarial inputs are always out-of-distribution relative to the training distribution of agent demonstrations. ### Three Mini-Scenarios **Scenario A – Campaign overlap** You train a malicious PDF detector. Later you discover that many test PDFs come from the same malware campaign as the training PDFs. Problem? Yes. The test set is not a valid measure of generalization to new campaigns. **Scenario B – Test set reuse** You use the test set multiple times to sharpen model parameters after seeing the results. Problem? Yes. The test set has become part of the development process. It is now validation with extra steps. **Scenario C – Unavailable feature** A feature in the dataset strongly predicts the label, but that feature would not be available in a real deployment system. Problem? Yes. Classic feature leakage. The model has learned a shortcut that works in the dataset and nowhere else. ### Key Takeaways * **Training, validation, and test serve fundamentally different purposes.** Conflating them destroys the validity of any performance claim. * **Validation is not neutral.** It shapes development decisions and therefore indirectly influences the final model. * **Test contamination is especially critical for LLMs.** Benchmark results from models trained on internet-scale corpora cannot be taken at face value without contamination analysis. * **Temporal leakage is the most common mistake in security ML.** Training and test data from the same time window measures recognition, not generalization. * **Good metrics on a flawed evaluation design prove nothing.** High accuracy, F1, or BLEU on a contaminated or leaked dataset is not evidence of robustness. ### References Magar & Schwartz (2022): Data Contamination: From Memorization to Exploitation: {% embed url="" %} Jacovi et al. (2023): Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination: {% embed url="" %} OpenAI (2023): GPT-4 Technical Report (Section on Contamination Analysis): {% embed url="" %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://sommercode.gitbook.io/ai-security/basic-concepts/training-data-validation-and-test-sets-why-the-split-matters-for-security.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.