Model Poisoning

“Model poisoning” is an adversarial Machine Learning (ML) attack in which a threat actor manipulates training data, training procedures, or model updates so that a ML model learns attacker-chosen behavior while appearing to function normally.

Expanded Explanation

1. Technical Function and Core Characteristics

Model poisoning alters the parameters or decision boundaries of a ML model by injecting malicious gradients, updates, or data during training or retraining. The attacker modifies the learning process so the final model embeds hidden or overt malicious behavior.

In federated and distributed learning, model poisoning often targets local client updates or aggregation mechanisms to introduce backdoors, misclassifications, or degraded performance. The attack can be stealthy, preserving aggregate accuracy while triggering incorrect outputs under specific inputs or conditions.

2. Enterprise Usage and Architectural Context

Enterprises that use ML for authentication, fraud detection, content filtering, recommendation, or industrial control can face model poisoning risks at data pipelines, training infrastructure, and model update channels. Adversaries may exploit weak data provenance, unsecured clients, or compromised orchestration systems.

Security and architecture teams address model poisoning within broader Artificial Intelligence (AI) security, data governance, and Machine Learning Operations (MLOps) frameworks using secure data collection, authenticated update protocols, robust aggregation, anomaly detection on gradients or parameters, and controlled retraining workflows that restrict untrusted input sources.

3. Related or Adjacent Technologies

Model poisoning relates to data poisoning, which targets the training dataset rather than model update channels, and to backdoor attacks, which implant triggers that cause targeted misclassifications. It also relates to adversarial examples, which manipulate inputs at inference time instead of the training process.

Defenses against model poisoning intersect with secure federated learning, Differential Privacy (DP), robust statistics, secure multiparty computation, and model auditing methods. Standards and guidance from security and standards bodies on AI risk management and adversarial ML provide reference practices for addressing these attacks.

4. Business and Operational Significance

Model poisoning can alter business decisions that depend on automated models, such as credit approvals, cyber defense alerts, or safety monitoring outcomes. It can create financial loss, compliance violations, and exposure of sensitive processes if models act in attacker-controlled ways.

Enterprises incorporate model poisoning into threat modeling, vendor due diligence, and AI risk assessments, and they adopt monitoring, validation, and incident response procedures for models in production. Governance programs for AI and data often include explicit controls to reduce opportunities for model poisoning attacks.