Securing Locally Deployed AI Agents: Evaluating Model-Level Backdoors in Open-Weight LLMs
Title: Securing Locally Deployed AI Agents: Evaluating Model-Level Backdoors in Open-Weight LLMs
DNr: Berzelius-2026-359
Project Type: LiU Berzelius
Principal Investigator: Dejan Manojlo Kostic <dmk@kth.se>
Affiliation: Kungliga Tekniska högskolan
Duration: 2026-11-17 – 2027-06-01
Classification: 10201
Homepage: https://www.kth.se/blogs/dai/
Keywords:

Abstract

Recent advances in large language models (LLMs) have enabled AI agents to perform complex tasks involving software development, scientific research, and autonomous tool use. Organizations are increasingly deploying open-weight models on their own infrastructure to retain control over sensitive data and reduce dependence on third-party providers. However, local deployment protects neither against compromised model weights nor against malicious behavior introduced through third-party adapters. Trust in the model itself remains a critical security assumption. Our preliminary work illustrates this risk. During summer 2026, we demonstrated that a lightweight Low-Rank Adaptation (LoRA) module could cause an LLM-based coding agent to invoke a predetermined tool when a particular phrase appeared in its context. Follow-up experiments in October 2026 produced a noise-resistant backdoor in several smaller open-weight models. These observations motivate a systematic study of the reliability, detectability, and practical implications of model-level behavioral modifications. This project will investigate three research questions. First, how do these behaviors depend on model architecture, fine-tuning, and contextual conditions, including semantic properties of tasks rather than exact trigger strings? Second, can existing agent security measures, such as tool-use policies, execution monitoring, and inspection of observable reasoning traces, reliably detect and contain unauthorized behavior? Third, how can we design reproducible evaluations that measure both security failures and the impact of interventions on legitimate task performance? We will conduct controlled experiments across open-weight LLM families and agent frameworks using parameter-efficient fine-tuning, synthetic coding tasks, isolated execution environments, and harmless simulated tool actions. Evaluations will compare model variants and monitoring mechanisms using measures of unauthorized tool invocation, false activation, normal-task accuracy, and robustness across task settings. No real-world systems or sensitive datasets will be targeted. The expected outcome is an experimental understanding of the limitations of self-hosted AI security, together with reproducible benchmarks and evidence-based guidance for verifying model integrity and enforcing tool-use boundaries. This work will help organizations evaluate the security assumptions behind deploying autonomous open-weight agents locally.