Microsoft, Nvidia push provenance checks as open-weight AI adoption accelerates

Microsoft and Nvidia are urging enterprises to assess the security of open-weight AI models throughout deployment, warning that changes to model weights and configurations can weaken safeguards even when a model retains its ability to perform assigned tasks.

In a joint paper published last week, the companies identified 30 risk types across five categories and outlined a process for evaluating, mitigating, and monitoring them. The guidance calls for enterprises to verify model provenance, test the configurations they intend to deploy, and reassess risks after material changes.

“Weight openness alone does not determine an AI system’s risk,” the paper said. “Risk depends on the model, use case, deployment, and safeguards.”

The framework covers cybersecurity, data protection, content, human-AI interaction, and bias. Its cybersecurity category includes malicious code generation, malicious tool calls, extraction of the model’s safety policies, excessive token generation, and behavior that diverges from the intended system goal.

The paper draws on existing NIST and ISO risk-management standards and is the first in a planned series on managing risks across open-weight AI deployments, the companies said in a blog post.

Model changes can weaken safeguards

Open-weight models can be fine-tuned, merged with other models, modified, or repackaged for different uses. But those changes can weaken a model’s safety protections even when it continues to perform its intended task well, Microsoft and Nvidia said.

“Safety behaviors are often the more fragile of the two,” the companies wrote. Changes to model weights, fine-tuning, quantization and configuration can “degrade safeguards while leaving task performance intact.”

That creates a challenge for enterprise security teams: a model may continue to deliver accurate results on routine tasks even after changes have affected how it responds to unsafe requests. The companies advise enterprises to assess safety and capability separately, test the specific model configuration they intend to deploy, and repeat evaluations after significant changes.

“No volume of behavioral testing establishes what an artifact is, only how it behaves,” the paper said. It recommends using cryptographic signatures to verify that a model matches the publisher’s original artifact.

“Where signatures are unavailable, record the unverified integrity as an open risk rather than treating evaluation results as closing it,” the paper recommended.

For enterprises, Gartner recommends independent adversarial testing, static scanning of model files, and provenance verification before and after deployment. Gartner’s proposed vetting process also calls for verifying model sources, checking file integrity, conducting adversarial testing, and recording approved models in an enterprise registry.

Security must extend beyond the model

Microsoft and Nvidia caution that evaluating a model alone does not establish the safety of the application built around it.

“Model evaluation alone does not establish the safety of an agent or application; enterprises must also assess permissions, data access, connected tools, runtime behavior, and operational controls,” the paper says.

The guidance recommends assessing models in their intended operating context and maintaining controls across the model, runtime, harness, and application layers. It also calls for monitoring model inputs, outputs, and tool use after deployment.

According to Gartner, open-weight model traffic must be routed through governed AI gateways to enforce authentication, prompt-injection defenses, sensitive-data protection and audit logging.

The research firm also recommends isolating self-hosted runtimes and applying controls suited to other deployment models, including private network paths and customer-managed encryption keys for cloud API consumption.

Gartner reported that client inquiries about open-weight models through its research inquiry service rose by more than 30% in the second quarter of 2026 compared with the first quarter. The firm projects that open-weight models will account for 50% of enterprise AI use cases by 2028, up from 20% at the time of its report.

Staffing and accountability matter

Sanchit Vir Gogia, chief analyst at Greyhound Research, said self-hosting requires enterprises to account for the people and processes needed to maintain safeguards.

“Self-hosting is a staffing decision as much as an infrastructure one, and the people to carry it must move with the model,” Gogia said.

He cautioned against assuming that self-hosting necessarily means replacing all the safeguards available through managed services. In his assessment, a significant gap can be the provider’s misuse-investigation function, while customer responsibilities depend on the service and its configuration.

Gogia said the Microsoft-Nvidia paper provides useful guidance but leaves important areas for future work. The authors say subsequent publications will extend risk-management guidance across the runtime, harness, and application layers.

“The sharper question is who is contractually obliged to re-test and patch the model when a new stripping technique lands,” Gogia said. For CISOs, Gogia recommends assessing the model, deployment, and intended use before approving production workloads. He also advises security teams to retain control of their risk registers rather than relying exclusively on a vendor-developed taxonomy.