Changing Risk Equation with AI

• Limited Visibility: Companies may procure an HRMS, CRM, cybersecurity tool, or SaaS platform without having complete visibility into the models, datasets, AI providers, or agentic capabilities embedded within it. AI can also be introduced through updates to existing software.
• Loss of Control Over Data: Third-party AI can access and process sensitive customer, employee, operational, or proprietary information. This can make it hard to understand where that data goes, whether it is retained, whether it can be used for model training, and what downstream parties can access it.
• Excessive System Access: AI agents can require credentials, APIs, databases, email, files, or other systems to perform useful work. The danger isn’t merely what an AI knows but what it is authorized to do. An overly permissive third-party agent can dramatically expand the consequences of unexpected behavior.
• Unpredictable Agentic Behavior: Traditional software generally operates within predefined workflows. An AI agent pursuing an objective may determine its own intermediate actions. Companies need to evaluate not only intended functionality but what an agent might attempt when its preferred route fails.
• Model and Product Changes: The AI assessed during procurement may not remain the same AI. Vendors can change models, prompts, integrations, capabilities, underlying providers, or autonomy levels. A one-time vendor assessment therefore becomes increasingly stale.

Preserving the Value of Third-Party AI

• Inventory of Third-Party AI: We cannot govern third-party AI we do not know we are using. Build an inventory that extends beyond vendors explicitly selling an “AI product” and identify AI capabilities embedded in HRMS, CRM, cybersecurity, productivity, analytics, cloud, customer service, and other platforms. For each system, document the vendor, AI capability, intended purpose, affected department, internal owner, underlying model or provider where known, data involved, integrations, and degree of autonomy.
• Vendor Pre-Assessment: Before introducing a third-party AI system, evaluate whether the vendor itself has adequate AI governance practices. Use this assessment to confirm how the vendor develops and tests AI, evaluates security and privacy risks, handles vulnerabilities, manages training and customer data, monitors deployed models, responds to incidents, and governs its own downstream providers.
• Use Case Assessment: Approving Microsoft, Salesforce, Workday, or another provider does not mean every AI capability it offers presents the same risk. An AI system summarizing internal documents is fundamentally different from one screening employment candidates, determining eligibility, or communicating with users. Each use case should therefore be assessed according to its purpose, affected population, data involved, potential consequences, degree of autonomy, and ability to influence consequential decisions.
• Map Data Access: Traditional vendor management asks what information a provider receives. AI governance must go further and define what the AI can access and how it can use that information. Identify whether a system can access sensitive records, employee information, emails, documents, databases, case files, credentials, or other sensitive information.
• Monitor Vendor and Model Changes: An AI assessment is a snapshot of a system that may continue changing. Vendors can replace foundation models, update model versions, introduce new agents, expand integrations, alter data practices, change subcontractors, or give existing AI capabilities greater autonomy. Set clear reassessment triggers rather than treating vendor approval as permanent. Material changes in the model, functionality, data access, use case, or downstream providers should prompt review appropriate to the resulting risk.
• Maintain an Audit Trail: Retain evidence showing how and why the third-party AI systems were approved and subsequently governed. Keep vendor assessments, use-case assessments, risk classifications, approvals, testing results, identified safeguards, contractual commitments, human-oversight requirements, model changes, incidents, remediation actions, and periodic reassessments.

Trust, but Govern