India’s Leading Domain & Web Hosting Provider
Support
Domain & Login
Blog
AI Security and Governance for GenAI Apps, Agents and Model Use
AI security and governance is the set of technical controls and management processes that keep generative AI systems safe to operate: preventing prompt injection and data leakage, controlling which models and data each application uses, testing systems adversarially, and assigning accountability for AI risk. NTSPL implements these controls and governance structures for regulated and data-sensitive Indian enterprises.
Delivered on
In scope
The GenAI Attack Surface: Prompts, Retrieved Content, Tools and Outputs
GenAI arrived in most organisations through the side door. Business units signed up for tools, developers called model APIs with personal keys, and pilots went live on customer data before security had reviewed them. The new attack surface is unfamiliar: instructions hidden in an uploaded PDF that make an assistant reveal its system prompt, an agent tricked into emailing a file, or a chatbot that reconstructs a customer's details from retrieved context. Conventional application security reviews do not test for these, and risk committees are asking for an AI policy without a technical basis to write it.
NTSPL works on both layers. Technically, we inventory every model use, put model traffic behind controlled endpoints, deploy input and output filters for injection and sensitive data, enforce least privilege on retrieval and tools, and red-team each application against the OWASP LLM Top 10. On governance, we set up an AI risk register, use-case approval tiers, model cards and incident playbooks, mapped to ISO/IEC 42001, NIST AI RMF, the DPDP Act 2023 and sector expectations from RBI, SEBI or IRDAI.
Signals That AI Use Has Outgrown Your Current Security Controls
Signals you need AI Security & Governance
- More than five GenAI applications or tools are in use and no central inventory lists their models, data and owners.
- A penetration test or audit has found prompt injection, system-prompt leakage or sensitive data in model logs.
- Agents or assistants can call tools that send email, update records or access customer data.
- The board or a regulator has asked for an AI policy, risk register or evidence of responsible-AI controls.
- Staff are using public AI tools with company data and security has no approved alternative or monitoring.
When to choose something else
- If the need is general cloud security posture, such as IAM sprawl or exposed storage, start with a cloud security assessment; AI controls sit on top of that baseline.
- If the question is only about measuring answer quality and hallucination, that belongs to AI observability and evaluation, though it links to governance.
- If no GenAI system is yet in design or use, a short policy and approval process is enough until the first project starts.
Controls and Governance Artefacts We Implement
AI inventory and data-flow mapping
Discovery of every model endpoint, AI SaaS tool and embedded AI feature, with data classes, owners, regions and vendors recorded.
Model access and network controls
Private endpoints, approved model lists, key removal in favour of managed identities, and egress blocks on unapproved AI services.
Input and output protection
Prompt-injection and jailbreak detection, PII and secret filtering, grounding checks and output content policies per application.
Retrieval and tool least privilege
Permission-aware retrieval, scoped agent identities, approval gates for high-impact tools and logging of every tool call.
Adversarial testing
Red-team exercises against OWASP LLM Top 10 categories, including indirect injection through documents, emails and web content.
Governance operating model
AI policy, risk tiers, use-case approval workflow, model cards, AI incident playbooks and committee reporting.
Layered Control Plane for Generative AI Applications
Layered Control Plane for GenAI Applications and Agents
User and document inputs pass through injection and sensitive-data screening before reaching the model through a private, identity-controlled endpoint; retrieval and tools run with the user's or agent's least privilege, and outputs are screened before delivery. All events feed a security log and the AI risk register links every application to its owner, tier and test results.
Select any component
Layered Control Plane for GenAI Applications and Agents: click any of its 8 components to see its role and the design choice NTSPL makes for it, or trace the 7-step primary path.
Bedrock Guardrails, Azure AI Content Safety and Model Armor in Practice
Bedrock Guardrails provides prompt-attack detection, denied topics, sensitive information filters and grounding checks; PrivateLink keeps model traffic private, and CloudTrail records model invocations for audit.
Prompt Shields detects direct and document-based attacks, Purview adds data security posture for AI use, and Defender for Cloud provides threat protection for AI workloads alongside Entra managed identities.
Model Armor screens prompts and responses for injection, harmful content and sensitive data; VPC Service Controls draw a perimeter around AI and data services, and Sensitive Data Protection handles de-identification.
Filter Placement, Data Boundaries, Approval Tiers and Red-Team Depth
Where to place filters
Filter at three points: user input, retrieved or tool-returned content, and model output. Indirect injection through documents is now more common in our tests than direct jailbreaks, so content filtering cannot be skipped. Put filters in the AI gateway for consistency across applications, with per-application policies. Expect 30-150 ms of added latency per filter stage and budget for it.
Data boundary for model calls
Classify data before choosing endpoints. Public and internal data may use approved global endpoints; confidential and personal data should use in-region deployments with private networking and no provider-side retention beyond abuse monitoring where terms allow. Restricted data, such as Aadhaar numbers or health records, should be masked before any model call unless a documented exception exists.
Use-case approval tiers
Three tiers work for most organisations. Tier 1, internal productivity with no personal data: self-service approval against a checklist. Tier 2, internal decisions or personal data: security and privacy review plus evaluation results. Tier 3, customer-facing, financial or autonomous actions: red-team report, committee sign-off and quarterly review. Approval should take days for tier 1, not months.
Red-team depth
Scale testing with risk. Tier 3 applications get 3-5 days of manual red-teaming across OWASP LLM categories plus automated attack libraries of 1,000+ prompts, repeated per major release. Tier 2 gets automated testing and a one-day manual review. Every finding is retested after fixes, and attack prompts that succeeded become permanent regression tests.
Inventory to Assurance: A Six-Step Security Programme
AI discovery and inventory
Find every model endpoint, AI tool and embedded AI feature through interviews, egress logs, cloud billing and expense data.
Risk tiering and policy
Tier each use case, draft the AI policy and approval workflow, and agree accountabilities with risk and compliance.
Platform controls
Deploy private endpoints, AI gateway policies, injection and data filters, and route telemetry to the SIEM.
Application hardening
Fix retrieval permissions, tool scopes and prompt handling in priority applications.
Red-team and retest
Run adversarial testing on tier 2 and 3 applications, fix findings and add successful attacks to regression suites.
Governance cadence
Monthly AI risk committee reporting, quarterly reviews of tier 3 systems and inventory refresh.
What you receive
Security and Governance Measures for AI Systems
| KPI | Typical target |
|---|---|
| AI systems recorded in inventory with owner and tier | 100% of known systems, refreshed quarterly |
| Model traffic through approved private endpoints or gateway | Above 95% |
| Red-team attack success rate on tier 3 apps after fixes | Under 2% of automated attack set |
| Sensitive data detected in model logs | Zero unmasked restricted identifiers |
| Median approval time for tier 1 use cases | Under 5 working days |
Gaps We Commonly Find in GenAI Deployments
| Risk | How NTSPL prevents it |
|---|---|
| Indirect prompt injection through uploaded documents or emails | Separate screening of retrieved and tool-returned content, content treated as data not instructions, and tool approvals for any action. |
| Governance becomes a bottleneck and pushes teams to shadow AI | Fast-track tier 1 approvals, an approved internal AI tool for everyday use and egress blocks only after alternatives exist. |
| Personal data retained in prompts and logs beyond purpose | Masking at the gateway, defined log retention per data class and deletion jobs aligned to DPDP obligations. |
| Controls drift as applications add features and tools | Tool and data-source changes trigger re-tiering, and regression attack suites run in every release pipeline. |
Do we need separate AI governance, or can existing IT and data governance cover GenAI?
Extend what you have rather than building a parallel structure, but do not assume it already covers GenAI. Existing IT security, data classification and change management handle much of the baseline. What they miss are risks specific to models: prompt injection, unreliable outputs used in decisions, agents acting on systems, model vendor changes and training-data questions. Add an AI inventory, risk tiers, AI-specific testing and a named owner for each AI system, and report them through the existing risk committee. ISO/IEC 42001 is designed to integrate with an ISO 27001 management system in exactly this way.
Key facts: AI Security & Governance
AI Security & Governance in Practice
2 AI Security & Governance scenarios, from Wealth advisory (SEBI-registered investment advisory) and Public sector (state welfare-scheme citizen helpdesk), plus related platform case studies.
Wealth advisory firm red-teams its client-letter assistant and closes 23 findings
State welfare-scheme helpdesk blocks injection and PII leaks before public launch
Three directories become one: identity consolidation after two acquisitions
Health-tech SaaS moves patient data behind KMS and Identity Center ahead of DPDP
AI Security & Governance: Frequently Asked Questions
What is prompt injection and why does it matter?
Prompt injection is an attack in which text supplied by a user, or hidden in a document, email or web page the model reads, overrides the application's instructions. It can make an assistant reveal confidential context or make an agent take unintended actions. OWASP ranks it as the top risk for LLM applications.
What does an AI security and governance programme cost?
A first programme covering inventory, policy, platform controls and red-teaming of three to five applications typically takes 10-12 weeks, with indicative NTSPL fees of ₹20-45 lakh. Ongoing costs include filter and gateway services, usually small, and periodic red-team testing for high-tier applications.
How do Indian regulations apply to GenAI systems?
The DPDP Act 2023 governs personal data in prompts, retrieval and logs. CERT-In's 2022 directions require incident reporting within six hours and 180-day log retention in India. Regulated entities should also align with sector expectations; RBI's FREE-AI committee report of August 2025 sets out principles for responsible AI in financial services.
Is ISO/IEC 42001 certification necessary?
Not mandatory, but useful when customers, partners or regulators ask for evidence of an AI management system. It integrates well with an existing ISO 27001 programme. Many organisations first implement the controls and governance aligned to 42001 and decide on certification after a year of operation.
Can we allow staff to use public AI tools safely?
Partly. Offer an approved enterprise AI tool with data protection terms, block or warn on unapproved services at the network or browser level, and use DLP to detect sensitive data being pasted. Banning AI without an alternative usually drives use onto personal devices, which is harder to control.
How often should AI applications be red-teamed?
Customer-facing, financial or agentic applications should be tested before launch and at each major release, and at least annually. Internal low-risk tools can rely on automated attack testing in their pipelines. Successful attacks from each exercise become permanent regression tests, so protection improves over time.