Quality Assurance has always been about understanding risk and validating systems before they reach production. After more than eight years in QA and now working in AI security, governance, and red teaming, I often compare traditional testing practices with the challenges introduced by AI systems.
While the risks have evolved from software defects to hallucinations, prompt injections, and data leakage, the goal remains the same: building confidence that systems behave safely, reliably, and as intended.
That is why I found IMDA’s Starter Kit for Testing LLM-Based Applications for Safety and Reliability particularly valuable. It approaches AI testing not as a one-time activity, but as a continuous process of identifying risks, validating controls, and building trust in AI systems.
What I Appreciated Most
One of the strongest aspects of the Starter Kit is its simplicity.
The framework narrows AI testing into five critical risk areas:
- Hallucination and Inaccuracy
- Bias in Decision Making
- Undesirable Content
- Data Leakage
- Vulnerability to Adversarial Prompts
One of the reasons these categories resonated with me is their relevance to many of the risks organizations are actively managing today. What also stood out to me was the strong alignment between these risk areas and the OWASP Top 10 for LLM Applications.
Risks such as prompt injection, sensitive information disclosure, and unreliable outputs are common findings during AI security assessments and red teaming exercises, making the framework practical for both AI assurance and security testing.
Real time example: When evaluating an AI application, particularly chatbots connected to internal knowledge sources, the biggest concerns are rarely model accuracy alone. Common findings include:
- Prompt injection vulnerabilities that bypass safety instructions
- Sensitive information exposure through retrieval pipelines
- Hallucinated responses presented with high confidence
- Unsafe outputs generated through multi-turn interactions
- Weak system prompts that can be manipulated by attackers
What impressed me was how effectively the framework captures these risks in a structured and actionable way. The categories are broad enough to apply across different use cases while remaining practical for implementation teams.
Combined with targeted testing examples, the Starter Kit provides a strong foundation for organizations looking to build or mature their AI assurance programs.
Output Testing vs. Component Testing: A Lesson Every Team Should Learn
One of the most valuable lessons in the Starter Kit is the distinction between output testing and component testing.
The guide emphasizes that testing should extend beyond model responses to include internal components such as system prompts, filters, retrieval systems (RAG), knowledge bases, and the underlying model.
This closely mirrors what we observe during application red teaming engagements.
A chatbot may appear safe during normal interactions while still containing weaknesses within its retrieval layer, prompt orchestration logic, or guardrails. In many assessments, vulnerabilities are discovered not because the final response appears dangerous, but because internal pathways can be manipulated to eventually produce unsafe outcomes.
In practice, some of the most critical findings originate from prompt templates that expose hidden instructions, retrieval mechanisms that return unauthorized information, weak content filtering controls, or insecure orchestration between models and tools. These issues often remain invisible during traditional output validation.
The key takeaway is simple:
1.Evaluating model responses alone rarely provides a complete picture of an application’s security posture.
Some of the most critical findings originate from:
- Prompt templates that expose hidden instructions
- Retrieval mechanisms that return unauthorized information
- Weak content filtering controls
- Insecure orchestration logic between models and tools
2.Component-level testing is often where organizations uncover the issues that traditional output validation misses. The Starter Kit’s emphasis on this distinction is one of its most practical recommendations.
Red Teaming Beyond Prompt Injection
One of the biggest misconceptions in the industry is that AI red teaming is simply about jailbreaks and prompt injections.
The Starter Kit takes a much broader view, describing red teaming as a method for uncovering blind spots, testing multi-turn interactions, and identifying subjective harms that benchmark testing may miss.
This aligns closely with real-world experience. Many impactful vulnerabilities do not emerge through a single adversarial prompt. Instead, they surface through:
- Context accumulation across conversations
- Role-playing attacks
- Retrieval manipulation
- Indirect prompt injection
- Goal hijacking through tool usage
- Multi-step attack chains
The framework reinforces an important reality: benchmark testing helps measure known risks, while red teaming helps uncover unknown risks. Both are necessary for a mature AI assurance program.
Static benchmarks can provide useful coverage and repeatability, but they cannot fully capture the unpredictable ways users, attackers, and complex business environments interact with AI systems. Human-led red teaming remains essential for uncovering those hidden failure modes.
What Could Be Added to the Starter Kit
The Starter Kit provides a strong foundation for today’s LLM applications, but the next generation of AI systems will require testing methodologies that evolve alongside them.
1. Stronger Coverage for Agentic AI
The industry is rapidly moving beyond chatbots toward autonomous agents capable of invoking tools, accessing external systems, maintaining memory, and taking actions on behalf of users.
Agentic systems introduce entirely new risk categories, including:
- Unauthorized tool execution
- Privilege escalation
- Excessive autonomy
- Unsafe task chaining
- Memory poisoning
- Tool output manipulation
- Agent-to-agent communication
- Human approval bypasses
Traditional prompt-based testing is often insufficient for evaluating these systems because the risk no longer resides solely in what the model says—it also resides in what the model can do.
An agent that can interact with ticketing systems, databases, cloud resources, or financial applications creates an entirely different attack surface. Future testing frameworks will need methodologies that evaluate decision-making processes, action execution, permission boundaries, and multi-agent interactions.
2. Expanded Focus on Continuous Governance
The document primarily focuses on pre-deployment testing.
In enterprise environments, however, risks frequently emerge after deployment when prompts, models, data sources, retrieval systems, or business workflows change.
AI assurance should be treated as a continuous lifecycle rather than a deployment checkpoint. In practice, many organizations face greater risk from post-deployment changes than from the initial model release itself.
Future versions could place greater emphasis on:
- Continuous monitoring
- AI asset inventory management
- Model and prompt change tracking
- Governance controls
- Periodic reassessments
- Risk trend analysis
This would help organizations maintain assurance as systems evolve over time.
3. Stronger Alignment with Emerging Regulations
The Starter Kit references established international frameworks, including NIST and ISO.
Future versions could go further by mapping testing activities directly to emerging governance and regulatory frameworks such as:
- OECD AI Principles
- UNESCO Recommendation on the Ethics of AI
- EU AI Act
- Other emerging national AI regulations
Such mappings would help organizations connect technical testing outcomes with broader governance, compliance, and risk management obligations.
My Key Takeaway
AI assurance is not about finding vulnerabilities—it is about building trust.
The framework follows a simple but powerful cycle:
Identify → Test → Assess → Mitigate → Re-test
This mirrors how mature AI security programs operate in practice.
The real value of testing is not simply identifying risks, but implementing mitigations, validating their effectiveness, and demonstrating measurable risk reduction over time. As AI adoption grows, organizations increasingly need evidence that their security, safety, and governance controls remain effective as systems evolve.
Ultimately, effective AI assurance is not measured by the number of vulnerabilities discovered, but by an organization’s ability to continuously reduce risk as AI systems evolve.
The IMDA Starter Kit successfully bridges the gap between theory and practice by providing a structured approach that startups, enterprises, security teams, testers, and governance professionals can realistically adopt.
As someone who spends every day evaluating AI systems through red teaming, governance, and security assessments, the Starter Kit felt less like a theoretical framework and more like a practical reflection of what effective AI assurance should look like in the real world.
The challenge is no longer whether AI systems should be tested, but whether our testing, governance, and assurance practices can evolve quickly enough to keep pace with increasingly autonomous and business-critical AI systems. IMDA’s Starter Kit provides an important foundation for that journey.
