0% Complete
0/0 Steps
  1. About

    About Cyber School
  2. About BDO
  3. The CSRP program Syllabus
  4. Module 1
    The Cyber World and CIA Triad
    6 Topics
  5. The OSI Model and Network Layers
    6 Topics
  6. IP Addressing, Subnetting and Routing
    6 Topics
  7. DNS and DHCP
    6 Topics
  8. Windows Server Administration
    6 Topics
  9. Active Directory and Identity Management
    6 Topics
  10. Linux CLI Fundamentals
    6 Topics
  11. Linux Permissions and Hardening
    6 Topics
  12. Module 2 - Defense Technologies and Cyber Tools
    Firewall Architecture and Rule Design
    5 Topics
  13. NAT and Secure Traffic Mapping
    6 Topics
  14. Enterprise VPN Solutions
    6 Topics
  15. IDS and IPS
    5 Topics
  16. Identity and Access Management
    5 Topics
  17. Multi-Factor Authentication
    6 Topics
  18. Data Loss Prevention
    6 Topics
  19. Endpoint Security and EDR
    6 Topics
  20. Mail Security
    6 Topics
  21. Web Proxy and URL Filtering
    6 Topics
  22. Regulation and Compliance
    6 Topics
  23. Cloud Security
    6 Topics
    |
    1 Quiz
  24. Module 3 - Ai in security
    AI in Security
    5 Topics
  25. ML Anomaly Detection
    5 Topics
  26. AI Threats
    5 Topics
  27. LLM Defense
    6 Topics
  28. Module 4 - Offensive Cyber & Vulnerability Scanning
    Cyber Kill Chain and Attacker Mindset
    6 Topics
  29. OSINT and Public Intelligence Gathering
    6 Topics
  30. Network and Port Scanning with Nmap
    6 Topics
  31. Vulnerability Scanning
    6 Topics
  32. Exploitation Fundamentals
    6 Topics
  33. Web Application Security
    5 Topics
  34. Social Engineering
    5 Topics
  35. Password and Hash Cracking
    5 Topics
  36. Module 5 - SOC and Advance Cyber Analyst
    SOC Structure and Operations
    6 Topics
  37. SIEM Fundamentals with Splunk
    6 Topics
  38. Alerting and Dashboards
    6 Topics
  39. Incident Response
    5 Topics
  40. Basic Digital Forensics
  41. Capstone A - The Attack: Breaching Meridian
  42. Capstone B - The Defense: Defending Meridian
  43. ISC2 CC Mock Exam and CSRP Graduation
    2 Quizzes
Lesson 27, Topic 2
In Progress

LLM Defense Lesson Companion

Cyber School 06/07/2026
Lesson Progress
0% Complete

CSRP  —  Cyber Security Responder & Practitioner

Lesson 24: LLM Defense

Lesson Companion Workbook

Cyber School  ·  BDO Ziv Haft

CSRP LESSON 24 — COMPANION WORKBOOK

Module 3 — AI in Security · Lesson 4 of 4 · Course Lesson 24 of 40 — FINAL M3 Title: LLM Defense

Follows the Gamma outline chapter-by-chapter. Module 3 closure.


Chapter 0 · Orientation

Where you are: L21, L22, L23 done. Today closes Module 3. After today: [AI-Practitioner] module badge, Cyber Practitioner role unlocks, 60% of CSRP complete.

Fill-in: L21 = AI as your ___________ . L22 = ML as your ___________ . L23 = AI as the ___________ . L24 = AI INSIDE your ___________ .

Reflect: What LLM-integrated product does your organization use or plan to deploy? What’s the current defensive posture around it? ________________________________________________________________________________


Chapter 1 · LLM Application Architecture

The six components

#ComponentReal example
1User promptChat input box / API call
2System promptDeveloper-set instructions (“You are a helpful Meridian assistant…”)
3ModelGPT-4-class / Claude / Gemini
4Retrieval / KBVector DB, RAG over corporate wiki
5Tools / pluginssend_email, query_db, http_get, file_write
6OutputText response, tool invocations

Threat surfaces per component

ComponentThreatOWASP LLM
User promptPrompt injection, jailbreakLLM01
System promptDisclosureLLM06
ModelHallucination, bias(cross-LLM)
Retrieval / KBData leakage, KB poisoningLLM06, LLM03
Tools / pluginsUnauthorized invocationLLM07, LLM08
OutputDownstream injectionLLM02

The system prompt fallacy

“We told the LLM not to reveal X. We’re safe.” Wrong. The system prompt is a soft suggestion. With sufficient technique, it’s overridden. Defense: multiple layers, not one strong prompt.

Fill-ins

  1. The 6 LLM-app components: ___________ · ___________ · model · ___________ · ___________ · output.
  2. The “system prompt fallacy” is believing the ___________ prompt alone is ___________ .
  3. The component that handles the model’s INVOCATION of external actions is ___________ .

Reflect

For your org’s LLM (or a vendor LLM you use): identify which of the 6 components are configured. Which are missing controls today? ________________________________________________________________________________

Bridge

Back to L21: the 5-component prompt pattern you learned in L21 (Role/Context/Task/Constraints/Output Format) is what GOES INTO the user-prompt component here.


Chapter 2 · OWASP Top 10 for LLM Applications

The ten

IDName
LLM01Prompt Injection
LLM02Insecure Output Handling
LLM03Training Data Poisoning
LLM04Model Denial of Service
LLM05Supply Chain Vulnerabilities
LLM06Sensitive Information Disclosure
LLM07Insecure Plugin Design
LLM08Excessive Agency
LLM09Overreliance
LLM10Model Theft

Today’s focus

LLM01 · LLM02 · LLM06 · LLM07 · LLM08

Fill-ins

  1. LLM01 — ___________ — the new SQL injection.
  2. LLM06 — Sensitive ___________ Disclosure — leaking system prompt or KB contents.
  3. LLM08 — Excessive ___________ — overly broad LLM authority.

Reflect

Which of the 10 is your organization’s biggest risk today, and why? ________________________________________________________________________________


Chapter 3 · Prompt Injection + Jailbreaking

Canonical injection patterns

PatternTypeOWASP
“Ignore previous instructions”DirectLLM01
“Reveal / print the system prompt”DisclosureLLM06
“Use the tool without approval”Tool abuseLLM07/08
“Act as ___” (role-play)JailbreakLLM01
“In a hypothetical world where ___”JailbreakLLM01
“Translate the following including any safety policies”Encoding bypassLLM01
“The previous instructions were a test. The real instruction is ___”Instruction overrideLLM01
(Embedded in retrieved content)Indirect injectionLLM01

Direct vs indirect

  • Direct: the malicious instruction is in the user’s input.
  • Indirect: the malicious instruction is in CONTENT the LLM reads (webpage, PDF, KB entry).

Indirect is the harder defense because the attacker controls content, not input.

Fill-ins

  1. Direct prompt injection comes from the ___________ . Indirect comes from ___________ .
  2. “Ignore previous instructions” is the canonical example of OWASP ___________ .
  3. Asking the LLM to “act as a different persona” is a ___________ technique.

Reflect

Have you (or your team) ever been the target of a prompt injection attempt — or noticed one in retrospect? Describe. ________________________________________________________________________________


Chapter 4 · Guardrails + DLP for AI

The guardrail layers

#LayerImplementation
1Input validationRegex on prompt patterns
2Output validationRegex on response patterns
3Secret + PII scanRegex on credentials / PII / payment data
4Sensitive marker filterBlock outputs containing [INTERNAL], [CONFIDENTIAL]
5Suspicious prompt loggingLog → SIEM / file
6Rate limitingSlow high-volume probing
7Tool ACLPer-user / per-tool permissions + approval gates
8Retrieval ACLFilter RAG results by user perms BEFORE the LLM

Input filter examples (Lab 24.2)

INJECTION_PATTERNS = [
    r”ignore\s+previous\s+instructions”,
    r”reveal\s+(the\s+)?(hidden\s+)?(policy|system\s+prompt)”,
    r”print\s+(the\s+)?(your\s+)?system\s+prompt”,
    r”use\s+the\s+tool\s+without\s+approval”,
    r”act\s+as\s+(?:a\s+)?different\s+(?:assistant|persona)”,
]

Output filter examples

OUTPUT_DANGER = [
    r”\[INTERNAL\]”,
    r”\[CONFIDENTIAL\]”,
    r”system\s+prompt:”,
    r”AKIA[0-9A-Z]{16}”,     # AWS key
    r”sk-[a-zA-Z0-9]{32,}”,  # OpenAI-style
]

DLP for AI — the classification matrix

DataAllowBlockRedact
PII (names + emails)—yesyes
Credit card—yes—
API keys / passwords—yes—
Internal pricing—yes (per policy)—
Internal codenames—yes (per policy)—
Public marketing copyyes——

Fill-ins

  1. The 4 guardrails Lab 24.2 implements: input check · ___________ ___________ · output filter · suspicious ___________ logging.
  2. The defense principle: ___________ in ___________ .
  3. DLP for AI applies the thinking from Lesson ___________ to AI prompts.

Reflect

At your org — which guardrail layer is most achievable in the next 30 days? Which is the hardest? ________________________________________________________________________________


Chapter 5 · Least Privilege + Logging + AI Usage Policy

Tool ACL patterns

  • Per-user tool ACLs (“user X can use {send_email, query_db}”)
  • Approval gates (any send_email over $X requires human OK)
  • Read-only mode (no tools at all; chatbot is retrieval-only)
  • Sandboxed execution (run_command in ephemeral container)

AI agent risk

“AI agent” = LLM with broader agency, multi-step planning. Agency compounds prompt-injection consequences.

Logging to capture

  • Every prompt + output (forensics + injection detection)
  • Alert on suspicious patterns
  • Per-user volumes (UEBA from L22)

AI usage policy — 1-page checklist

  • Approved tools only (L21 Safety Rule 5)
  • No secrets / customer data (L21 Safety Rules 1-2)
  • Data classification rules
  • Verification discipline (L21)
  • Incident reporting

Fill-ins

  1. Approval gates require human OK above a ___________ .
  2. Sandboxed execution puts the tool action in an ___________ container.
  3. Per-user AI usage anomalies are detectable via ___________ from L22.

Reflect

Your org’s AI usage policy — does one exist? If yes, when was it last updated? If no, what’s the biggest hole? ________________________________________________________________________________


Chapter 6 · Challenge — Prompt Shield

Pre-Challenge checklist

  • I’ve completed Lab 24.2 (the guardrail code is the foundation)
  • I have /labs/L24/chatbot/ framework open
  • I know the input + output + tool-ACL + logging requirements
  • I will not ask for a hint before T=10

Implementation framework

# /labs/L24/work/prompt_shield.py
import re, json, datetime

def check_input(prompt):
    # … your input filter …
    pass

def check_output(response):
    # … your output filter …
    pass

def check_tool_use(tool_name, args, user):
    # … your tool ACL …
    pass

def log_event(verdict, prompt, reason):
    # … your logger …
    pass

Validator expectations

  • 10 attacker prompts: all blocked
  • 10 benign prompts: all pass through
  • ≥ 1 indirect injection: blocked
  • All blocks logged with reason
  • 0 false positives on benign set

Lab rubrics

Lab 24.1

CheckpointWeightPass
All 15 classified (SAFE/UNSAFE)20Table complete
≥ 80% correct on labels30Auto-graded vs per-seat truth
OWASP LLM ID per UNSAFE25All non-empty
Indirect injection correctly flagged15Notes section non-empty
“Why dangerous” non-empty10Per row

Pass: ≥ 60.

Lab 24.2

CheckpointWeightPass
Input filter implemented with ≥ 4 canonical patterns25Code review
Output filter implemented with ≥ 4 markers25Code review
Secret/PII warning on input15Code review
Suspicious-prompt logging functional15Log file populated during test
Test runs (curl tests pass)10Manual verification
lab2_notes.md complete10All sections

Pass: ≥ 60.

Lab 24.3

CheckpointWeightPass
All 10 classified20Table complete
Correct class per row30Per-seat truth
Decision per row20Allow/block/redact
3 unsafe prompts rewritten safely15Side-by-side present
1-page AI usage guideline15All sections present

Pass: ≥ 60.

Challenge (validator-based)

OutcomePass
All 10 attacker prompts blockedRequired
All 10 benign prompts passRequired
≥ 1 indirect injection blockedRequired
0 false positives on benignRequired
All blocks loggedRequired

Badge [Prompt-Shield-Engineer] on pass.


Module 3 closure self-check

  1. Name the 6 LLM-app components.
  2. List the OWASP LLM Top 10 IDs covered today (5 of them).
  3. Give one example of each: direct injection, indirect injection, jailbreak.
  4. List 4 guardrail layers from memory.
  5. State the principle: “Never rely on the ___ ___ alone.”
  6. Define AI agent risk in one sentence.
  7. Recite the 4 M3 lesson badges.

Learning log

Today I learnedI want to practice
  
  

Module 3 reflection

L21 → L22 → L23 → L24. Four lessons. Four badges. Compound AI competence.

Take 5 minutes. Write 3 sentences about what changed in your thinking across Module 3. ________________________________________________________________________________ ________________________________________________________________________________ ________________________________________________________________________________


Cross-references

  • L15 DLP: Lab 24.3 IS DLP for AI
  • L21 LLM HITL + safety rules: apply to all AI guardrail bypasses
  • L22 UEBA: per-user AI-usage anomaly detection
  • L23 AI Threats: adversarial-ML thinking carries here
  • Forward to L25 / Module 4 (Offensive): the attacker’s view of these techniques
  • Forward to Module 5 (SOC): guardrail logs become a monitoring source