How antivirus and EDR detect you on Windows: AMSI, ETW, hooks, AppLocker and CLM explained
What Defender or an EDR checks when you run a tool (signatures, AMSI, ETW, hooks, kernel callbacks), how to tell what protects a box, and AppLocker and CLM.
You upload winPEAS and it vanishes. You run an IEX and PowerShell answers "This script contains malicious content". Your reverse shell dies after ten seconds. Those are signs of an antivirus or EDR, and in the catalogue they show up on machines like Acute, Sizzle or Giddy (antivirus and EDR evasion). This guide is not an evasion kit: it explains how detection works, because understanding it is what lets you choose well, whether you attack or defend. The evasion cheat sheet goes deeper into each layer.
The layers of detection
A modern product does not have a single control but several overlapping ones. What "catches" you depends on the layer:
| Layer | When it acts | What it looks at | How you notice |
|---|---|---|---|
| Static signatures | When the file is written to disk | Byte sequences, strings, known hashes | The file disappears right after upload |
| AMSI | Before a script runs (PowerShell, VBScript, JScript, Office macros, in-memory .NET) | The content of the script, already decoded | A "malicious content" error even without touching disk |
| ETW | While it runs | Events: .NET assembly loads, process creation, network calls | You are detected later, not immediately |
| User-mode hooks | On every API call (in ntdll.dll) |
Suspicious calls: opening LSASS, writing to another process's memory | The process closes when it does something specific |
| Kernel callbacks and minifilters | When processes and threads start, images load, files and registry change | Everything, from the kernel | Cannot be "patched" from user mode |
| Behaviour and cloud | Combining all of the above | Chains of actions: Word starts PowerShell, which connects out | Alerts even when each step looks normal |
Microsoft Defender covers the first four layers and part of the behavioural one. An EDR (Defender for Endpoint, CrowdStrike, SentinelOne, Elastic) adds full telemetry and, above all, people watching the alerts.
What protects this machine
Before changing anything, find out:
Get-MpComputerStatus | Select AMServiceEnabled,RealTimeProtectionEnabled,IsTamperProtected,AntivirusSignatureLastUpdated
sc query windefend
Get-Process | Where-Object { $_.ProcessName -match 'MsMpEng|MsSense|CSFalcon|SentinelAgent|elastic|cb|cylance|sophos' }
# Exclusions (you must be an administrator to read them)
Get-MpPreference | Select ExclusionPath,ExclusionProcess,ExclusionExtension
# LSASS protection
reg query HKLM\SYSTEM\CurrentControlSet\Control\Lsa /v RunAsPPL
Note
A folder or process exclusion is, in practice, an area the antivirus does not look at. Admins add them so their software works. It is the first thing checked in an assessment, and the first thing a defender should review.
AppLocker, WDAC and Constrained Language Mode
These are not antivirus: they are allow lists. They decide which executables, scripts and DLLs may run, usually by path, publisher or hash (AppLocker and CLM).
Get-AppLockerPolicy -Effective | Select -ExpandProperty RuleCollections
$ExecutionContext.SessionState.LanguageMode # FullLanguage or ConstrainedLanguage
When AppLocker or WDAC enforce script rules, PowerShell drops into Constrained Language Mode (CLM): no arbitrary .NET types, no Add-Type, and most offensive PowerShell tools (PowerView, PowerUp) stop working. That is why things that worked on another box fail.
What it means in a CTF:
- Check which paths the policy allows. The default rules allow running from
C:\Windows\andC:\Program Files\, and inside those folders there are subfolders users can write to (such asC:\Windows\TasksorC:\Windows\Tempon some versions). - Run tools from your box (Impacket, NetExec, bloodhound-python) instead of on the machine: AppLocker never sees them.
- Signed Microsoft binaries that execute code (the so-called LOLBAS, such as
MSBuild.exeorInstallUtil.exe) are the classic gap in badly written policies. Good policies block them explicitly.
On Sizzle, for example, there is AppLocker and CLM, and part of the challenge is working within those restrictions.
Your options in a CTF
On a practice machine the goal is to learn the technique, not to win a race against the antivirus. In order, what usually works:
- Do less on the machine. Almost everything in AD can be done from your box with credentials: enumeration, Kerberoasting, ADCS, remote dumping.
- Use what is already there. System commands (
net,reg,sc,schtasks,wmic) instead of uploaded binaries. - Compile the tools yourself. A binary you build from source does not match the public binary's signatures. It is also safer: you know what you are running.
- Pick lesser-known variants of the same technique (the Potatoes and lateral movement guides list several for each case).
Warning
Turning Defender off (Set-MpPreference -DisableRealtimeMonitoring $true) works if you are an administrator and there is no Tamper Protection, but on a real engagement it is the loudest thing you can do: it raises an immediate alert. On an authorised job, ask first and write it in the report.
For defenders
What helps most against all of the above, and what is usually missing on the catalogue's machines:
- PowerShell logging (Script Block Logging, event 4104) and Sysmon: you see the decoded script even if it arrives obfuscated.
- LSASS as PPL and Credential Guard: LSASS dumping stops working.
- WDAC in enforced mode, with Microsoft's recommended LOLBAS block rules.
- Defender ASR rules: stop Office from spawning child processes, or anything from stealing credentials from LSASS.
- Regularly reviewing exclusions and who can change them.
What next
If what blocks you is the escalation itself, go back to services and DLLs or Potatoes with a different variant. For command and control (C2) and how it gets detected, see the C2 and post-exploitation cheat sheet, and the AD red teaming path to practise everything in order.