Carving AT Job Files

The post explains why it’s important an incident responder checks scheduled tasks for signs of lateral movement and most importantly, how to carve unnamed scheduled tasks (aka “At jobs”) from a blob of data, such as a physical memory image or page file. The Python script for carving those JOB files is available here: at_jobs_carver.py.

When the attackers move laterally across the network, it’s a common practice to use the built-in utilities to stay under the radar, e.g. “makecab” and “expand” instead of WinRAR (for more examples, see Patrick Olsen’s post). It’s therefore frequent to see the attackers drop their tools on a remote system via a network share and then use the command line utility “at” to schedule tasks that will execute them. Some attackers clean up their tools and others take it even a step further and try to remove the artefacts left in the system – event logs, entries in “schedlgu.txt”, and At#.job files, to name a few. What follows is a short story that illustrates when carving job files from a memory image

When investigating a Windows system recently I discovered some JOB files that confirmed the attackers’ activity but I suspected some of the older JOB files might be missing. Since the system wasn’t shut down for over 100 days I decided to take an image of the physical memory and see if I can dig them up.

I remembered that At1.job file that already existed on the box had a Unicode text (“Created by NetScheduleJobAdd”) that appeared very unique. Luckily for me, searching it across the memory image yielded hits and soon I discovered new commands that were executed on the system. I didn’t even have to carve the job files because the commands were also in Unicode and closely preceded the comment I used for searching.

The only piece of a puzzle that was missing were the timestamps the scheduled jobs were created and executed on. For this, and other metadata, I relied on Gleeda’s JOB file parser. She also very kindly wrote a blog post describing the file’s data structure and referencing the MSDN’s “JOB File Format”. Once I tinkered around, I figured that I needed to grab 0x48 bytes before the command (most of the time it was “cmd.exe”) and an arbitrary amount following it (but enough so the structure doesn’t get corrupted) as the parser will ignore the excess bytes.

After having recovered 5 out of 10 JOB files I found, I started wondering how feasible would it be to automate the carving process, making sure that it works even if the command wasn’t “cmd.exe”. I quickly Googled for carving JOB files but a mention of the manual carving process on the SANS blog was all I could find, and so I decided to give it ago myself. The main obstacle was identifying where does the data structure start and this was overcame by the observation that two sequential fields that are always at the same offset – Error Code and Status, have fixed values. This led me to the idea that we can identify the starting position by finding bytes that match the values of these two fields, precede the comment and are in close proximity.

This resulted in a carver that is available on GitHub and worked well when tested on raw memory images of Windows 2003 SP2 x86 and Windows Server 2008 SP2 x64. I don’t suspect the comment has changed between Windows versions but please do let me know if that’s not the case. Below is an example of running the script:

** Memory image of a clean VM **

bart:~ bart.inglot$ python at_jobs_carver.py before.raw ~/jobs

[*] Created output folder: /Users/bart.inglot/carved_jobs

[*] Searching...

[*] Done

** Memory image of a VM after creating an AT job **

bart:~ bart.inglot$ python at_jobs_carver.py after.raw ~/jobs

[*] Searching...

[+] Found hit: 0x3a95d000

[-] Failed verification

[+] Found hit: 0x7a5ad518

[*] Done

bart:~ bart.inglot$ cat ~/jobs/carved_1.job

I�GW� M����

 s�!�

       !cmd.exeZ/c "\\WIN-TESTBOX\c$\Windows\System32\netstat.exe > \\WIN-TESTBOX\c$\netstat.txt"SYSTEMCreated by NetScheduleJobAdd0�

As mentioned earlier, the carving process relies on finding the comment that is unique to JOB files created by the “at” utility, and therefore it won’t find JOB files created by other means (e.g. GUI or “schtasks.exe”).

Have fun and find moar evil!