Overview
The Koala benchmark suite enables characterization of performance-oriented research targeting the POSIX shell. It consists of 18 real-world program sets from diverse domains ranging from CI/CD and AI/ML to biology and the humanities. They are accompanied by real inputs that facilitate small- and large-scale performance characterization and varying opportunities for optimization.
Projects Using Koala
- SysXCHG: Refining Privilege with Adaptive System Call Filters — CCS 2023
- Fractal: Fault-Tolerant Shell-Script Distribution — NSDI '26
- Rt: Regular Types for the Streaming Shell — OSDI '26
- Incr: Faster Re-execution via Bolt-on Incrementalization — OSDI '26
- hS: Speculative Script Reordering at Subprocess Granularity — OSDI '26
- Controlling Opaque-Component Effects with Semisolates and Try — OSDI '26
- Ahead-of-time Analysis of Shell Program Effects — SOSP '26
If any aspect of the suite is useful, please use the following citation:
@inproceedings{koala2025atc,
title = {The Koala Benchmarks for the Shell: Characterization and Implications},
author = {Evangelos Lamprou and Ethan Williams and Georgios Kaoukis and Zhuoxuan Zhang
and Michael Greenberg and Konstantinos Kallas and Lukas Lazarek and Nikos Vasilakis},
booktitle = {Proceedings of the 2025 USENIX Annual Technical Conference (USENIX ATC '25)},
year = {2025},
address = {Santa Clara, CA},
publisher = {USENIX Association},
}Benchmarks
The complete list of benchmarks in the Koala suite is as follows:
| Name | Description |
|---|---|
analytics |
Real-world network logs filtering to extract and summarize key events. |
bio |
Genomic and transcriptomic analysis using population and RNA-seq data. |
ci-cd |
Open-source software project build and test. |
covid |
Analysis of public transit activity during the COVID-19 pandemic. |
etcetera |
Prime generation via a sieve and command execution inside an overlay sandbox. |
file-mod |
File compression, encryption, and conversion. |
inference |
Media-related inference tasks using large foundation models. |
interact |
Interactive shell programs (oh-my-zsh installer, snake game). |
ml |
A full machine learning pipeline using scikit-learn. |
net |
Network reconnaissance and firewall configuration tasks. |
nlp |
Book processing using shell-based NLP pipelines from Unix for Poets. |
oneliners |
Classic and modern one-liner shell pipelines. |
pkg |
AUR package build and npm packages permission inference. |
rand |
Random password generation and random team sampling from name lists. |
repl |
Security auditing and git development workflow replay. |
unixfun |
Text-processing problems from UNIX's 50-year anniversary. |
weather |
Analysis & visualization of historical weather statistics. |
web-search |
Crawling, indexing, and querying of Wikipedia data. |
Quick Setup
Koala can be obtained using the following ways:
- Run
curl up.kben.sh | shfrom your terminal, or - Clone the repo and run
cd koala && ./setup.sh, or - Fetch a Docker container by running
docker pull ghcr.io/kbensh/koala:latest, or - Build a Docker container from scratch.
More information in the README.
More Info
Recent News
- Koala's latest version expands the suite with four new benchmark sets!
- Koala received a Best Paper Award at ATC'25!
- Vagos and Lukas will be presenting this work at ATC'25! Join us!
- The suite earned all USENIX ATC Artifact badges!
- ATC'25 paper on the first benchmark suite for the shell.
Further Reading
- GitHub organization
- Short tutorial
- Contribution guide
Issues That Need Help
Community & More
- GitHub page: @kbensh
- Bug reports: koala/issues