kben.sh

Overview

The Koala benchmark suite enables characterization of performance-oriented research targeting the POSIX shell. It consists of 14 real-world program sets from diverse domains ranging from CI/CD and AI/ML to biology and the humanities. They are accompanied by real inputs that facilitate small- and large-scale performance characterization and varying opportunities for optimization.

If any aspect of the suite is useful, please use the following citation:

    
@inproceedings{koala2025atc,
  title = {The Koala Benchmarks for the Shell: Characterization and Implications},
  author = {Evangelos Lamprou and Ethan Williams and Georgios Kaoukis and Zhuoxuan Zhang
        and Michael Greenberg and Konstantinos Kallas and Lukas Lazarek and Nikos Vasilakis},
  booktitle = {Proceedings of the 2025 USENIX Annual Technical Conference (USENIX ATC '25)},
  year = {2025},
  address = {Santa Clara, CA},
  publisher = {USENIX Association},
}

Benchmarks

The complete list of benchmarks in the Koala suite is as follows:

Name	Description
`analytics`	Real-world network logs filtering to extract and summarize key events.
`bio`	Genomic and transcriptomic analysis using population and RNA-seq data.
`ci-cd`	Open-source software project build and test.
`covid`	Analysis of public transit activity during the COVID-19 pandemic.
`file-mod`	File compression, encryption, and conversion.
`inference`	Media-related inference tasks using large foundation models.
`ml`	A full machine learning pipeline using scikit-learn.
`nlp`	Book processing using shell-based NLP pipelines from Unix for Poets.
`oneliners`	Classic and modern one-liner shell pipelines.
`pkg`	AUR package build and npm packages permission inference.
`repl`	Security auditing and git development workflow replay.
`unixfun`	Text-processing problems from UNIX's 50-year anniversary.
`weather`	Analysis & visualization of historical weather statistics.
`web-search`	Crawling, indexing, and querying of Wikipedia data.

Quick Setup

Koala can be obtained using the following ways:

Run curl up.kben.sh | sh from your terminal, or
Clone the repo and run cd koala && ./setup.sh, or
Fetch a Docker container by running docker pull ghcr.io/kbensh/koala:latest, or
Build a Docker container from scratch.

More Info

Recent News

Vagos and Lukas will be presenting this work at ATC'25! Join us!
The suite earned all USENIX ATC Artifact badges!
ATC'25 paper on the first benchmark suite for the shell.

Further Reading

GitHub organization
Short tutorial
Contribution guide

Issues That Need Help

Community & More

GitHub page: @kbensh
Bug reports: koala/issues