Figure provenance in Matplotlib: commit, versions, and parameters in the file

The problem

A referee asks about fig3_final_v2.png a year after you made it, and you do not know which commit drew it or which data went in. Figure provenance is that information, written into the file: the git commit and whether tracked files had changed since, the Python and library versions, the run's parameters, a checksum of each input file, and the time. A PNG or PDF stores named text fields next to the picture, its metadata, and savefig writes them through its metadata= argument. The recipe writes it into the PNG, the PDF, and a JSON file beside them. The example is a smoothed noisy signal from a generated file; swap in your own figure, parameters, and input files.

The code

The example writes its own input file to have one to checksum; the knobs explain the checksum and the git calls. Pillow, the image library Matplotlib installs with itself, reads the record back: Image.open(...).text holds the PNG's text fields.

Copy

import hashlib, json, platform, subprocess
from datetime import datetime, timezone
from pathlib import Path
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from PIL import Image

# ---- parameters and input: replace with your own
params = {"seed": 3, "n_points": 200, "window": 15}
rng = np.random.default_rng(params["seed"])
t = np.linspace(0, 10, params["n_points"])                          # s
y = np.sin(0.8 * t) + 0.4 * rng.standard_normal(t.size)
np.savetxt("signal.csv", np.column_stack([t, y]), fmt="%.6f", delimiter=",", header="t_s,signal")
t, y = np.loadtxt("signal.csv", delimiter=",", unpack=True)

# ---- provenance record
def git(*args):
    try:
        out = subprocess.run(["git", *args], capture_output=True, text=True)
    except FileNotFoundError:                                        # git is not installed
        return None
    return out.stdout.strip() if out.returncode == 0 else None

commit = git("rev-parse", "HEAD")
record = {
    "commit": commit or "not in a git repository",
    "dirty": git("status", "--porcelain", "--untracked-files=no") != "" if commit else None,
    "python": platform.python_version(),
    "versions": {"numpy": np.__version__, "matplotlib": matplotlib.__version__},
    "parameters": params,
    "inputs": {"signal.csv": hashlib.sha256(Path("signal.csv").read_bytes()).hexdigest()},
    "created": datetime.now(timezone.utc).isoformat(timespec="seconds"),
}

# ---- plot
w = params["window"]
avg = np.convolve(y, np.ones(w) / w, mode="valid")
t_avg = t[w // 2 : w // 2 + avg.size]               # "valid" drops w - 1 points: keep the window centers
fig, ax = plt.subplots(figsize=(7, 3.6), dpi=110)
ax.plot(t, y, "o", color="#1f2a44", ms=4, alpha=0.6)
ax.plot(t_avg, avg, color="#c8553d", lw=1.8)
ax.text(4.0, 1.35, f"moving average, window = {w}", color="#c8553d", va="center")
ax.text(6.3, -1.8, "signal", color="#1f2a44", va="center")
ax.set(xlabel="t / s", ylabel="signal / a.u.", ylim=(-2.2, 2.2))
ax.spines[["top", "right"]].set_visible(False)

# ---- save with the record
text = json.dumps(record)
fig.savefig("signal_smoothed.png", dpi=150, metadata={"provenance": text})
fig.savefig("signal_smoothed.pdf", metadata={"Subject": text})
Path("signal_smoothed.json").write_text(json.dumps(record, indent=2))

# ---- read it back from the PNG
stored = json.loads(Image.open("signal_smoothed.png").text["provenance"])
stored.pop("created")                                                # differs on every run
print(json.dumps(stored, indent=2))
plt.show()

{
  "commit": "e8ff8a9374d77ab9013c4163860dc2475d0ceb47",
  "dirty": true,
  "python": "3.12.3",
  "versions": {
    "numpy": "2.4.3",
    "matplotlib": "3.11.2"
  },
  "parameters": {
    "seed": 3,
    "n_points": 200,
    "window": 15
  },
  "inputs": {
    "signal.csv": "ec26891bf4a5d27da139e0ba2015954bd40e5b4a26a5fafa3cc30bec53d80e56"
  }
}

The knobs

git(*args) hands its arguments to subprocess.run, which runs a command as you would type it in a terminal and returns what it printed. git rev-parse HEAD prints the 40-character hash of the commit you have checked out. git status --porcelain prints one line per file that differs from that commit and nothing when none does, so empty output means clean. --untracked-files=no leaves out files git has never been told about, so commit a new script before you trust the flag. Git runs in the current working directory, not in the script's folder: start the script or notebook from inside your project, or the record names another repository or none. The checksum is SHA-256, a 64-character fingerprint of the file's bytes that changes completely when one byte changes; for seed 3 it starts with ec26891bf4a5. The parameters live in one dictionary that the plotting code reads too, so the record cannot drift from the run. Inputs go in by relative name, one entry per file, and nothing in the record is an absolute path or a user name: a figure you send to a journal has no business saying whose disk it came from.

Formats differ in what they accept. A PNG stores any key, which is why the record sits under provenance, next to a Software key Matplotlib writes by itself. A PDF takes only its nine document-info keys, Title, Subject and seven more, and Matplotlib warns "Unknown infodict keyword" for anything else, so the record goes into Subject. An SVG takes Dublin Core keys and raises a ValueError for others; the JSON fits in Description. JPEG and TIFF refuse metadata= altogether. You read the PNG copy with Pillow as the code does, the PDF copy in a viewer's document properties, and the sidecar with json.loads. The versions printed above are the ones in this page's header, and the commit and flag belong to whatever checkout executed it; in a downloaded notebook outside a repository the record says "not in a git repository". The record says what ran, not that a rerun draws the same figure: an unpinned dependency or an unseeded generator changes the numbers under the same commit. The reproducible research topic covers the rest.

Pitfalls

A dirty tree recorded as a clean commit. The commit in the file does not contain the code that drew the figure, because the script was edited after the last commit and only rev-parse HEAD went into the record. Record the flag with the commit, and do not call a figure final while the flag is true. In a notebook, running it and Jupyter's autosave rewrite the tracked .ipynb with new outputs and execution counts, so the flag turns true when no code changed, and it cannot tell that from an edit. For the final figure, commit, then run jupyter nbconvert --to notebook --execute analysis.ipynb. It writes the executed copy to a new, untracked analysis.nbconvert.ipynb and leaves the committed notebook alone, so the flag is false exactly when the code is the commit's. If the PNG and sidecar are committed too, the rerun overwrites them after the record is taken, so the flag in them is still right; the next run finds them changed and records true until you commit them.

Metadata lost on the way. The PNG from the journal's proof, or one cropped in an image editor or converted to JPEG, has no provenance key. Most editors and converters drop text fields they do not know. Keep the sidecar JSON next to the figure under the same stem, signal_smoothed.png and signal_smoothed.json, and commit both. The sidecar is the copy that survives.

Versions from the environment, not the module. The record names a NumPy version that is not the one that ran. importlib.metadata.version("numpy") and pip freeze read what is installed on disk now, and after a pip install --upgrade inside a running Jupyter kernel the imported module can still be the old one until you restart. Take __version__ from the modules the script imported, as the code does with np.__version__ and matplotlib.__version__.