Skip to content

The Data-Science Layout

Projects generated with the data-science layer are two things at once: a Python package with a tested, linted src/ tree, and a research workspace with a place for every artifact analysis produces. This guide walks the workspace — where data lives, how notebooks and SQL fit, how sharing works, and how the pieces reach the Quarto paper. The Kaggle flavour of online_judge shares the machinery (its src/ tree is configs/ data/ input/ output/ features/ logs/ models/ notebook/ scripts/ utils/ instead); everything below applies to it with the paths adjusted.

The two trees

data/            the research workspace (git-ignored by default)
  raw/           untouched inputs, exactly as obtained
  interim/       intermediate transformations
  processed/     the canonical, derived datasets
  external/      third-party data (never regenerated by your code)
  queries/       SQL, loaded from Python with DuckDB (see below)
  sharing/       the outbound gate: TRANSFER_LOG.csv
  DEIDENTIFICATION.md   the de-identification standard this repo commits to
src/             the installable package (src/<name>/) beside the
                 analysis dirs the questionnaire creates as stubs:
                 data/ features/ models/ visualization/ (.gitkeep each;
                 rename or extend via the structured `src_dirs` answer)
notebooks/       exploratory notebooks (marimo / Jupyter)
outputs/         tables and figures the paper reads in
reports/         rendered reports and figures/
paper/, slides/  the Quarto scaffold (see the how-to)

The direction of flow is the convention that keeps this honest: analysis code writes; data/ and outputs/ are written-to, never hand-edited; the paper reads outputs/. Data, models and reports are git-ignored — the repo carries the code that regenerates them, not the artifacts. The analysis dirs start as .gitkeep stubs: the code that fills them is yours to write, tested like any other module in the package.

The SQL workspace

data/queries/ holds SQL as files, loaded from Python with DuckDB (duckdb is a base dependency, alongside polars and pyarrow):

import duckdb

con = duckdb.connect("data/pipeline.duckdb")
con.execute(open("data/queries/example.sql").read())

The shipped README.md in that directory states the boundary this layout wants: transformations that produce derived tables belong in data/processed/ (with the code that runs them in src/data/); exploratory queries live in data/queries/. Keep the two apart — a query that graduated from exploration to pipeline is code, and code gets tests.

Notebooks: marimo or Jupyter

Two tasks launch the two editors; both are already among the dev dependencies:

task marimo      # marimo edit notebooks/
task notebook    # jupyter lab notebooks/ --allow-root

Marimo notebooks are Python files — they version like code, run top to bottom with no hidden state, and the generated lint/type-check applies to them like any other source. Keep scratch work in notebooks/; the moment a cell is worth keeping, promote it into src/ (with a test) and call it from the notebook.

Sharing data out: the de-identification gate

data/sharing/ exists because "send someone the extract" is a decision, not a file copy. data/DEIDENTIFICATION.md is a one-page standard — techniques per ISO/IEC 20889 and NIST IR 8053, the anonymisation bar per GDPR Recital 26 — whose four steps are, in order:

  1. Suppress direct identifiers.
  2. Generalise quasi-identifiers and pseudonymise residual stable identifiers with HMAC-SHA256(secret_salt, id) — the salt never enters the repository, mapping tables are held separately.
  3. Guardrail: every quasi-identifier combination occurs at least k ≥ 5 times; suppress smaller cells.
  4. Review with a motivated-intruder pass by someone who did not build the extract, then record the transfer in sharing/TRANSFER_LOG.csv with the SHA-256 fingerprint of the shipped files.

The page ends with the sentence that matters: pseudonymised data is still personal data — below the anonymisation bar, the extract moves only under a signed data transfer agreement. Two more sheets can join data/ when the questionnaire's answers select them: DUO.md (data-use limitations, on data_reusable) and CARE.md (collective-benefit and governance, on data_ethics or the medtech domain trait).

The paper connection

outputs/ is the seam between this layout and the manuscript: analysis code writes tables and figures there, and the Quarto paper includes them at render time — numbers are never pasted. The full loop is documented in the Quarto paper how-to; the short version:

$ task marimo                # explore, build the table
$ # promote the cell into a tested module; that module writes outputs/
$ task paper                 # the paper reads outputs/ and renders

Where a web API layer coexists (include_web_api on a data-science base), app/ holds request/response handling and src/ keeps the analysis — routers expose results computed by src/ code, and heavy processing never moves into app/. The generated README's data-science section spells out this split.

GPU and the experiment extras

Dockerfile.gpu builds on a CUDA base image (the newest torch-bearing index, checked weekly by the upstream-drift tool) and the experiment extra carries the heavier stack (torch, marimo, …) the CPU path does not need. Use it when training on a GPU box; the CPU toolchain stays the default everywhere else.