Skip to content

Commit de79bd1

Browse files
agrouazeagrouaze
committed
lint+README improvment
1 parent aea9ab6 commit de79bd1

2 files changed

Lines changed: 184 additions & 24 deletions

File tree

‎README.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,132 @@
1313

1414
<!-- SPHINX-START -->
1515

16+
## Purpose
17+
18+
**turboblast** is a Python library for submitting high-throughput job arrays to
19+
a [Slurm](https://slurm.schedmd.com/) cluster using
20+
[submitit](https://github.com/facebookincubator/submitit). It is designed for
21+
workflows where you have a large list of command-line tasks (e.g. processing
22+
satellite files) that need to be distributed across many compute nodes in
23+
parallel.
24+
25+
The core idea is simple: you provide a text file where each line is a set of
26+
arguments, and turboblast dispatches each line as an independent Slurm task
27+
running a bash script of your choice. Large input lists are automatically split
28+
into chunks of 1000 to stay within Slurm array limits.
29+
30+
### Dependencies
31+
32+
| Package | Role |
33+
| --------------------------------------------------------- | ------------------------------------------------- |
34+
| [submitit](https://github.com/facebookincubator/submitit) | Submits and monitors Slurm job arrays from Python |
35+
| Python ≥ 3.10 | Required runtime |
36+
37+
## Installation
38+
39+
```bash
40+
pip install turboblast
41+
```
42+
43+
Or with conda:
44+
45+
```bash
46+
conda install -c conda-forge turboblast
47+
```
48+
49+
## Usage
50+
51+
### Prepare your inputs
52+
53+
Create a plain text file where each line contains the arguments for one task:
54+
55+
```
56+
# inputs.txt
57+
--input /data/file_001.nc --output /results/
58+
--input /data/file_002.nc --output /results/
59+
--input /data/file_003.nc --output /results/
60+
```
61+
62+
### Write your bash script
63+
64+
turboblast will call `bash your_script.sh <args>` for each line. Example:
65+
66+
```bash
67+
#!/bin/bash
68+
# process.sh
69+
python my_processor.py "$@"
70+
```
71+
72+
### Submit the job array
73+
74+
```bash
75+
turboblaster \
76+
--listing-input inputs.txt \
77+
--bash-slurm-exec process.sh \
78+
--slurm-partition gpu \
79+
--timeout-min 60 \
80+
--mem-gb 8 \
81+
--cpus-per-task 4 \
82+
--slurm-array-parallelism 50 \
83+
--output-dir submitit_logs
84+
```
85+
86+
### Full CLI reference
87+
88+
```
89+
usage: turboblaster [-h] [--num-tasks NUM_TASKS] [--timeout-min TIMEOUT_MIN]
90+
[--mem-gb MEM_GB] [--cpus-per-task CPUS_PER_TASK]
91+
[--slurm-partition SLURM_PARTITION]
92+
--listing-input LISTING_INPUT
93+
--bash-slurm-exec BASH_SLURM_EXEC
94+
[--output-dir OUTPUT_DIR]
95+
[--slurm-array-parallelism SLURM_ARRAY_PARALLELISM]
96+
97+
options:
98+
--listing-input Path to a file containing input lines (one task per line) [required]
99+
--bash-slurm-exec Path to the bash script to execute for each task [required]
100+
--num-tasks Number of tasks (unused if reading from file) [default: 20]
101+
--timeout-min Timeout in minutes for each task [default: 20]
102+
--mem-gb Memory in GB for each task [default: 2]
103+
--cpus-per-task Number of CPUs per task [default: 1]
104+
--slurm-partition Slurm partition to use [default: cpu]
105+
--output-dir Directory to store submitit logs [default: submitit_logs_array]
106+
--slurm-array-parallelism Max number of tasks running concurrently [default: 20]
107+
```
108+
109+
Submitit logs (`.out` / `.err` files) are written to a timestamped subdirectory
110+
under `--output-dir`:
111+
112+
```
113+
submitit_logs/
114+
└── 20260309T143000/
115+
├── 12345_0_0.out
116+
├── 12345_1_0.out
117+
└── ...
118+
```
119+
120+
Monitor a specific task with:
121+
122+
```bash
123+
tail -f submitit_logs/20260309T143000/12345_0_0.out
124+
```
125+
126+
## Project structure
127+
128+
```
129+
turboblast/
130+
├── src/
131+
│ └── turboblast/
132+
│ ├── __init__.py # Package entry point, exposes __version__
133+
│ ├── blaster.py # Core logic: argument parsing, job submission, task execution
134+
│ └── logo.py # ASCII art logo used in the CLI help message
135+
├── tests/
136+
│ ├── test_package.py # Package metadata tests (version check)
137+
│ └── test_blaster.py # Unit tests for blaster.py
138+
├── pyproject.toml # Build config, dependencies, tool settings
139+
└── README.md
140+
```
141+
16142
<!-- prettier-ignore-start -->
17143
[actions-badge]: https://github.com/umr-lops/turboblast/workflows/CI/badge.svg
18144
[actions-link]: https://github.com/umr-lops/turboblast/actions

‎tests/test_blaster.py‎

100755100644
Lines changed: 58 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,31 @@
11
import argparse
22
import subprocess
3+
from pathlib import Path
4+
from typing import ClassVar
35
from unittest.mock import MagicMock, patch
46

57
import pytest
68

79
from turboblast.blaster import main, parser_args, process_line
810

9-
1011
# ─── process_line ────────────────────────────────────────────────────────────
1112

13+
1214
class TestProcessLine:
1315
def test_success(self):
1416
with patch("turboblast.blaster.subprocess.run") as mock_run:
1517
mock_run.return_value = MagicMock(returncode=0)
1618
process_line("/path/to/script.sh", "--input a.nc --output /tmp")
1719
mock_run.assert_called_once()
1820
cmd = mock_run.call_args[0][0]
19-
assert cmd == ["bash", "/path/to/script.sh", "--input", "a.nc", "--output", "/tmp"]
21+
assert cmd == [
22+
"bash",
23+
"/path/to/script.sh",
24+
"--input",
25+
"a.nc",
26+
"--output",
27+
"/tmp",
28+
]
2029

2130
def test_passes_env_with_pythonunbuffered(self):
2231
with patch("turboblast.blaster.subprocess.run") as mock_run:
@@ -48,20 +57,23 @@ def test_options_with_quoted_strings(self):
4857

4958
# ─── parser_args ─────────────────────────────────────────────────────────────
5059

60+
5161
class TestParserArgs:
52-
BASE_ARGS = [
53-
"--listing-input", "/data/inputs.txt",
54-
"--bash-slurm-exec", "/scripts/run.sh",
62+
BASE_ARGS: ClassVar[list[str]] = [
63+
"--listing-input",
64+
"/data/inputs.txt",
65+
"--bash-slurm-exec",
66+
"/scripts/run.sh",
5567
]
5668

5769
def test_required_args(self):
58-
with patch("sys.argv", ["blaster"] + self.BASE_ARGS):
70+
with patch("sys.argv", ["blaster", *self.BASE_ARGS]):
5971
args = parser_args()
6072
assert args.listing_input == "/data/inputs.txt"
6173
assert args.bash_slurm_exec == "/scripts/run.sh"
6274

6375
def test_defaults(self):
64-
with patch("sys.argv", ["blaster"] + self.BASE_ARGS):
76+
with patch("sys.argv", ["blaster", *self.BASE_ARGS]):
6577
args = parser_args()
6678
assert args.num_tasks == 20
6779
assert args.timeout_min == 20
@@ -72,29 +84,41 @@ def test_defaults(self):
7284
assert args.output_dir == "submitit_logs_array"
7385

7486
def test_custom_values(self):
75-
with patch("sys.argv", ["blaster"] + self.BASE_ARGS + [
76-
"--timeout-min", "60",
77-
"--mem-gb", "8",
78-
"--slurm-partition", "gpu",
79-
]):
87+
with patch(
88+
"sys.argv",
89+
[
90+
"blaster",
91+
*self.BASE_ARGS,
92+
"--timeout-min",
93+
"60",
94+
"--mem-gb",
95+
"8",
96+
"--slurm-partition",
97+
"gpu",
98+
],
99+
):
80100
args = parser_args()
81101
assert args.timeout_min == 60
82102
assert args.mem_gb == 8
83103
assert args.slurm_partition == "gpu"
84104

85105
def test_missing_required_args_exits(self):
86-
with patch("sys.argv", ["blaster"]):
87-
with pytest.raises(SystemExit):
88-
parser_args()
106+
with patch("sys.argv", ["blaster"]), pytest.raises(SystemExit):
107+
parser_args()
89108

90109

91110
# ─── main ─────────────────────────────────────────────────────────────────────
92111

112+
93113
class TestMain:
94-
def _make_args(self, tmp_path, lines=None):
114+
def _make_args(
115+
self, tmp_path: Path, lines: list[str] | None = None
116+
) -> argparse.Namespace:
95117
input_file = tmp_path / "inputs.txt"
96-
content = "\n".join(lines) if lines is not None else "--input a.nc\n--input b.nc"
97-
input_file.write_text(content)
118+
content = (
119+
"\n".join(lines) if lines is not None else "--input a.nc\n--input b.nc"
120+
)
121+
input_file.write_text(content, encoding="utf-8")
98122
return argparse.Namespace(
99123
listing_input=str(input_file),
100124
bash_slurm_exec="/scripts/run.sh",
@@ -113,15 +137,19 @@ def test_submits_jobs(self, tmp_path):
113137
mock_job.job_id = "12345"
114138
mock_executor.map_array.return_value = [mock_job, mock_job]
115139

116-
with patch("turboblast.blaster.submitit.AutoExecutor", return_value=mock_executor):
140+
with patch(
141+
"turboblast.blaster.submitit.AutoExecutor", return_value=mock_executor
142+
):
117143
main(args)
118144
mock_executor.map_array.assert_called_once()
119145

120146
def test_empty_input_file_aborts(self, tmp_path):
121147
args = self._make_args(tmp_path, lines=[])
122148
mock_executor = MagicMock()
123149

124-
with patch("turboblast.blaster.submitit.AutoExecutor", return_value=mock_executor):
150+
with patch(
151+
"turboblast.blaster.submitit.AutoExecutor", return_value=mock_executor
152+
):
125153
main(args)
126154
mock_executor.map_array.assert_not_called()
127155

@@ -133,18 +161,24 @@ def test_chunks_large_input(self, tmp_path):
133161
mock_job.job_id = "99"
134162
mock_executor.map_array.return_value = [mock_job]
135163

136-
with patch("turboblast.blaster.submitit.AutoExecutor", return_value=mock_executor):
164+
with patch(
165+
"turboblast.blaster.submitit.AutoExecutor", return_value=mock_executor
166+
):
137167
main(args)
138168
assert mock_executor.map_array.call_count == 3 # 1000 + 1000 + 500
139169

140170
def test_blank_lines_ignored(self, tmp_path):
141-
args = self._make_args(tmp_path, lines=["--input a.nc", "", " ", "--input b.nc"])
171+
args = self._make_args(
172+
tmp_path, lines=["--input a.nc", "", " ", "--input b.nc"]
173+
)
142174
mock_executor = MagicMock()
143175
mock_job = MagicMock()
144176
mock_job.job_id = "1"
145177
mock_executor.map_array.return_value = [mock_job, mock_job]
146178

147-
with patch("turboblast.blaster.submitit.AutoExecutor", return_value=mock_executor):
179+
with patch(
180+
"turboblast.blaster.submitit.AutoExecutor", return_value=mock_executor
181+
):
148182
main(args)
149183
submitted = mock_executor.map_array.call_args[0][1]
150-
assert len(submitted) == 2
184+
assert len(submitted) == 2

0 commit comments

Comments
 (0)