Skip to content

Commit ea6ef68

Browse files
mudongliangmudongliang
authored
Merge branch 'pwncollege:master' into master
2 parents 70ba32d + 6fdeb18 commit ea6ef68

506 files changed

Lines changed: 38190 additions & 8074 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/skills/run-tests/SKILL.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
---
2+
name: run-tests
3+
description: Run dojo tests, analyze failures, and debug issues. Use when testing code changes, running pytest, verifying fixes, or debugging test failures.
4+
---
5+
6+
# Running Tests
7+
8+
Tests take ~10 minutes to run, so run them as a background process.
9+
10+
## Running All Tests
11+
12+
Start tests in background:
13+
```bash
14+
./deploy.sh -b -t 2>&1 | tee /tmp/test.log &
15+
```
16+
17+
Monitor progress:
18+
```bash
19+
tail -f /tmp/test.log
20+
```
21+
22+
## Running Specific Tests
23+
24+
To run only specific test files quickly, edit `deploy.sh` line:
25+
```bash
26+
test_container pytest --order-dependencies --timeout=60 -v . "$@"
27+
```
28+
29+
Change it to run a specific file:
30+
```bash
31+
test_container pytest --order-dependencies --timeout=60 -v test_belts.py "$@"
32+
```
33+
34+
Then run:
35+
```bash
36+
./deploy.sh -b -t 2>&1 | tee /tmp/test.log &
37+
```
38+
39+
**Important:** After specific tests pass, restore the original line and rerun ALL tests.
40+
41+
## Analyzing Failures
42+
43+
After tests complete, analyze `/tmp/test.log`:
44+
1. Search for `FAILED` to find failing tests
45+
2. Search for `ERROR` to find errors
46+
3. Look at the full traceback for each failure
47+
48+
## Debugging Workflow
49+
50+
1. **Identify failures** - Analyze `/tmp/test.log` for FAILED/ERROR
51+
2. **Find root cause** - Look at tracebacks, check CTFd logs with:
52+
```bash
53+
docker exec $(basename "$PWD") docker logs ctfd 2>&1 | tail -100
54+
```
55+
You can also look at all any of the containers inside of `docker exec $(basename "$PWD")`, the whole list including names is in docker-compose.yml.
56+
3. **If root cause unclear** - Add logging to help understand, then rerun tests
57+
4. **Fix the root cause** - Don't just fix symptoms
58+
5. **Run specific tests** - Edit deploy.sh to run just the affected test file
59+
6. **Verify fix** - Ensure the specific test passes
60+
7. **Run ALL tests** - Restore deploy.sh and run full suite before declaring success
61+
62+
## Key Commands
63+
64+
```bash
65+
# Start all tests (background)
66+
./deploy.sh -b -t 2>&1 | tee /tmp/test.log &
67+
68+
# Check if tests are still running
69+
jobs
70+
71+
# View test output
72+
cat /tmp/test.log
73+
74+
# Search for failures
75+
grep -E "(FAILED|ERROR|error)" /tmp/test.log
76+
77+
# View CTFd logs for debugging
78+
docker exec $(basename "$PWD") docker logs ctfd 2>&1 | tail -200
79+
80+
# Run DB queries for debugging
81+
docker exec -i $(basename "$PWD") dojo db
82+
```
83+
84+
## Multinode Testing
85+
86+
After all singlenode tests pass, run multinode tests to verify the full cluster setup:
87+
88+
```bash
89+
./deploy.sh -b -M -t 2>&1 | tee /tmp/test-multinode.log &
90+
```
91+
92+
Monitor progress:
93+
```bash
94+
tail -f /tmp/test-multinode.log
95+
```
96+
97+
Multinode mode starts 3 containers (1 main + 2 workspace nodes) and tests the distributed architecture. Some things that only run on the main node (like stats-worker) need special handling in multinode.
98+
99+
## Critical Reminder
100+
101+
**ALWAYS rerun all singlenode tests before declaring everything passed.** Fixing one test can break others.
102+
103+
**After singlenode tests pass, run multinode tests (`-M` flag) to verify the full cluster setup works correctly.**

‎.devcontainer/devcontainer.json‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
{
2+
"name": "Development",
3+
"build": {
4+
"dockerfile": "../Dockerfile",
5+
"context": ".."
6+
},
7+
"workspaceFolder": "/opt/pwn.college",
8+
"workspaceMount": "source=${localWorkspaceFolder},target=/opt/pwn.college,type=bind",
9+
"runArgs": ["-v", "/tmp/dojo-data-${devcontainerId}:/data"],
10+
"privileged": true,
11+
"overrideCommand": false,
12+
"forwardPorts": [22, 80, 443],
13+
"portsAttributes": {
14+
"22": {
15+
"label": "SSH",
16+
"onAutoForward": "silent"
17+
},
18+
"80": {
19+
"label": "HTTP",
20+
"protocol": "http",
21+
"onAutoForward": "openPreview"
22+
},
23+
"443": {
24+
"label": "HTTPS",
25+
"onAutoForward": "silent"
26+
}
27+
},
28+
"customizations": {
29+
"vscode": {
30+
"extensions": [
31+
"ms-python.python",
32+
"ms-azuretools.vscode-docker",
33+
"ms-vscode-remote.remote-containers"
34+
],
35+
"tasks": [
36+
{
37+
"label": "Wait for dojo to be ready",
38+
"type": "shell",
39+
"command": "dojo wait",
40+
"runOptions": {
41+
"runOn": "folderOpen"
42+
}
43+
}
44+
]
45+
}
46+
}
47+
}

‎.dockerignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1 +1,2 @@
11
data/
2+
cache/

‎.github/workflows/ci.yml‎

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
name: DOJO CI
2+
on:
3+
workflow_dispatch:
4+
pull_request:
5+
push:
6+
branches:
7+
- master
8+
schedule:
9+
- cron: "42 06 * * *"
10+
jobs:
11+
test:
12+
name: "${{ matrix.mode }} DOJO Test"
13+
runs-on: ubuntu-latest
14+
timeout-minutes: 60
15+
strategy:
16+
matrix:
17+
mode: [singlenode, multinode]
18+
fail-fast: false
19+
steps:
20+
- uses: actions/checkout@v4
21+
22+
- name: Host information
23+
run: |
24+
echo "::group::Host information"
25+
echo "Hostname: $(hostname)"
26+
echo "OS: $(lsb_release -d | cut -f2)"
27+
echo "Kernel: $(uname -r)"
28+
echo "Architecture: $(uname -m)"
29+
echo "IP: $(hostname -I)"
30+
echo "::endgroup::"
31+
echo "::group::Repo"
32+
pwd
33+
git log </dev/null | head -n20
34+
echo "::endgroup::"
35+
echo "::group::df -h"
36+
df -h
37+
echo "::endgroup::"
38+
echo "::group::free -h"
39+
free -h
40+
echo "::endgroup::"
41+
echo "::group::lscpu"
42+
lscpu
43+
echo "::endgroup::"
44+
echo "::group::docker images"
45+
docker images
46+
echo "::endgroup::"
47+
48+
- name: Set cache mode
49+
run: |
50+
if [ "${{ matrix.mode }}" = "singlenode" ] && \
51+
{ [ "${{ github.event_name }}" = "schedule" ] || [ "${{ github.event_name }}" = "workflow_dispatch" ]; }; then
52+
echo "SAVE_CACHE=yes" >> "$GITHUB_ENV"
53+
else
54+
echo "LOAD_CACHE=yes" >> "$GITHUB_ENV"
55+
sudo chown $USER:$USER /mnt
56+
fi
57+
58+
- name: Restore docker and workspace cache
59+
if: env.LOAD_CACHE == 'yes'
60+
uses: actions/cache/restore@v4
61+
with:
62+
path: |
63+
/mnt/cache.img
64+
key: dojo-cache-${{ matrix.mode }}-${{ github.run_id }}
65+
restore-keys: |
66+
dojo-cache-
67+
68+
- name: Setup cache filesystem
69+
run: |
70+
sudo chown root:root /mnt
71+
set -x
72+
df -h
73+
sudo mkdir -p /mnt/cache
74+
sudo mkdir -p /mnt/data
75+
if [ ! -e /mnt/cache.img ]; then
76+
sudo fallocate -l 9.8G /mnt/cache.img
77+
sudo mkfs.btrfs -L cache /mnt/cache.img
78+
fi
79+
80+
if [ "$SAVE_CACHE" != "yes" ]; then
81+
sudo truncate -s +20G /mnt/cache.img
82+
fi
83+
84+
sudo mount -o loop,compress=zstd:9 /mnt/cache.img /mnt/cache
85+
sudo btrfs filesystem resize max /mnt/cache
86+
87+
sudo chown $USER:$USER /mnt/cache
88+
sudo chown $USER:$USER /mnt/data
89+
mkdir -p /mnt/cache/dojo-image
90+
mkdir -p /mnt/cache/test-image
91+
mkdir -p /mnt/data/docker
92+
mkdir -p /mnt/data/workspace
93+
if [ "${{ matrix.mode }}" = "multinode" ]; then
94+
sudo rm -rf /mnt/data/docker-node1 /mnt/data/docker-node2
95+
sudo cp -a /mnt/data/docker /mnt/data/docker-node1
96+
sudo cp -a /mnt/data/docker /mnt/data/docker-node2
97+
fi
98+
99+
- name: Set up Docker storage
100+
run: |
101+
echo "::group::Current disk usage"
102+
df -h
103+
echo "::endgroup::"
104+
sudo systemctl stop docker
105+
sudo rm -rf /var/lib/docker
106+
sudo mkdir -p /mnt/docker
107+
sudo ln -s /mnt/docker /var/lib/docker
108+
sudo systemctl start docker
109+
110+
- uses: docker/setup-buildx-action@v3
111+
112+
- name: Build outer docker image
113+
uses: docker/build-push-action@v5
114+
with:
115+
context: .
116+
tags: pwncollege/dojo
117+
load: true
118+
cache-from: type=local,src=/mnt/cache/dojo-image
119+
cache-to: type=local,dest=/mnt/cache/dojo-image,mode=max
120+
121+
- name: Build test docker image
122+
uses: docker/build-push-action@v5
123+
with:
124+
context: test
125+
tags: dojo-test
126+
load: true
127+
cache-from: type=local,src=/mnt/cache/test-image
128+
cache-to: type=local,dest=/mnt/cache/test-image,mode=max
129+
130+
- name: Start the dojo in ${{ matrix.mode }} mode
131+
timeout-minutes: 15
132+
run: |
133+
./deploy.sh -g -D /mnt/data/docker -W /mnt/data/workspace ${{ matrix.mode == 'multinode' && '-M' || '' }} -C
134+
135+
- name: Run ${{ matrix.mode }} dojo tests
136+
timeout-minutes: 15
137+
run: |
138+
./deploy.sh -g -D /mnt/data/docker -W /mnt/data/workspace ${{ matrix.mode == 'multinode' && '-M' || '' }} -C -N -t
139+
140+
- name: Upload coverage reports to Codecov
141+
if: matrix.mode == 'singlenode'
142+
uses: codecov/codecov-action@v5
143+
with:
144+
token: ${{ secrets.CODECOV_TOKEN }}
145+
146+
- name: Output logs
147+
if: always()
148+
run: |
149+
echo "::group::Host docker ps"
150+
docker ps
151+
echo "::endgroup::"
152+
echo "::group::Host logs"
153+
journalctl -u '*' -b
154+
echo "::endgroup::"
155+
echo "::group::Main node systemd logs"
156+
docker exec dojo journalctl -u '*' -b || echo "No main outer container..."
157+
echo "::endgroup::"
158+
echo "::group::Main node compose logs"
159+
docker exec dojo dojo compose logs || echo "No main outer container..."
160+
echo "::endgroup::"
161+
if [ "${{ matrix.mode }}" = "multinode" ]; then
162+
echo "::group::Workspace node 1 systemd logs"
163+
docker exec dojo-node1 journalctl -u '*' -b || echo "No node1 outer container..."
164+
echo "::endgroup::"
165+
echo "::group::Workspace node 1 compose logs"
166+
docker exec dojo-node1 dojo compose logs || echo "No node1 outer container..."
167+
echo "::endgroup::"
168+
echo "::group::Workspace node 2 systemd logs"
169+
docker exec dojo-node2 journalctl -u '*' -b || echo "No node2 outer container..."
170+
echo "::endgroup::"
171+
echo "::group::Workspace node 2 compose logs"
172+
docker exec dojo-node2 dojo compose logs || echo "No node2 outer container..."
173+
echo "::endgroup::"
174+
fi
175+
176+
- name: Prepare the cache
177+
if: env.SAVE_CACHE == 'yes'
178+
run: |
179+
./deploy.sh -K -g
180+
df -h
181+
sudo umount /mnt/cache
182+
du -sm /mnt/cache.img
183+
184+
- name: Save docker and workspace cache
185+
if: env.SAVE_CACHE == 'yes'
186+
uses: actions/cache/save@v4
187+
with:
188+
path: |
189+
/mnt/cache.img
190+
key: dojo-cache-${{ matrix.mode }}-${{ github.run_id }}
191+
192+
- name: Final filesystem information
193+
if: always()
194+
run: |
195+
echo "::group::Filesystem"
196+
df -h
197+
echo "::endgroup::"
198+
199+
upload:
200+
name: Upload Artifacts
201+
runs-on: ubuntu-latest
202+
needs: test
203+
if: github.ref == 'refs/heads/master'
204+
steps:
205+
- name: Login to Docker Hub
206+
uses: docker/login-action@v3
207+
with:
208+
username: ${{ secrets.DOCKERHUB_USERNAME }}
209+
password: ${{ secrets.DOCKERHUB_TOKEN }}
210+
211+
- name: Set up Docker Buildx
212+
uses: docker/setup-buildx-action@v3
213+
214+
- name: Build and push
215+
uses: docker/build-push-action@v6
216+
with:
217+
push: true
218+
tags: pwncollege/dojo:latest

0 commit comments

Comments
 (0)