You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implement containerd and tar image import functionality
- Added `containerd.rs` for handling image pushes from containerd store, including manifest loading, layer uploads, and config handling.
- Introduced `tar.rs` for extracting layers and config from docker-save tar archives, with support for concurrent uploads.
- Updated `mod.rs` to include new modules for containerd and tar imports.
- Refactored `main.rs` to integrate new import functionalities and adjusted command handling for containerd and tar imports.
- Enhanced `push.rs` to support layer uploads and manifest pushing for tar archives, ensuring proper progress reporting and error handling.
A memory-optimized Docker image transfer tool designed to handle large Docker images without excessive memory usage. This tool addresses the common problem of memory exhaustion when pulling or pushing multi-gigabyte Docker images.
9
-
10
-
## 🎯 Problem Statement
11
-
12
-
Traditional Docker image tools often load entire layers into memory, which can cause:
13
-
-**Memory exhaustion** with large images (>1GB)
14
-
-**System instability** when processing multiple large layers
15
-
-**Failed transfers** due to insufficient RAM
16
-
-**Poor performance** on resource-constrained systems
17
-
18
-
## 🚀 Solution
19
-
20
-
This tool implements **streaming-based layer processing** using the OCI client library:
21
-
22
-
- ✅ **Streaming Downloads**: Layers are streamed directly to disk without loading into memory
23
-
- ✅ **Sequential Processing**: Processes one layer at a time to minimize memory footprint
24
-
- ✅ **Chunked Uploads**: Layers ≥500MB stream in ~5MB chunks (auto-expands when registries demand larger slices)
25
-
- ✅ **Local Caching**: Efficient caching system for faster subsequent operations
26
-
- ✅ **Progress Monitoring**: Real-time feedback on transfer progress and layer sizes
27
-
28
-
## 🆕 What's New in 0.5.4
29
-
30
-
-**Pipeline lives in `oci-core`** – The concurrent extraction/upload queue, blob-existence checks, rate limiting, and telemetry now ship inside the reusable `oci-core::blobs` module. Other projects can embed the exact same uploader without copy/paste.
31
-
-**Prefetch-aware chunk uploads** – Large layers read an initial chunk into memory before network I/O begins, giving registries a steady stream immediately and honoring any server-provided chunk size hints mid-flight.
32
-
-**Tar importer emits shared `LocalLayer` structs** – `tar_import.rs` now returns the exact structs consumed by `LayerUploadPool`, eliminating adapter code and reducing memory copies during extraction.
33
-
-**Cleaner push workflow** – `src/push.rs` delegates scheduling to `LayerUploadPool`, so the CLI only worries about plan setup, manifest publishing, and user prompts. The parallelism cap and chunk sizing still respect the same CLI flags as before.
34
-
-**Docs caught up** – This README now documents the pipeline-focused architecture, the new reusable uploader, and the 0.5.4 feature set.
35
-
36
-
## OCI Core Library
37
-
38
-
The OCI functionality now lives inside `crates/oci-core`, an MIT-licensed library crate that
39
-
can be embedded in other tools. It exposes:
10
+
A tiny, memory‑friendly Docker/OCI image pusher for large images and constrained hosts.
40
11
41
-
-`reference` – a no-dependency reference parser with rich `OciError` signals
42
-
-`auth` – helpers for anonymous/basic auth negotiation
43
-
-`client` – an async `reqwest` uploader/downloader that understands chunked blobs,
44
-
real-time telemetry, and registry-provided chunk hints
12
+
Highlights:
13
+
- Streaming layer uploads with bounded memory
14
+
- Chunked uploads for large layers (auto‑adapts to registry hints)
- Prompts for image selection if you omit arguments.
116
-
- Produces a sanitized tar such as `./nginx_latest.tar`.
117
45
118
46
3. **Push the tar archive**
119
47
```bash
120
-
docker-image-pusher push ./nginx_latest.tar
48
+
docker-image-pusher push --tar ./nginx_latest.tar
121
49
```
122
-
- The RepoTag embedded in the tar is combined with the most recent registry you authenticated against (or`--target/--registry` overrides).
123
-
- If the destination image was confirmed previously, we auto-continue after a short pause; otherwise we prompt before uploading.
50
+
- Use `--target` to set the exact destination. If omitted, we infer from tar metadata and (optionally)`--registry`.
51
+
- If the destination was confirmed previously, we auto-continue after a short pause; otherwise we prompt once.
124
52
125
53
### Command Reference
126
54
127
55
| Command | When to use | Key flags |
128
56
|---------|-------------|-----------|
129
-
|`save [IMAGE ...]`| Export one or more local images to tar archives |`--runtime`, `--output-dir`, `--force`|
130
-
|`push <tar>`| Upload a docker-save tar archive directly to a registry |`-t/--target`, `--registry`, `--username/--password`, `--blob-chunk`|
57
+
|`save [IMAGE ...]`| Export local images to tar/portable folder |`--out`, `--root`, `--namespace`, `--digest`|
58
+
|`push --tar <TAR>`| Upload a docker-save tar directly to a registry |`-t/--target`, `--registry`, `--username/--password`, `--blob-chunk`|
59
+
|`push --image <IMAGE>`| Push directly from containerd (no tar) |`--root`, `--namespace`, `-t/--target`, `--username/--password`, `--blob-chunk`|
131
60
|`login <registry>`| Persist credentials for future pushes |`--username`, `--password`|
132
61
133
-
The `push`command now handles most of the bookkeeping automatically:
134
-
135
-
- infers a sensible destination from tar metadata, the last five targets, or stored credentials
136
-
- prompts once when switching registries (or auto-confirms if you accepted it before)
137
-
- imports `docker save` archives on the fly before uploading
138
-
- reuses saved logins unless you pass explicit `--username/--password`
62
+
### Destination overrides
139
63
64
+
- If the tar contains `docker.io/nginx:v1` and you pass `--target gitea.corp.com/project1`,
65
+
the final destination resolves to `gitea.corp.com/project1/nginx:v1` (repo/tag taken from tar).
66
+
- If you pass a full target like `--target gitea.corp.com/project1/nginx:custom`, it is used as-is.
67
+
- `--registry` only affects inference when `--target` is not provided; it forces the registry host while keeping repo/tag from tar metadata.
140
68
141
-
## 🏗️ Architecture
69
+
## 🧭 Scenarios
142
70
143
-
### Memory Optimization Strategy
71
+
### 1) Push a docker-save tar to a private registry
144
72
145
-
```
146
-
Traditional Approach (High Memory):
147
-
[Registry] → [Full Image in Memory] → [Local Storage]
148
-
↓
149
-
❌ Memory usage scales with image size
150
-
❌ Can exceed available RAM with large images
151
-
152
-
Optimized Approach (Low Memory):
153
-
[Registry] → [Stream Layer by Layer] → [Local Storage]
154
-
↓
155
-
✅ Constant memory usage regardless of image size
156
-
✅ Handles multi-GB images efficiently
73
+
```bash
74
+
docker-image-pusher login harbor.xxx.com --username USER --password PASS
75
+
docker-image-pusher push --tar ./app_1.0.0.tar \
76
+
--target harbor.xxx.com/org/app:1.0.0
157
77
```
158
78
159
-
### State Directory
79
+
Notes:
80
+
- Use `--target` to specify the destination (registry/org/repo:tag)
81
+
- Use `--registry harbor.xxx.com` only if a host override is needed
82
+
- `--blob-chunk` sets chunk size (MiB) for large layers
160
83
161
-
Credential material and push history are stored under `.docker-image-pusher/`:
84
+
### 2) Push directly from containerd (no tar)
162
85
163
-
```
164
-
.docker-image-pusher/
165
-
├── credentials.json # registry → username/password pairs from `login`
166
-
└── push_history.json # most recent destinations (used for inference/prompts)
86
+
```bash
87
+
docker-image-pusher push \
88
+
--root ~/.local/share/containerd \
89
+
--namespace default \
90
+
--image org/app:1.0.0 \
91
+
--target harbor.xxx.com/org/app:1.0.0
167
92
```
168
93
169
-
Tar archives produced by `save` live wherever you choose to write them (current directory by default). They remain ordinary `docker save` outputs, so you can transfer them, scan them, or delete them independently of the CLI state.
94
+
Optional: exportfor offline delivery/audit, then push when needed:
0 commit comments