You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The advancement of real-time 3D scene reconstruction and novel view synthesis has been significantly propelled by 3D Gaussian Splatting (3DGS). However, effectively training large-scale 3DGS and rendering it in real-time across various scales remains challenging. This paper introduces CityGaussian (CityGS), which employs a novel divide-and-conquer training approach and Level-of-Detail (LoD) strategy for efficient large-scale 3DGS training and rendering. Specifically, the global scene prior and adaptive training data selection enables efficient training and seamless fusion. Based on fused Gaussian primitives, we generate different detail levels through compression, and realize fast rendering across various scales through the proposed block-wise detail levels selection and aggregation strategy. Extensive experimental results on large-scale scenes demonstrate that our approach attains state-of-the-art rendering quality, enabling consistent real-time rendering of large-scale scenes across vastly different scales. **Welcome to visit our [Project Page](https://dekuliutesla.github.io/citygs/)**.
18
+
This repo contains official implementations of our series of work in large-scale scene reconstruction with Gaussian Splatting, Star ⭐ us if you like it!
19
+
-[CityGaussianV2: Efficient and Geometrically Accurate Reconstruction for Large-Scale Scenes](https://dekuliutesla.github.io/CityGaussianV2/)
20
+
-[CityGaussian: Real-time High-quality Large-Scale Scene Rendering with Gaussians](https://arxiv.org/pdf/2404.01133) (ECCV 2024)
The links above points to the papers. Users could follow the [instruction](doc/citygaussianv1_readme.md) to download COLMAP results, checkpoints, and try the V1 of our CityGaussian. The code of V2 is also coming soon.
33
23
34
24
## 📰 News
25
+
**[2024.11.04]** Announcement of our [CityGaussianV2](https://dekuliutesla.github.io/CityGaussianV2/)!
26
+
35
27
**[2024.10.12]** Checkpoints on main datasets have been released!
36
28
37
29
**[2024.10.11]** Updates FAQ! If you are stucked, please first check whether it can solves the problem.
@@ -42,115 +34,13 @@ The advancement of real-time 3D scene reconstruction and novel view synthesis ha
42
34
43
35
**[2024.07.18]** Camera Ready version now can be accessed through arXiv. More insights are included.
44
36
45
-
## 🥏 Model of CityGaussian
46
-
47
-
This repository contains the official implementation of the paper ["CityGaussian: Real-time High-quality Large-Scale Scene Rendering with Gaussians"](https://arxiv.org/pdf/2404.01133). Star ⭐ us if you like it!
Note that the configs for five large-scale scenes: MatrixCity, Rubble, Building, Residence and Sci-Art has been prepared in `config` folder. Data of these datasets can be prepared according to [Data Preparation](doc/data_preparation.md). For COLMAP, we recommend to directly use our generated results:
If you use your own dataset, please follow instruction in [Custom Dataset Instructions](doc/custom_dataset.md) to prepare. We also prepared templates in `./config` and `LargeLightGaussian/scripts`.
112
-
113
-
### Training and Vanilla Rendering
114
-
To train a scene, config the hyperparameters of pretraining and finetuning stage with your yaml file, then replace the `COARSE_CONFIG` and `CONFIG` in `run_citygs.sh`. The `max_block_id`, `out_name`, and `TEST_PATH` in `run_citygs.sh` should be set according to your dataset as well. Then you can train your scene by simply using:
115
-
```bash
116
-
bash scripts/run_citygs.sh
117
-
```
118
-
This script will also render and evaluate the result without LoD.
119
-
120
-
### Rendering with LoD
121
-
First, the LoD generation is realized by the following command:
122
-
```bash
123
-
cd LargeLightGaussian
124
-
bash scripts/run_prune_finetune_$your_scene.sh
125
-
bash scripts/run_distill_finetune_$your_scene.sh
126
-
bash scripts/run_vectree_quantize_$your_scene.sh
127
-
cd ..
128
-
```
129
-
After that, configure the LoD setting in another yaml file. Then replace `CONFIG`, `TEST_PATH`, and `out_name` with yours in `run_citygs_lod.sh`. Then you can render the scene with LoD by using:
130
-
```bash
131
-
bash scripts/run_citygs_lod.sh
132
-
```
133
-
Note that the LoD selection is now based on Nyquist sampling rate instead of manually defined distance threshold. This modification enables better generalization and anti-aliasing performance.
134
-
135
-
### Viewer
136
-
We borrowed Web viewer from [Gaussian Lightning](https://github.com/yzslab/gaussian-splatting-lightning). Take the scene Rubble as an example. To render the scene with no LoD, you can use the following command:
137
-
```bash
138
-
python viewer.py output/rubble_c9_r4
139
-
```
140
-
To render the scene with LoD, you can use the following command:
141
-
```bash
142
-
# copy cameras.json first for direction initialization
title={CityGaussianV2: Efficient and Geometrically Accurate Reconstruction for Large-Scale Scenes},
58
+
author={Yang Liu and Chuanchen Luo and Zhongkai Mao and Junran Peng and Zhaoxiang Zhang},
59
+
year={2024},
60
+
eprint={2411.00771},
61
+
archivePrefix={arXiv},
62
+
primaryClass={cs.CV},
63
+
url={https://arxiv.org/abs/2411.00771},
64
+
}
65
+
```
66
+
165
67
```latex
166
68
@article{liu2024citygaussian,
167
69
title={Citygaussian: Real-time high-quality large-scale scene rendering with gaussians},
168
70
author={Liu, Yang and Guan, He and Luo, Chuanchen and Fan, Lue and Wang, Naiyan and Peng, Junran and Zhang, Zhaoxiang},
169
-
journal={arXiv preprint arXiv:2404.01133},
71
+
booktitle={ECCV},
170
72
year={2024}
171
73
}
172
74
```
@@ -175,11 +77,3 @@ If you find this repository useful, please use the following BibTeX entry for ci
175
77
176
78
This repo benefits from [3DGS](https://github.com/graphdeco-inria/gaussian-splatting), [LightGaussian](https://github.com/VITA-Group/LightGaussian), [Gaussian Lightning](https://github.com/yzslab/gaussian-splatting-lightning). Thanks for their great work!
177
79
178
-
## ❓ FAQ
179
-
-_No available GPU or only single GPU can be used for parallel training._ This may due to lack of enough memory on the GPU. `get_available_gpu()` function in the script will only return GPU id with occupied memory less than 500M. You can rise the `mem_threshold` to 5000(M) for a looser start condition. Another way is to simply kill unnecessary progress.
180
-
181
-
-_Out of memory occurs in training._ For block_all of MatrixCity aerial view, we noticed that it may cost memory more than 24G when fine-tuning specific blocks. To finish training with limited VRAM, downsampling images or adjusting max_cache_num (we used a rather large 1024) in train_large.py can be a useful practice. This problem will be resolved in our coming V2 version.
182
-
183
-
-_Generation of COLMAP results._ We use the ground-truth poses offered by datasets and separately match the train and test sets. And this will be faster and more robust than match from scratch. But indeed it still costs a lot of time.
184
-
185
-
-_Most blocks are not trained._ The main reason here is the data assigned to most blocks are too few (<50), and to prevent overfitting these blocks won't get trained. This can be attributed to unreasonable aabb setting, please try to adjust it and see if things work.
The advancement of real-time 3D scene reconstruction and novel view synthesis has been significantly propelled by 3D Gaussian Splatting (3DGS). However, effectively training large-scale 3DGS and rendering it in real-time across various scales remains challenging. This paper introduces CityGaussian (CityGS), which employs a novel divide-and-conquer training approach and Level-of-Detail (LoD) strategy for efficient large-scale 3DGS training and rendering. Specifically, the global scene prior and adaptive training data selection enables efficient training and seamless fusion. Based on fused Gaussian primitives, we generate different detail levels through compression, and realize fast rendering across various scales through the proposed block-wise detail levels selection and aggregation strategy. Extensive experimental results on large-scale scenes demonstrate that our approach attains state-of-the-art rendering quality, enabling consistent real-time rendering of large-scale scenes across vastly different scales. **Welcome to visit our [Project Page](https://dekuliutesla.github.io/citygs/)**.
Note that the configs for five large-scale scenes: MatrixCity, Rubble, Building, Residence and Sci-Art has been prepared in `config` folder. Data of these datasets can be prepared according to [Data Preparation](doc/data_preparation.md). For COLMAP, we recommend to directly use our generated results:
If you use your own dataset, please follow instruction in [Custom Dataset Instructions](doc/custom_dataset.md) to prepare. We also prepared templates in `./config` and `LargeLightGaussian/scripts`.
100
+
101
+
### Training and Vanilla Rendering
102
+
To train a scene, config the hyperparameters of pretraining and finetuning stage with your yaml file, then replace the `COARSE_CONFIG` and `CONFIG` in `run_citygs.sh`. The `max_block_id`, `out_name`, and `TEST_PATH` in `run_citygs.sh` should be set according to your dataset as well. Then you can train your scene by simply using:
103
+
```bash
104
+
bash scripts/run_citygs.sh
105
+
```
106
+
This script will also render and evaluate the result without LoD.
107
+
108
+
### Rendering with LoD
109
+
First, the LoD generation is realized by the following command:
110
+
```bash
111
+
cd LargeLightGaussian
112
+
bash scripts/run_prune_finetune_$your_scene.sh
113
+
bash scripts/run_distill_finetune_$your_scene.sh
114
+
bash scripts/run_vectree_quantize_$your_scene.sh
115
+
cd ..
116
+
```
117
+
After that, configure the LoD setting in another yaml file. Then replace `CONFIG`, `TEST_PATH`, and `out_name` with yours in `run_citygs_lod.sh`. Then you can render the scene with LoD by using:
118
+
```bash
119
+
bash scripts/run_citygs_lod.sh
120
+
```
121
+
Note that the LoD selection is now based on Nyquist sampling rate instead of manually defined distance threshold. This modification enables better generalization and anti-aliasing performance.
122
+
123
+
### Viewer
124
+
We borrowed Web viewer from [Gaussian Lightning](https://github.com/yzslab/gaussian-splatting-lightning). Take the scene Rubble as an example. To render the scene with no LoD, you can use the following command:
125
+
```bash
126
+
python viewer.py output/rubble_c9_r4
127
+
```
128
+
To render the scene with LoD, you can use the following command:
129
+
```bash
130
+
# copy cameras.json first for direction initialization
This work is under the <arel="license"href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.
141
+
142
+
## 🤗 Citation
143
+
If you find this repository useful, please use the following BibTeX entry for citation.
144
+
```latex
145
+
@article{liu2024citygaussian,
146
+
title={Citygaussian: Real-time high-quality large-scale scene rendering with gaussians},
147
+
author={Liu, Yang and Guan, He and Luo, Chuanchen and Fan, Lue and Wang, Naiyan and Peng, Junran and Zhang, Zhaoxiang},
148
+
journal={arXiv preprint arXiv:2404.01133},
149
+
year={2024}
150
+
}
151
+
```
152
+
153
+
## 👏 Acknowledgements
154
+
155
+
This repo benefits from [3DGS](https://github.com/graphdeco-inria/gaussian-splatting), [LightGaussian](https://github.com/VITA-Group/LightGaussian), [Gaussian Lightning](https://github.com/yzslab/gaussian-splatting-lightning). Thanks for their great work!
156
+
157
+
## ❓ FAQ
158
+
-_No available GPU or only single GPU can be used for parallel training._ This may due to lack of enough memory on the GPU. `get_available_gpu()` function in the script will only return GPU id with occupied memory less than 500M. You can rise the `mem_threshold` to 5000(M) for a looser start condition. Another way is to simply kill unnecessary progress.
159
+
160
+
-_Out of memory occurs in training._ For block_all of MatrixCity aerial view, we noticed that it may cost memory more than 24G when fine-tuning specific blocks. To finish training with limited VRAM, downsampling images or adjusting max_cache_num (we used a rather large 1024) in train_large.py can be a useful practice. This problem will be resolved in our coming V2 version.
161
+
162
+
-_Generation of COLMAP results._ We use the ground-truth poses offered by datasets and separately match the train and test sets. And this will be faster and more robust than match from scratch. But indeed it still costs a lot of time.
163
+
164
+
-_Most blocks are not trained._ The main reason here is the data assigned to most blocks are too few (<50), and to prevent overfitting these blocks won't get trained. This can be attributed to unreasonable aabb setting, please try to adjust it and see if things work.
0 commit comments