graphdeco-inria · alexxx-yyer · Oct 24, 2024 · Oct 24, 2024 · Oct 24, 2024 · Oct 24, 2024
diff --git a/.gitmodules b/.gitmodules
@@ -1,10 +0,0 @@
-[submodule "submodules/simple-knn"]
-	path = submodules/simple-knn
-	url = https://gitlab.inria.fr/bkerbl/simple-knn.git
-[submodule "submodules/diff-gaussian-rasterization"]
-	path = submodules/diff-gaussian-rasterization
-	url = https://github.com/graphdeco-inria/diff-gaussian-rasterization.git
-	branch = dr_aa
-[submodule "SIBR_viewers"]
-	path = SIBR_viewers
-	url = https://gitlab.inria.fr/sibr/sibr_core.git

diff --git a/README.md b/README.md
@@ -37,12 +37,11 @@ This research was funded by the ERC Advanced grant FUNGRAPH No 788065. The autho
 
 ## NEW FEATURES !
 
-We have limited resources for maintaining and updating the code. However, we have added a few new features since the original release that are inspired by some of the excellent work many other researchers have been doing on 3DGS. We will be adding other features within the ability of our resources. 
+We have limited resources for maintaining and updating the code. However, we have added a few new features since the original release that are inspired by some of the excellent work many other researchers have been doing on 3DGS. We will be adding other features within the ability of our resources.
 
-Update of August 2024:
-We have added/corrected the following features: [Depth regularization](#depth-regularization) for training, [anti-aliasing](#anti-aliasing) and [exposure compensation](#exposure-compensation). We have enhanced the SIBR real time viewer by correcting bugs and adding features in the [Top View](#sibr-top-view) that allows visualization of input and user cameras. Please note that it is currently not possible to use depth regularization with the training speed acceleration since they use different rasterizer versions.
+**Update of October 2024**: We integrated [training speed acceleration](#training-speed-acceleration) and made it compatible with [depth regularization](#depth-regularization), [anti-aliasing](#anti-aliasing) and [exposure compensation](#exposure-compensation). We have enhanced the SIBR real time viewer by correcting bugs and adding features in the [Top View](#sibr-top-view) that allows visualization of input and user cameras.
 
-Update of Spring 2024:
+**Update of Spring 2024**:
 Orange Labs has kindly added [OpenXR support](#openxr-support) for VR viewing. 
 
 ## Step-by-step Tutorial
@@ -492,23 +491,60 @@ python convert.py -s <location> --skip_matching [--resize] #If not resizing, Ima
 </details>
 <br>
 
+### Training speed acceleration
+
+We integrated the drop-in replacements from [Taming-3dgs](https://humansensinglab.github.io/taming-3dgs/)<sup>1</sup> with [fused ssim](https://github.com/rahul-goel/fused-ssim/tree/main) into the original codebase to speed up training times. Once installed, the accelerated rasterizer delivers a **$\times$ 1.6 training time speedup** using `--optimizer_type default` and a **$\times$ 2.7 training time speedup** using `--optimizer_type sparse_adam`.
+
+To get faster training times you must first install the accelerated rasterizer to your environment:
+
+```bash
+pip uninstall diff-gaussian-rasterization -y
+cd submodules/diff-gaussian-rasterization
+rm -r build
+git checkout 3dgs_accel
+pip install .
+```
+
+Then you can add the following parameter to use the sparse adam optimizer when running `train.py`:
+
+```bash
+--optimizer_type sparse_adam
+```
+
+*Note that this custom rasterizer has a different behaviour than the original version, for more details on training times please see [stats for training times](results.md/#training-times-comparisons)*.
+
+*1. Mallick and Goel, et al. ‘Taming 3DGS: High-Quality Radiance Fields with Limited Resources’. SIGGRAPH Asia 2024 Conference Papers, 2024, https://doi.org/10.1145/3680528.3687694, [github](https://github.com/humansensinglab/taming-3dgs)*
+
+
 ### Depth regularization
 
+To have better reconstructed scenes we use depth maps as priors during optimization with each input images. It works best on untextured parts ex: roads and can remove floaters. Several papers have used similar ideas to improve various aspects of 3DGS; (e.g. [DepthRegularizedGS](https://robot0321.github.io/DepthRegGS/index.html), [SparseGS](https://formycat.github.io/SparseGS-Real-Time-360-Sparse-View-Synthesis-using-Gaussian-Splatting/), [DNGaussian](https://fictionarry.github.io/DNGaussian/)). The depth regularization we integrated is that used in our [Hierarchical 3DGS](https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/) paper, but applied to the original 3DGS; for some scenes (e.g., the DeepBlending scenes) it improves quality significantly; for others it either makes a small difference or can even be worse. For example results showing the potential benefit and statistics on quality please see here: [Stats for depth regularization](results.md).
 
-Two preprocessing steps are required to enable depth regularization when training a scene:
-  To have better reconstructed scenes we use depth maps as priors during optimization with each input images. It works best on untextured parts ex: roads and can remove floaters. Several papers have used similar ideas to improve various aspects of 3DGS; (e.g. [DepthRegularizedGS](https://robot0321.github.io/DepthRegGS/index.html), [SparseGS](https://formycat.github.io/SparseGS-Real-Time-360-Sparse-View-Synthesis-using-Gaussian-Splatting/), [DNGaussian](https://fictionarry.github.io/DNGaussian/)). The depth regularization we integrated is that used in our [Hierarchical 3DGS](https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/) paper, but applied to the original 3DGS; for some scenes (e.g., the DeepBlending scenes) it improves quality significantly; for others it either makes a small difference or can even be worse. For details statistics please see here: [Stats for depth regularization](results.md).
+When training on a synthetic dataset, depth maps can be produced and they do not require further processing to be used in our method.
 
-When training on a synthetic dataset, depth maps can be produced and they do not require further processing to be used in our method. For real world datasets please do the following: 
-1. Get depth maps for each input images, to this effect we suggest using [Depth anything v2](https://github.com/DepthAnything/Depth-Anything-V2?tab=readme-ov-file#usage).
-2. Generate a `depth_params.json` file using:
+For real world datasets depth maps should be generated for each input images, to generate them please do the following:
+1. Clone [Depth Anything v2](https://github.com/DepthAnything/Depth-Anything-V2?tab=readme-ov-file#usage):
+    ```
+    git clone https://github.com/DepthAnything/Depth-Anything-V2.git
+    ```
+2. Download weights from [Depth-Anything-V2-Large](https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth?download=true) and place it under `Depth-Anything-V2/checkpoints/`
+3. Generate depth maps:
+   ```
+   python Depth-Anything-V2/run.py --encoder vitl --pred-only --grayscale --img-path <path to input images> --outdir <output path>
+   ```
+5. Generate a `depth_params.json` file using:
     ```
     python utils/make_depth_scale.py --base_dir <path to colmap> --depths_dir <path to generated depths>
     ```
 
 A new parameter should be set when training if you want to use depth regularization `-d <path to depth maps>`.
 
 ### Exposure compensation
-To compensate for exposure changes in the different input images we optimize an affine transformation for each image just as in [Hierarchical 3dgs](https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/). Add the following parameters to enable it:
+To compensate for exposure changes in the different input images we optimize an affine transformation for each image just as in [Hierarchical 3dgs](https://repo-sam.inria.fr/fungraph/hierarchical-3d-gaussians/).  
+
+This can greatly improve reconstruction results for "in the wild" captures, e.g., with a smartphone when the exposure setting of the camera is not fixed. For example results showing the potential benefit and statistics on quality please see here: [Stats for exposure compensation](results.md).
+
+Add the following parameters to enable it:
 ```
 --exposure_lr_init 0.001 --exposure_lr_final 0.0001 --exposure_lr_delay_steps 5000 --exposure_lr_delay_mult 0.001 --train_test_exp
 ```

diff --git a/SIBR_viewers b/SIBR_viewers
diff --git a/SIBR_viewers/.gitignore b/SIBR_viewers/.gitignore
@@ -0,0 +1,45 @@
+extlibs/
+build/
+install/
+src/projects/*
+cmake-gui.exe.stackdump
+__pycache__/
+
+# emacs garbage
+\#*
+.\#*
+
+# vim garbage
+*.swp
+*.swo
+*.idea/
+*.log
+*.sh
+*.tmp
+
+hs_err_*
+
+# re include common public projects
+!src/projects/ulr/
+!src/projects/dataset_tools/
+
+# more vim garbage
+# Swap
+[._]*.s[a-v][a-z]
+!*.svg  # comment out if you don't need vector files
+[._]*.sw[a-p]
+[._]s[a-rt-v][a-z]
+[._]ss[a-gi-z]
+[._]sw[a-p]
+
+# Session
+Session.vim
+Sessionx.vim
+
+# Temporary
+.netrwhist
+*~
+# Auto-generated tag files
+tags
+# Persistent undo
+[._]*.un~
diff --git a/SIBR_viewers/CMakeLists.txt b/SIBR_viewers/CMakeLists.txt
@@ -0,0 +1,213 @@
+# Copyright (C) 2020, Inria
+# GRAPHDECO research group, https://team.inria.fr/graphdeco
+# All rights reserved.
+# 
+# This software is free for non-commercial, research and evaluation use 
+# under the terms of the LICENSE.md file.
+# 
+# For inquiries contact sibr@inria.fr and/or George.Drettakis@inria.fr
+
+CMAKE_MINIMUM_REQUIRED(VERSION 3.22)
+
+set (CMAKE_SYSTEM_VERSION 10.0.15063.0 CACHE INTERNAL "Cmake system version" FORCE)
+PROJECT(sibr_projects)
+
+set(REQUIRED_VERSION "3.22.0")
+set(CHECKED_VERSION "3.27.0")
+
+if (CMAKE_VERSION VERSION_LESS REQUIRED_VERSION)
+    message(WARNING "Deprecated version of cmake. Please update to at least ${REQUIRED_VERSION} (${CHECKED_VERSION} recommended).")
+elseif (CMAKE_VERSION VERSION_GREATER CHECKED_VERSION)
+	message(WARNING "Untested version of cmake. If you checked everything is working properly, please update ${CHECKED_VERSION} in the main CmakeLists.txt with the version you tested.")
+endif()
+
+## Include cmake stuff (functions/macros) : Modules files
+if(WIN32)
+list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake/windows)
+list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake/windows/Modules)
+else()
+list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake/linux)
+list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake/linux/Modules)
+endif()
+set_property(GLOBAL PROPERTY USE_FOLDERS ON)
+
+## To maintain cmake versions compatibilities
+include(cmake_policies)
+setPolicies()
+
+include(git_describe)
+git_describe(GIT_BRANCH SIBR_CORE_BRANCH GIT_COMMIT_HASH SIBR_CORE_COMMIT_HASH GIT_TAG SIBR_CORE_TAG GIT_VERSION SIBR_CORE_VERSION)
+
+message(STATUS "SIBR version :\n BRANCH ${SIBR_CORE_BRANCH}\n COMMIT_HASH ${SIBR_CORE_COMMIT_HASH}\n TAG ${SIBR_CORE_TAG}\n VERSION ${SIBR_CORE_VERSION}")
+
+if(NOT WIN32)
+set(CMAKE_CXX_STANDARD 17)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+endif()
+
+
+if (WIN32)
+	## Allow C++11 + other flags
+	include(CheckCXXCompilerFlag)
+	get_filename_component(currentBuildTool ${CMAKE_BUILD_TOOL} NAME_WE)	# tool that can launch the native build system. returned value may be the full path
+	if(${currentBuildTool} MATCHES "(msdev|devenv|nmake|MSBuild)")
+
+		add_compile_options("$<$<COMPILE_LANGUAGE:CXX>:/W3;/DNOMINMAX;/MP;-D_USE_MATH_DEFINES>")
+	    #add_definitions(/W3 /DNOMINMAX /MP -D_USE_MATH_DEFINES)# /D_ITERATOR_DEBUG_LEVEL=1 because you need all external DLl to compile with this flag too
+	    set(CMAKE_CONFIGURATION_TYPES "RelWithDebInfo;Release;Debug" CACHE STRING "" FORCE)
+	    set(CMAKE_CXX_STANDARD 14)
+		set(CMAKE_CXX_STANDARD_REQUIRED ON)
+		set(CMAKE_CXX_EXTENSIONS OFF)
+	elseif(${currentBuildTool} MATCHES "(make|gmake)")
+	    add_definitions("-Wall -Wno-unknown-pragmas -Wno-sign-compare -g -std=c++14 -D__forceinline=\"inline\ __attribute__((always_inline))\"")
+		# CHECK_CXX_COMPILER_FLAG("-std=gnu++11" COMPILER_SUPPORTS_CXX11)
+		# CHECK_CXX_COMPILER_FLAG("-std=gnu++0x" COMPILER_SUPPORTS_CXX0X)
+		# if(COMPILER_SUPPORTS_CXX11)
+		# 	add_definitions(-std=gnu++11)
+		# elseif(COMPILER_SUPPORTS_CXX0X)
+		# 	add_definitions(-std=gnu++0x)
+		# else()
+		# 	message(SEND_ERROR "The compiler ${CMAKE_CXX_COMPILER} has no C++14 support. Please use a different C++ compiler.")
+		# endif()
+	elseif(APPLE) ## \todo TODO: do a better test and send error on unsupported c++14 compiler
+	    add_definitions(-std=c++14 -stdlib=libc++)
+	endif()
+else()
+	## Allow C++11 + other flags
+	include(CheckCXXCompilerFlag)
+	get_filename_component(currentBuildTool ${CMAKE_BUILD_TOOL} NAME_WE)	# tool that can launch the native build system. returned value may be the full path
+	if(${currentBuildTool} MATCHES "(msdev|devenv|nmake|MSBuild)")
+
+		add_compile_options("$<$<COMPILE_LANGUAGE:CXX>:/W3;/DNOMINMAX;/MP;-D_USE_MATH_DEFINES>")
+	    #add_definitions(/W3 /DNOMINMAX /MP -D_USE_MATH_DEFINES)# /D_ITERATOR_DEBUG_LEVEL=1 because you need all external DLl to compile with this flag too
+	    set(CMAKE_CONFIGURATION_TYPES "RelWithDebInfo;Release;Debug" CACHE STRING "" FORCE)
+	    set(CMAKE_CXX_STANDARD 14)
+		set(CMAKE_CXX_STANDARD_REQUIRED ON)
+		set(CMAKE_CXX_EXTENSIONS OFF)
+	elseif(${currentBuildTool} MATCHES "(make|gmake|ninja)")
+		add_definitions("-fpermissive -fPIC -Wall -Wno-unknown-pragmas -Wno-sign-compare -g -std=c++17 -D__forceinline=\"inline\ __attribute__((always_inline))\"")
+	elseif(APPLE) ## \todo TODO: do a better test and send error on unsupported c++14 compiler
+	    add_definitions(-std=c++17 -stdlib=libc++)
+	endif()
+endif()
+
+set(INSTALL_STANDALONE ON)
+
+## Set default build output binaries (used also in sub CMakeLists.txt) :
+set(BIN_BUILT_DIR "bin")
+if(CMAKE_SIZEOF_VOID_P EQUAL 8)
+	set(ARCHI_BUILT_DIR "x64")
+	set(LIB_BUILT_DIR "lib64")
+else()
+	set(ARCHI_BUILT_DIR "x86")
+	set(LIB_BUILT_DIR "lib")
+endif()
+
+option(SEPARATE_CONFIGURATIONS "Clearer separation between configurations" OFF)
+SET(CMAKE_INSTALL_ROOT ${CMAKE_CURRENT_SOURCE_DIR}/install)
+SET(CMAKE_INSTALL_PREFIX ${CMAKE_INSTALL_ROOT})
+
+if(DEFINED CMAKE_BUILD_TYPE) ## for mono config type (make/nmake/ninja based)
+	if(${CMAKE_BUILD_TYPE} MATCHES "Debug")
+		set(CMAKE_DEBUG_POSTFIX "_d")
+	elseif(${CMAKE_BUILD_TYPE} MATCHES "RelWithDebInfo")
+		set(CMAKE_RELWITHDEBINFO_POSTFIX "_rwdi")
+	elseif(${CMAKE_BUILD_TYPE} MATCHES "MinSizeRel")
+		set(CMAKE_MINSIZEREL_POSTFIX "_msr")
+	elseif(${CMAKE_BUILD_TYPE} MATCHES "Release")
+		set(CMAKE_RELEASE_POSTFIX "")
+	endif()
+
+	if(SEPARATE_CONFIGURATIONS)
+		SET(CMAKE_INSTALL_PREFIX_${CMAKE_BUILD_TYPE} 	${CMAKE_INSTALL_ROOT}/${CMAKE_BUILD_TYPE})
+	else()
+		SET(CMAKE_INSTALL_PREFIX_${CMAKE_BUILD_TYPE} 	${CMAKE_INSTALL_ROOT})
+	endif()
+
+	MESSAGE(STATUS "Install path set to ${CMAKE_INSTALL_PREFIX}.")
+	SET(CMAKE_OUTPUT_LIB_${CMAKE_BUILD_TYPE} 	${CMAKE_INSTALL_PREFIX_${CMAKE_BUILD_TYPE}}/lib)
+	SET(CMAKE_OUTPUT_BIN_${CMAKE_BUILD_TYPE} 	${CMAKE_INSTALL_PREFIX_${CMAKE_BUILD_TYPE}}/bin)
+
+	set(CMAKE_LIBRARY_OUTPUT_DIRECTORY_${CMAKE_BUILD_TYPE} 	${CMAKE_OUTPUT_LIB_${CMAKE_BUILD_TYPE}})
+	set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY_${CMAKE_BUILD_TYPE} 	${CMAKE_OUTPUT_LIB_${CMAKE_BUILD_TYPE}})
+	set(CMAKE_RUNTIME_OUTPUT_DIRECTORY_${CMAKE_BUILD_TYPE} 	${CMAKE_OUTPUT_BIN_${CMAKE_BUILD_TYPE}})
+	set(CMAKE_PDB_OUTPUT_DIRECTORY_${CMAKE_BUILD_TYPE} 		${CMAKE_OUTPUT_BIN_${CMAKE_BUILD_TYPE}})
+endif()
+foreach(CONFIG_TYPES ${CMAKE_CONFIGURATION_TYPES}) ## for multi config types (MSVC based)
+	string(TOUPPER ${CONFIG_TYPES} CONFIG_TYPES_UC)
+	if(${CONFIG_TYPES} MATCHES "Debug")
+		set(CMAKE_DEBUG_POSTFIX "_d")
+	elseif(${CONFIG_TYPES} MATCHES "RelWithDebInfo")
+		set(CMAKE_RELWITHDEBINFO_POSTFIX "_rwdi")
+	elseif(${CONFIG_TYPES} MATCHES "MinSizeRel")
+		set(CMAKE_MINSIZEREL_POSTFIX "_msr")
+	elseif(${CMAKE_BUILD_TYPE} MATCHES "Release")
+		set(CMAKE_RELEASE_POSTFIX "")
+	endif()
+
+	if(SEPARATE_CONFIGURATIONS)
+		SET(CMAKE_INSTALL_PREFIX_${CONFIG_TYPES_UC} ${CMAKE_INSTALL_ROOT}/${CONFIG_TYPES})
+	else()
+		SET(CMAKE_INSTALL_PREFIX_${CONFIG_TYPES_UC} ${CMAKE_INSTALL_ROOT})
+	endif()
+
+	MESSAGE(STATUS "Install path for ${CONFIG_TYPES} set to ${CMAKE_INSTALL_PREFIX_${CONFIG_TYPES_UC}}.")
+	SET(CMAKE_OUTPUT_LIB_${CONFIG_TYPES_UC} 	${CMAKE_INSTALL_PREFIX_${CONFIG_TYPES_UC}}/lib)
+	SET(CMAKE_OUTPUT_BIN_${CONFIG_TYPES_UC} 	${CMAKE_INSTALL_PREFIX_${CONFIG_TYPES_UC}}/bin)
+
+	set(CMAKE_LIBRARY_OUTPUT_DIRECTORY_${CONFIG_TYPES_UC} 	${CMAKE_OUTPUT_LIB_${CONFIG_TYPES_UC}})
+	set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY_${CONFIG_TYPES_UC} 	${CMAKE_OUTPUT_LIB_${CONFIG_TYPES_UC}})
+	set(CMAKE_RUNTIME_OUTPUT_DIRECTORY_${CONFIG_TYPES_UC} 	${CMAKE_OUTPUT_BIN_${CONFIG_TYPES_UC}})
+	set(CMAKE_PDB_OUTPUT_DIRECTORY_${CONFIG_TYPES_UC} 		${CMAKE_OUTPUT_BIN_${CONFIG_TYPES_UC}})
+endforeach()
+
+
+# Settings for RPATH
+if (NOT WIN32)
+	# Default config of Fedora at INRIA has no LD_LIBRARY_PATH (for security reasons I guess)
+	# So at least I had "./" in RPATH and found link paths
+	#set(CMAKE_SKIP_RPATH TRUE)
+	#SET(CMAKE_SKIP_BUILD_RPATH  FALSE)
+	SET(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
+
+	SET(CMAKE_INSTALL_RPATH "$ORIGIN")
+	#SET(CMAKE_INSTALL_RPATH "./")
+	#SET(CMAKE_INSTALL_RPATH "./:/usr/lib64/:/usr/lib/:/usr/local/lib64/:/usr/local/lib/") # This one causes be a problem -> a "default" version of libGL (swrast) is located in /usr/lib64 and was selected instead of nvidia one (in /usr/lib64/nividia)
+
+	SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
+endif()
+
+
+set(SIBR_PROGRAMARGS "" CACHE STRING "Default program arguments used in Visual Studio target properties")
+if ("${SIBR_PROGRAMARGS}" STREQUAL "")
+  if (DEFINED ENV{SIBR_PROGRAMARGS})
+    set(SIBR_PROGRAMARGS "$ENV{SIBR_PROGRAMARGS}" CACHE STRING "Default program arguments used in Visual Studio target properties" FORCE)
+    message( STATUS "Using program options found in environment variable 'SIBR_PROGRAMARGS' => '${SIBR_PROGRAMARGS}'")
+  else()
+    message(
+      "Note you can provide default program options for Visual Studio target properties by either setting"
+      " a value for the cmake cached variable 'SIBR_PROGRAMARGS' or by setting a new environment "
+      "variable 'SIBR_PROGRAMARGS'")
+  endif()
+endif()
+
+add_custom_target(PREBUILD ALL)
+
+## Include all projects
+set(SIBR_PROJECTS_SAMPLES_SUBPAGE_REF "")
+set(SIBR_PROJECTS_OURS_SUBPAGE_REF "")
+set(SIBR_PROJECTS_TOOLBOX_SUBPAGE_REF "")
+set(SIBR_PROJECTS_OTHERS_SUBPAGE_REF "")
+set(SIBR_PROJECTS_SAMPLES_REF_REF "")
+set(SIBR_PROJECTS_OURS_REF_REF "")
+set(SIBR_PROJECTS_TOOLBOX_REF_REF "")
+set(SIBR_PROJECTS_OTHERS_REF_REF "")
+set(DOXY_APP_SPECIFIC_IMG_PATH "")
+set(DOXY_DOC_EXCLUDE_PATTERNS_DIRS "")
+ADD_SUBDIRECTORY(src)
+
+
+## handle documentation
+if (WIN32)
+ADD_SUBDIRECTORY(docs)
+endif()