Description
I am struggling to create a feature field which looks like the one provided in the README.
I get the following visualisation, which looks nothing like the example image:
These are the commands I ran to produce this feature field using scene 1:
Step 1: Train model according to step 2 in the documentation:
python train_emernerf.py \
--config_file configs/default_config.yaml \
--output_root output \
--project test_dino \
--run_name 1_flow \
data.pixel_source.load_features=True \
data.pixel_source.feature_model_type=dinov2_vitb14 \
data.pixel_source.skip_feature_extraction=False \
nerf.model.head.enable_feature_head=True \
nerf.model.head.enable_learnable_pe=True \
data.scene_idx=1 \
data.start_timestep=2 \
data.end_timestep=3 \
logging.saveckpt_freq=8000 \
optim.num_iters=8000
This gives the following metrics:
I20240327 12:56:14 root video_utils.py:93] Eval over 6 images:
I20240327 12:56:14 root video_utils.py:94] PSNR: 36.4330
I20240327 12:56:14 root video_utils.py:95] SSIM: 0.9412
I20240327 12:56:14 root video_utils.py:96] Feature PSNR: 24.9425
I20240327 12:56:14 root video_utils.py:97] Masked PSNR: 34.0290
I20240327 12:56:14 root video_utils.py:98] Masked SSIM: 0.9251
I20240327 12:56:14 root video_utils.py:99] Masked Feature PSNR: 23.2345
The generated rgbs and dino_features are the following:
Step 2: Add the --visualize_voxel and resume_from flags and remove the iterations and save checkpoint frequency (since I don't want to train again):
python train_emernerf.py \
--config_file configs/default_config.yaml \
--output_root output \
--project test_dino \
--run_name 1_flow \
--visualize_voxel \
resume_from=output/test_dino/1_flow/checkpoint_08000.pth \
data.pixel_source.load_features=True \
data.pixel_source.feature_model_type=dinov2_vitb14 \
data.pixel_source.skip_feature_extraction=False \
nerf.model.head.enable_feature_head=True \
nerf.model.head.enable_learnable_pe=True \
data.scene_idx=1 \
data.start_timestep=2 \
data.end_timestep=3
This produces the feature field shown above.
Does anyone know what could be causing this issue? I am currently attempting to train with a much longer timestep but other than that I do not see any glaring errors in my procedure.
Any help would be greatly appreciated. I am happy to provide further information.