checkpoint/restore of /dev/shm in containers #2674
Description
I am not sure if this is scope of runc or criu but I just want to check if there is a way to include the files within tmpfs (ex. /dev/shm) within the snapshot of container?
I am currently working with applications that is actively utilizing /dev/shm. During checkpoint, everything is fine but it is causing concern during restore, since tmpfs volumes are mounted empty by default during initialization.
During checkpoint:
(00.112009) vma 7289949cc000 borrows vfi from previous 7289949cb000
(00.112019) Handling VMA with the following smaps entry: 7289949cf000-7289949d0000 rw-s 00000000 00:1a 26 /dev/shm/pym-43-q4izhbog (deleted)
(00.112023) Found regular file mapping, OK
(00.112031) Dumping path for -3 fd via self 15 [/dev/shm/pym-43-q4izhbog (deleted)]
(00.112033) Strip ' (deleted)' tag from './dev/shm/pym-43-q4izhbog (deleted)'
(00.112034) Dumping ghost file for fd 15 id 0x96
(00.112035) mnt: Path `/dev/shm/pym-43-q4izhbog' resolved to `./dev/shm' mountpoint
(00.112037) Dumping ghost file contents (id 0x1)
(00.112080) Only file size could be stored for validation for file /dev/shm/pym-43-q4izhbog
During restore:
(00.407305) 95: Restoring size 0x10000 for 0x7db21
(00.407306) 1: Opening 0x007bb55ffde000-0x007bb55ffe8000 0x0000000002c000 (20000041) vma
(00.407309) 43: 4885: Link dev/shm/pym-43-q4izhbog.cr.1.ghost -> dev/shm/pym-43-q4izhbog
(00.407312) 1: Opening 0x007bb55ffe8000-0x007bb55ffea000 0x00000000036000 (41) vma
(00.407312) 95: Send fd 10 to /crtools-fd-95-571179c1-2c99-4e2e-a2e9-34686deabc01
(00.407316) 1: Opening 0x007bb55ffea000-0x007bb55ffec000 0x00000000038000 (41) vma
(00.407324) 95: Create fd for 45
(00.407327) 95: Creating pipe pipe_id=0x7db21 id=0x138
(00.407332) 95: Further fle=0x7de8b0bd5540, pid=95
(00.407336) 95: Create fd for 46
(00.407339) 95: epoll: Restore eventpoll: id 0x000139 flags 0x02
(00.407342) 43: File dev/shm/pym-43-q4izhbog could only be validated with file size
(00.407346) 95: Create fd for 47
(00.407347) 43: Unlink: 4885:dev/shm/pym-43-q4izhbog
(00.407351) 95: Creating pipe pipe_id=0x7db22 id=0x13a
(00.407359) 95: Restoring size 0x10000 for 0x7db22
(00.407361) 43: Create fd for 18
------
(00.409543) pie: 94: mmap(0x7cab54732000 -> 0x7cab54733000, 0x1 0x12 4)
(00.409545) 43: Opening 0x00728859426000-0x00728859427000 0x00000000015000 (41) vma
(00.409549) 43: Opening 0x00728859427000-0x00728859428000 0x00000000016000 (41) vma
(00.409551) 43: Opening 0x00728859429000-0x00728859437000 0000000000000000 (20000041) vma
(00.409552) pie: 94: mmap(0x7cab54733000 -> 0x7cab54734000, 0x3 0x12 4)
(00.409560) pie: 94: mmap(0x7cab54734000 -> 0x7cab54735000, 0x3 0x12 4)
(00.410385) mnt: Switching to new ns to clean ghosts
(00.410451) Unlink remap /tmp/.criu.mntns.rTqpuG/mnt-0000004885/pym-43-q4izhbog.cr.1.ghost
(00.410888) Error (criu/cr-restore.c:2324): Restoring FAILED.
(00.412187) Error (criu/cgroup.c:1998): cg: cgroupd: recv req error: No such file or directory
The error is related to ghost file but I assume it has more to do with the handling files within /dev/shm which is considered as external mount.
I initially tested using skip-mnt /dev/shm which resulted in fsnotify and irmap related error, so it looks like it was not the option to use here.
If it is not yet possible to include tmpfs files in snapshot, is it possible to manually manage it with action script? I inspected the scripts/tmp-files.sh and it looks like it was doable.
But my concern on this is that when should we extract the tmpfs files during restore? I assume running it on post-setup-namespaces would be the best place since most of the mounts were already setup like this?
(00.188311) 1: mnt-v2: Move mount 4879 from /tmp/.criu.mntns.rTqpuG/mnt-0000004879 to /tmp/.criu.mntns.rTqpuG/12-0000000000/dev
(00.188325) 1: mnt-v2: Move mount 4883 from /tmp/.criu.mntns.rTqpuG/mnt-0000004883 to /tmp/.criu.mntns.rTqpuG/12-0000000000/dev/mqueue
(00.188340) 1: mnt-v2: Move mount 4884 from /tmp/.criu.mntns.rTqpuG/mnt-0000004884 to /tmp/.criu.mntns.rTqpuG/12-0000000000/dev/pts
(00.188354) 1: mnt-v2: Move mount 4885 from /tmp/.criu.mntns.rTqpuG/mnt-0000004885 to /tmp/.criu.mntns.rTqpuG/12-0000000000/dev/shm
(00.188369) 1: mnt-v2: Move mount 4886 from /tmp/.criu.mntns.rTqpuG/mnt-0000004886 to /tmp/.criu.mntns.rTqpuG/12-0000000000/dev/termination-log
(00.188385) 1: mnt-v2: Move mount 4881 from /tmp/.criu.mntns.rTqpuG/mnt-0000004881 to /tmp/.criu.mntns.rTqpuG/12-0000000000/proc
(00.188402) 1: mnt-v2: Move mount 4219 from /tmp/.criu.mntns.rTqpuG/mnt-0000004219 to /tmp/.criu.mntns.rTqpuG/12-0000000000/proc/bus
(00.188418) 1: mnt-v2: Move mount 4220 from /tmp/.criu.mntns.rTqpuG/mnt-0000004220 to /tmp/.criu.mntns.rTqpuG/12-0000000000/proc/fs
(00.188433) 1: mnt-v2: Move mount 4392 from /tmp/.criu.mntns.rTqpuG/mnt-0000004392 to /tmp/.criu.mntns.rTqpuG/12-0000000000/proc/irq
Is writing those tmpfs files on /tmp/.criu.mntns.rTqpuG/12-0000000000/dev/shm should do the trick?
Thank you very much in advance!