Open
Description
docker daemon unable to start conatinerd . I see the following trace .
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: panic: freepages: failed to get all reachable pages (key[0]=(hex)35623764373364343838373233636333396537396261656365323161333731386339376437663133303033316261366566643762613837613564636633343438 on leaf page(2) needs to be < than key of the next element in ancestor (hex)30346135303461633136653430646331666166646331666666633065386133353030663139396662653836656561663961383830323938333734333964343366. Pages stack: [11 2])
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 71 gp=0xc000503880 m=12 mp=0xc000601808 [running]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: panic({0x5611da793340?, 0xc000445c40?})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/panic.go:779 +0x158 fp=0xc00049bf88 sp=0xc00049bed8 pc=0x5611d8f551f8
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*DB).freepages.func2()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:1202 +0x8d fp=0xc00049bfe0 sp=0xc00049bf88 pc=0x5611d97a2e4d
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by go.etcd.io/bbolt.(*DB).freepages in goroutine 48
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:1200 +0x1e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 1 gp=0xc0000061c0 m=nil [select]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc00059f770?, 0x2?, 0x60?, 0x0?, 0xc00059f59c?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000787438 sp=0xc000787418 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc000787770, 0xc00059f598, 0x5611da1e1efc?, 0x0, 0xc00005fa90?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000787558 sp=0xc000787438 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/cmd/containerd/command.App.func1(0xc00062a000)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/cmd/containerd/command/main.go:215 +0x8e5 fp=0xc000787900 sp=0xc000787558 pc=0x5611d9793085
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/urfave/cli.HandleAction({0x5611da7cd4c0?, 0x5611daaf57a8?}, 0xc0000fa700?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/github.com/urfave/cli/app.go:524 +0x50 fp=0xc000787918 sp=0xc000787900 pc=0x5611d976d7b0
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/urfave/cli.(*App).Run(0xc0000fa700, {0xc0001102d0, 0x3, 0x3})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/github.com/urfave/cli/app.go:286 +0x79b fp=0xc000787ef8 sp=0xc000787918 pc=0x5611d976ae1b
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: main.main()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/cmd/containerd/main.go:39 +0x2d fp=0xc000787f50 sp=0xc000787ef8 pc=0x5611da1d80ad
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.main()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:271 +0x29d fp=0xc000787fe0 sp=0xc000787f50 pc=0x5611d8f5951d
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000787fe8 sp=0xc000787fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000098fa8 sp=0xc000098f88 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goparkunlock(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:408
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.forcegchelper()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000098fe0 sp=0xc000098fa8 pc=0x5611d8f597d8
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000098fe8 sp=0xc000098fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.init.6 in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:314 +0x1a
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000099780 sp=0xc000099760 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goparkunlock(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:408
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.bgsweep(0xc0000c2000)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgcsweep.go:318 +0xdf fp=0xc0000997c8 sp=0xc000099780 pc=0x5611d8f410ff
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcenable.gowrap1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc0000997e0 sp=0xc0000997c8 pc=0x5611d8f357e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000997e8 sp=0xc0000997e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcenable in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:203 +0x66
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x10000?, 0x5611da47b840?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000099f78 sp=0xc000099f58 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goparkunlock(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:408
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.(*scavengerState).park(0x5611db90d0c0)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000099fa8 sp=0xc000099f78 pc=0x5611d8f3e8e9
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.bgscavenge(0xc0000c2000)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000099fc8 sp=0xc000099fa8 pc=0x5611d8f3eeb9
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcenable.gowrap2()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000099fe0 sp=0xc000099fc8 pc=0x5611d8f35785
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcenable in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:204 +0xa5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 18 gp=0xc000104380 m=nil [finalizer wait]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc000098648?, 0x5611d8f27a25?, 0xa8?, 0x1?, 0xc0000061c0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000098620 sp=0xc000098600 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.runfinq()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc0000987e0 sp=0xc000098620 pc=0x5611d8f347a7
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000987e8 sp=0xc0000987e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.createfing in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mfinal.go:164 +0x3d
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 21 gp=0xc0003be540 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc0000947a8?, 0x5611d8f1f931?, 0x60?, 0x60?, 0x1?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000094750 sp=0xc000094730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000947e0 sp=0xc000094750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000947e8 sp=0xc0000947e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 22 gp=0xc0003befc0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000094f50 sp=0xc000094f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000094fe0 sp=0xc000094f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000094fe8 sp=0xc000094fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 23 gp=0xc0003bf180 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000095750 sp=0xc000095730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000957e0 sp=0xc000095750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000957e8 sp=0xc0000957e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 24 gp=0xc0003bf340 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000095f50 sp=0xc000095f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000095fe0 sp=0xc000095f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 6 gp=0xc000007c00 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009a750 sp=0xc00009a730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00009a7e0 sp=0xc00009a750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009a7e8 sp=0xc00009a7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 7 gp=0xc000007dc0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009af50 sp=0xc00009af30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00009afe0 sp=0xc00009af50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009afe8 sp=0xc00009afe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 8 gp=0xc0000fa000 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009b750 sp=0xc00009b730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00009b7e0 sp=0xc00009b750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009b7e8 sp=0xc00009b7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 9 gp=0xc0000fa1c0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00009bf50 sp=0xc00009bf30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00009bfe0 sp=0xc00009bf50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009bfe8 sp=0xc00009bfe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 35 gp=0xc00041c000 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000422750 sp=0xc000422730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0004227e0 sp=0xc000422750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004227e8 sp=0xc0004227e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 36 gp=0xc00041c1c0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000422f50 sp=0xc000422f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000422fe0 sp=0xc000422f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000422fe8 sp=0xc000422fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 25 gp=0xc0003bf500 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb11c66?, 0x3?, 0xb5?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000096750 sp=0xc000096730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000967e0 sp=0xc000096750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000967e8 sp=0xc0000967e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 10 gp=0xc0000fa380 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb1567c?, 0x1?, 0xd8?, 0x8b?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00041e750 sp=0xc00041e730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00041e7e0 sp=0xc00041e750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00041e7e8 sp=0xc00041e7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 37 gp=0xc00041c380 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb14f1a?, 0x3?, 0xb2?, 0xa6?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000423750 sp=0xc000423730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0004237e0 sp=0xc000423750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004237e8 sp=0xc0004237e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 38 gp=0xc00041c540 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb0d95e?, 0x0?, 0x0?, 0x0?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000423f50 sp=0xc000423f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000423fe0 sp=0xc000423f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000423fe8 sp=0xc000423fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 26 gp=0xc0003bf6c0 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x5611db972420?, 0x1?, 0x45?, 0x7a?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000096f50 sp=0xc000096f30 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000096fe0 sp=0xc000096f50 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000096fe8 sp=0xc000096fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 39 gp=0xc00041c700 m=nil [GC worker (idle)]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xab5aeb0db19?, 0x1?, 0xd2?, 0xa4?, 0x0?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000424750 sp=0xc000424730 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gcBgMarkWorker()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0004247e0 sp=0xc000424750 pc=0x5611d8f378e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004247e8 sp=0xc0004247e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 48 gp=0xc000602380 m=nil [chan send]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0x6?, 0x5611d8f751eb?, 0x80?, 0x0?, 0x16e?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000062a60 sp=0xc000062a40 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.chansend(0xc000017380, 0xc000062b58, 0x1, 0x5611d8f27601?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/chan.go:259 +0x38d fp=0xc000062ad0 sp=0xc000062a60 pc=0x5611d8f1f0cd
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.chansend1(0x5611da27261b?, 0x6e?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/chan.go:145 +0x17 fp=0xc000062b00 sp=0xc000062ad0 pc=0x5611d8f1ed37
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.verifyKeyOrder(0x2, {0x5611da1dc67b, 0x4}, 0x1, {0x7fabd83fb210, 0x40, 0x40}, {0x7fabd83fb1c0, 0x40, 0x40}, ...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:188 +0x96d fp=0xc000062be8 sp=0xc000062b00 pc=0x5611d97afaed
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).recursivelyCheckPagesInternal(0xc00001a000, 0x2, {0x7fabd8404040, 0x40, 0x40}, {0x7fabd8404080, 0x40, 0x40}, {0xc0004143e8, 0x1, ...}, ...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:155 +0x465 fp=0xc000062d30 sp=0xc000062be8 pc=0x5611d97aecc5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).recursivelyCheckPagesInternal(0xc00001a000, 0xb, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:147 +0x59e fp=0xc000062e78 sp=0xc000062d30 pc=0x5611d97aedfe
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).recursivelyCheckPages(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:121
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).checkBucket(0xc00001a000, 0xc000291780, 0xc000063310, 0xc000063250, {0x5611dab0f268, 0x5611db96ffc0}, 0xc000017380)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:105 +0x1a5 fp=0xc000062f88 sp=0xc000062e78 pc=0x5611d97ae0e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).checkBucket.func2({0x7fabd8407030?, 0xc000018d20?, 0xc0004143e0?})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:110 +0x90 fp=0xc000063000 sp=0xc000062f88 pc=0x5611d97ae290
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Bucket).ForEachBucket(0x0?, 0xc000063108)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/bucket.go:411 +0x96 fp=0xc000063048 sp=0xc000063000 pc=0x5611d979b596
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*Tx).checkBucket(0xc00001a000, 0xc00001a018, 0xc000063310, 0xc000063250, {0x5611dab0f268, 0x5611db96ffc0}, 0xc000017380)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/tx_check.go:108 +0x255 fp=0xc000063158 sp=0xc000063048 pc=0x5611d97ae195
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*DB).freepages(0xc00048f8c8)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:1205 +0x225 fp=0xc000063370 sp=0xc000063158 pc=0x5611d97a2c45
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*DB).loadFreelist.func1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:417 +0xc5 fp=0xc0000633a0 sp=0xc000063370 pc=0x5611d979f605
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: sync.(*Once).doSlow(0x5611d979fd20?, 0xc00048fa90?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/sync/once.go:74 +0xc2 fp=0xc000063400 sp=0xc0000633a0 pc=0x5611d8fa3162
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: sync.(*Once).Do(...)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/sync/once.go:65
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.(*DB).loadFreelist(0xc00048f8c8?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:413 +0x45 fp=0xc000063430 sp=0xc000063400 pc=0x5611d979f525
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: go.etcd.io/bbolt.Open({0xc000018c80, 0x4f}, 0x1a4, 0xc00034f5a0)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/go.etcd.io/bbolt/db.go:295 +0x430 fp=0xc0000634f8 sp=0xc000063430 pc=0x5611d979ef50
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/metadata/plugin.init.1.func1(0xc0003ad280)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/metadata/plugin/plugin.go:159 +0x728 fp=0xc000063850 sp=0xc0000634f8 pc=0x5611d9813548
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/plugin.(*Registration).Init(0xc0004005a0, 0xc0003ad280)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/plugin/plugin.go:132 +0x28 fp=0xc000063890 sp=0xc000063850 pc=0x5611d9665228
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/services/server.New({0x5611dab1c4b8, 0xc0004c8d70}, 0xc0000fa8c0)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/services/server/server.go:251 +0x1645 fp=0xc000063ef8 sp=0xc000063890 pc=0x5611d9764265
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/cmd/containerd/command.App.func1.1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/cmd/containerd/command/main.go:194 +0x85 fp=0xc000063fe0 sp=0xc000063ef8 pc=0x5611d9793b45
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by github.com/containerd/containerd/cmd/containerd/command.App.func1 in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/cmd/containerd/command/main.go:191 +0x85b
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 68 gp=0xc0000fa540 m=nil [select]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc0000a8c78?, 0x4?, 0x50?, 0x0?, 0xc0000a8a70?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0000a8900 sp=0xc0000a88e0 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc0000a8c78, 0xc0000a8a68, 0xc000111860?, 0x0, 0xc00018c3c0?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc0000a8a20 sp=0xc0000a8900 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/docker/go-events.(*Broadcaster).run(0xc0004103c0)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/github.com/docker/go-events/broadcast.go:117 +0x14a fp=0xc0000a8fc8 sp=0xc0000a8a20 pc=0x5611d965e92a
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/docker/go-events.NewBroadcaster.gowrap1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/github.com/docker/go-events/broadcast.go:39 +0x25 fp=0xc0000a8fe0 sp=0xc0000a8fc8 pc=0x5611d965e1a5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000a8fe8 sp=0xc0000a8fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by github.com/docker/go-events.NewBroadcaster in goroutine 48
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/vendor/github.com/docker/go-events/broadcast.go:39 +0x1b9
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 54 gp=0xc0000fafc0 m=nil [select]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc00061c7a0?, 0x2?, 0x0?, 0x0?, 0xc00061c72c?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00061c5d8 sp=0xc00061c5b8 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc00061c7a0, 0xc00061c728, 0x0?, 0x0, 0x0?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00061c6f8 sp=0xc00061c5d8 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/cmd/containerd/command.handleSignals.func1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/cmd/containerd/command/main_unix.go:43 +0xe5 fp=0xc00061c7e0 sp=0xc00061c6f8 pc=0x5611d97912a5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00061c7e8 sp=0xc00061c7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by github.com/containerd/containerd/cmd/containerd/command.handleSignals in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/cmd/containerd/command/main_unix.go:40 +0xd3
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 55 gp=0xc0000fb180 m=nil [select, locked to thread]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc00061cfa8?, 0x2?, 0x9?, 0x9c?, 0xc00061cf94?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00061ce38 sp=0xc00061ce18 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc00061cfa8, 0xc00061cf90, 0x0?, 0x0, 0x0?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00061cf58 sp=0xc00061ce38 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.ensureSigM.func1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/signal_unix.go:1034 +0x1a5 fp=0xc00061cfe0 sp=0xc00061cf58 pc=0x5611d8f894e5
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00061cfe8 sp=0xc00061cfe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by runtime.ensureSigM in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/signal_unix.go:1017 +0xc8
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 56 gp=0xc0000fb340 m=3 mp=0xc00009f008 [syscall]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.notetsleepg(0x5611db971740, 0xffffffffffffffff)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/lock_futex.go:246 +0x29 fp=0xc00061d7a0 sp=0xc00061d778 pc=0x5611d8f26049
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: os/signal.signal_recv()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc00061d7c0 sp=0xc00061d7a0 pc=0x5611d8f8f749
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: os/signal.loop()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc00061d7e0 sp=0xc00061d7c0 pc=0x5611d9782233
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00061d7e8 sp=0xc00061d7e0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by os/signal.Notify.func1.1 in goroutine 1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/os/signal/signal.go:151 +0x1f
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: goroutine 70 gp=0xc0005036c0 m=nil [select]:
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.gopark(0xc000420f88?, 0x2?, 0x40?, 0xe?, 0xc000420f74?)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000420e18 sp=0xc000420df8 pc=0x5611d8f5996e
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.selectgo(0xc000420f88, 0xc000420f70, 0x0?, 0x0, 0x0?, 0x1)
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000420f38 sp=0xc000420e18 pc=0x5611d8f6bf85
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: github.com/containerd/containerd/metadata/plugin.init.1.func1.1()
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/metadata/plugin/plugin.go:152 +0xaf fp=0xc000420fe0 sp=0xc000420f38 pc=0x5611d981394f
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: runtime.goexit({})
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000420fe8 sp=0xc000420fe0 pc=0x5611d8f933e1
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: created by github.com/containerd/containerd/metadata/plugin.init.1.func1 in goroutine 48
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28614]: /root/rpmbuild/BUILD/metadata/plugin/plugin.go:149 +0x709
Jan 10 10:07:42 onxl7532.ott.ciena.com dockerd[28519]: time="2025-01-10T10:07:42.850030681Z" level=error msg="containerd did not exit successfully" error="signal: aborted (core dumped)" module=libcontainerd
Jan 10 10:07:48 onxl7532.ott.ciena.com dockerd[28519]: failed to start containerd: timeout waiting for containerd to start
Jan 10 10:07:48 onxl7532.ott.ciena.com systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
I have also tried to recover using bbolt surgery command but it does not help . recovered db has the same issue .
bbolt@latest surgery revert-meta-page meta.db new2.db
The meta page is reverted.
[root@onxl7532 io.containerd.metadata.v1.bolt]# go run go.etcd.io/bbolt/cmd/bbolt@latest check new2.db
panic: freepages: failed to get all reachable pages (key[0]=(hex)35623764373364343838373233636333396537396261656365323161333731386339376437663133303033316261366566643762613837613564636633343438 on leaf page(2) needs to be < than key of the next element in ancestor (hex)30346135303461633136653430646331666166646331666666633065386133353030663139396662653836656561663961383830323938333734333964343366. Pages stack: [11 2])
goroutine 17 [running]:
go.etcd.io/bbolt.(*DB).freepages.func2()
/root/go/pkg/mod/go.etcd.io/bbolt@v1.3.11/db.go:1204 +0x8d
created by go.etcd.io/bbolt.(*DB).freepages in goroutine 1
/root/go/pkg/mod/go.etcd.io/bbolt@v1.3.11/db.go:1202 +0x1c5
exit status 2
version info .
docker info
Client: Docker Engine - Community
Version: 27.3.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.17.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.7
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 115
Running: 114
Paused: 0
Stopped: 1
Images: 176
Server Version: 27.3.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: docker-runc io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc version: v1.1.14-0-g2c9f560
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-206.153.7.1.el8uek.x86_64
Operating System: Oracle Linux Server 8.10
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 93.84GiB
Name: onxl7183.ott.ciena.com
ID: a4c018be-2ba9-4c42-9297-5312ec73af01
Docker Root Dir: /opt/ciena/data/docker
Debug Mode: false
Labels:
host_id=0
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false