Bug reporting

our kubernetes cluster stopped taking commands due to etcd3 running out of space and alarming about NOSPACE.

endpoint status, as json:
[{"Endpoint":"coreos01.kub.modio.se:2379","Status":{"header":{"cluster_id":1416696078359407448,"member_id":5715034692686022789,"revision":2190654,"raft_term":5829},"version":"3.0.10","dbSize":952512512,"leader":12583913577278567278,"raftIndex":5817555,"raftTerm":5829}},{"Endpoint":"coreos01.kub.do.modio.se:2379","Status":{"header":{"cluster_id":1416696078359407448,"member_id":9911998787147328850,"revision":2190654,"raft_term":5829},"version":"3.0.10","dbSize":356352,"leader":12583913577278567278,"raftIndex":5817555,"raftTerm":5829}},{"Endpoint":"coreos03.kub.do.modio.se:2379","Status":{"header":{"cluster_id":1416696078359407448,"member_id":12583913577278567278,"revision":2190654,"raft_term":5829},"version":"3.0.10","dbSize":3777458176,"leader":12583913577278567278,"raftIndex":5817555,"raftTerm":5829}}]

After that we did a compaction:

compaction 2190654
compacted revision 2190654

after compact, before defrag:

coreos01.kub.modio.se:2379, 4f4fe1c600958885, 3.0.10, 952 MB, false, 5829, 5817819
coreos01.kub.do.modio.se:2379, 898e7e618204ed52, 3.0.10, 369 kB, false, 5829, 5817819
coreos03.kub.do.modio.se:2379, aea30ab9e1e7736e, 3.0.10, 3.8 GB, true, 5829, 5817821

after compact, after defrag:

coreos01.kub.modio.se:2379, 4f4fe1c600958885, 3.0.10, 299 kB, false, 5829, 5817955
coreos01.kub.do.modio.se:2379, 898e7e618204ed52, 3.0.10, 299 kB, false, 5829, 5817956
coreos03.kub.do.modio.se:2379, aea30ab9e1e7736e, 3.0.10, 299 kB, true, 5829, 5817957

And after clearing the alarms, we could use the kubernetes cluster again.