Skip to content

Using minidumps from crash reports

Jump to bottom Edit New page
Margen67 edited this page Apr 22, 2021 · 4 revisions

Minidumps and breakpad

With #3236, OpenRCT2 gained another library: breakpad.

Breakpad is a small project whose aim is to handle producing minidumps across all three major systems. As of 2016-07-12 it is only enabled for Windows, as this is the biggest part of our target users and the only one that does not produce any meaningful reports by default.

When a crash happens, breakpad will make sure a minidump is created and users gets notified about it and (if possible) a save of currently loaded park will get created as well.

The upload process is manual and so is decoding said dumps users kindly provide.

For minidumps to work, the original binary and attached symbol file (.pdb) is required. It is not possible to generate .pdb post-factum or re-create binary for which the minidump was done. We provide both of these conveniently packaged as part of our AppVeyor build for each CI run.

Linux

It is possible to extract at least some information from the provided dumps.

There are some prebuilt versions of breakpad and its tools available from https://android.googlesource.com/platform/prebuilts/google-breakpad/linux-x86/+/studio-master-dev (other platforms are available too, check for most recent branch) You can download them a single base64-encoded binary by going to a particular file, then clicking txt in lower-right corner.

You will need minidump_stackwalk (part of breakpad) and dump_syms.exe (windows version of it, part of breakpad), besides the debug symbols and the dump itself.

You can download dump_syms.exe from https://hg.mozilla.org/mozilla-central/raw-file/94c926911767/toolkit/crashreporter/tools/win32/dump_syms_vc1600.exe, it will require msdia100.dll, which is a part of Microsoft Visual C++ 2010 Redistributable, which you can obtain with winetricks or (probably) from here: https://www.microsoft.com/en-us/download/details.aspx?id=26999

For minidump_stackwalk either compile breakpad manually or use a package provided for your distribution. For Arch see https://aur.archlinux.org/packages/google-breakpad-git/.

Let's take an example dump from here: https://github.com/OpenRCT2/OpenRCT2/issues/4062#issuecomment-232133914. You will see in its filename it was done for build on commit 6c702fb. You can download debug symbols from here: https://ci.appveyor.com/project/IntelOrca/openrct2-ject9/build/0.0.5.3787/artifacts

$ wine dump_syms_vc1600.exe openrct2.pdb > openrct2.sym
$ # you can check where to place symbols with:
$ minidump_stackwalk 2434566d-15dd-41e7-861e-6f589d7979db.6c702fb.dmp ./symbols 2>&1 | grep symbols
2016-07-12 22:52:21: simple_symbol_supplier.cc:196: INFO: No symbol file at ./symbols/openrct2.pdb/9F28AE8D39D2495F958890B66BA3C2A11/openrct2.sym
$ head -n1 openrct2.sym
MODULE windows x86 9F28AE8D39D2495F958890B66BA3C2A11 openrct2.pdb
$ mkdir -p ./symbols/openrct2.pdb/9F28AE8D39D2495F958890B66BA3C2A11/
$ mv openrct2.sym ./symbols/openrct2.pdb/9F28AE8D39D2495F958890B66BA3C2A11/
$ # minidump_stackwalk should now be able to produce stacktrace, following output truncated:
$ minidump_stackwalk 2434566d-15dd-41e7-861e-6f589d7979db.6c702fb.dmp ./symbols
Thread 0 (crashed)
 0  openrct2.dll!memcpy [memcpy.asm : 319 + 0x0]
    eip = 0x65d7a990   esp = 0x000cf6d8   ebp = 0x000cf724   ebx = 0x00000000
    esi = 0x0d7c00e4   edi = 0x00000000   eax = 0x0d7c00e4   ecx = 0x00004d57
    edx = 0x0001355c   efl = 0x00210202
    Found by: given as instruction pointer in context
 1  openrct2.dll!MemoryStream::Read(void *,unsigned __int64) [memorystream.cpp : 148 + 0x10]
    eip = 0x65a78407   esp = 0x000cf6e4   ebp = 0x000cf724
    Found by: call frame info
 2  openrct2.dll!MemoryStream::TryRead(void *,unsigned __int64) [memorystream.cpp : 156 + 0x10]
    eip = 0x65a784a2   esp = 0x000cf72c   ebp = 0x000cf754
    Found by: call frame info
 3  openrct2.dll!ImageTable::Read(IReadObjectContext *,IStream *) [imagetable.cpp : 68 + 0xd]
    eip = 0x65aaf08f   esp = 0x000cf75c   ebp = 0x000cf7e0
    Found by: call frame info
 4  openrct2.dll!SmallSceneryObject::ReadLegacy(IReadObjectContext *,IStream *) [smallsceneryobject.cpp : 62 + 0xc]
    eip = 0x65ab4c1b   esp = 0x000cf7e8   ebp = 0x000cf814
    Found by: call frame info
 5  openrct2.dll!ObjectFactory::ReadObjectLegacy(Object *,IReadObjectContext *,IStream *) [objectfactory.cpp : 92 + 0x13]
    eip = 0x65ab0254   esp = 0x000cf81c   ebp = 0x000cf88c
    Found by: call frame info
 6  openrct2.dll!ObjectFactory::CreateObjectFromLegacyFile(char const *) [objectfactory.cpp : 149 + 0xb]
    eip = 0x65ab041d   esp = 0x000cf894   ebp = 0x000cf8e0
    Found by: call frame info
 7  openrct2.dll!ObjectRepository::ScanObject(char const *) [objectrepository.cpp : 291 + 0xf]
    eip = 0x65ab1eaa   esp = 0x000cf8e8   ebp = 0x000cf924
    Found by: call frame info
 8  openrct2.dll!ObjectRepository::AddObject(rct_object_entry const *,void const *,unsigned int) [objectrepository.cpp : 213 + 0x12]
    eip = 0x65ab19a9   esp = 0x000cf92c   ebp = 0x000cfa8c
    Found by: call frame info
 9  openrct2.dll!object_load_packed [objectrepository.cpp : 808 + 0xe]
    eip = 0x65ab3081   esp = 0x000cfa94   ebp = 0x00013620
    Found by: call frame info
10  openrct2.dll!_close_nolock [close.cpp : 70 + 0x35]
    eip = 0x65d91300   esp = 0x000cfab8   ebp = 0x000cfac8
    Found by: stack scanning
11  openrct2.dll!S6Importer::LoadSavedGame(SDL_RWops *) [s6importer.cpp : 119 + 0x7]
    eip = 0x65ade8db   esp = 0x000cfad0   ebp = 0x000cfb04
    Found by: call frame info
12  openrct2.dll!game_load_network [s6importer.cpp : 470 + 0x8]
    eip = 0x65adfb0c   esp = 0x000cfb0c   ebp = 0x000cfb3c
    Found by: call frame info
13  openrct2.dll!Network::Client_Handle_MAP(NetworkConnection &,NetworkPacket &) [network.cpp : 1622 + 0x7]
    eip = 0x65aa3f83   esp = 0x000cfb44   ebp = 0x65f7f508
    Found by: call frame info
14  openrct2.dll!Network::ProcessPacket(NetworkConnection &,NetworkPacket &) [network.cpp : 1249 + 0xc]
    eip = 0x65aa2abc   esp = 0x000cfc74   ebp = 0x65f7f508
    Found by: call frame info
15  openrct2.dll!Network::ProcessConnection(NetworkConnection &) [network.cpp : 1211 + 0xa]
    eip = 0x65aa29be   esp = 0x000cfc88   ebp = 0x65f7f508
    Found by: call frame info
16  openrct2.dll!Network::UpdateClient() [network.cpp : 479 + 0xa]
    eip = 0x65a9ebb7   esp = 0x000cfda4   ebp = 0x65f7f508
    Found by: call frame info
17  openrct2.dll!openrct2_loop [openrct2.c : 439 + 0x5]
    eip = 0x65a6dba0   esp = 0x000cfec0   ebp = 0x65f7f508
    Found by: call frame info
18  openrct2.exe + 0xf572bc
    eip = 0x013572bc   esp = 0x000cfedc   ebp = 0x000cfee0
    Found by: stack scanning
19  openrct2.dll!openrct2_launch [openrct2.c : 324 + 0x5]
    eip = 0x65a6d7e3   esp = 0x000cfee8   ebp = 0x000cff88
    Found by: previous frame's frame pointer
20  openrct2.dll!StartOpenRCT [windows.c : 106 + 0x5]
    eip = 0x65ad6ded   esp = 0x000cfeec   ebp = 0x000cff88
    Found by: call frame info
21  openrct2.exe + 0x69e4
    eip = 0x004069e4   esp = 0x000cfef0   ebp = 0x000cff88
    Found by: call frame info
22  kernel32.dll + 0x1338a
    eip = 0x76a1338a   esp = 0x000cff90   ebp = 0x000cff94
    Found by: previous frame's frame pointer
23  ntdll.dll + 0x39902
    eip = 0x77c59902   esp = 0x000cff9c   ebp = 0x000cffd4
    Found by: previous frame's frame pointer
24  ntdll.dll + 0x398d5
    eip = 0x77c598d5   esp = 0x000cffdc   ebp = 0x000cffec
    Found by: previous frame's frame pointer
Add a custom footer
Clone this wiki locally