With the new css-position spec and overlay CSS property, it is possible for popovers, dialogs, and their ::backdrops to be in the top layer while transitioning from showing to hidden. During this time, :popover-open and :modal don't apply, but we still want the user-agent styles to apply as if these elements were still showing in the top layer. Otherwise, for example, dialog elements will jump to the top of the page as the exit animation starts.

I tried to fix this by adding an internal pseudo selector to match top layer elements and moving the user agent styles for dialog and popover into it here, but it was suggested to move the discussion to the csswg: whatwg/html#9387

Some possible solutions, some of which were discussed in that HTML PR:

1. Do what my HTML PR did, which is to add an internal pseudo selector to match elements which are in the top layer and move the user-agent styles there. This has the benefit of not requiring the author to add any extra code to make this work.
2. Add an externally exposed pseudo selector to match top layer elements which authors have access to, and also move user agent styles there. This also has the benefit of not requiring the author to add any extra code, and also has the benefit of allowing the author to do other stuff with it and override the user agent styles in a more ergonomic way.
3. Add a shorthand value for transitioning called overlay-dialog and overlay-popover which expands both overlay and other relevant user-agent properties. Instead of using transition: overlay, you would use transition: overlay-dialog and transition: overlay-popover.

I prefer option 1 or 2 since it doesn't require the author to add extra stuff.

@tabatkins @fantasai