Skip to content

Add support for foveation #1195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 11, 2021
Merged

Add support for foveation #1195

merged 3 commits into from
May 11, 2021

Conversation

cabanier
Copy link
Member

@cabanier cabanier commented May 4, 2021

Closes #1194
This PR adds support for foveated rendering as we discussed during today's call.
I basically copied what was in the layers spec. I know @Manishearth wanted a better definition of what the numbers mean but I think it's reasonable to leave as-is.

(I would add reviewer to this PR but I don't seem to have access to that feature.)


Preview | Diff

@cabanier
Copy link
Member Author

cabanier commented May 4, 2021

@Manishearth
Copy link
Contributor

I know @Manishearth wanted a better definition of what the numbers mean but I think it's reasonable to leave as-is.

My concern is more that this does not really define "foveation" in any way, I'd expect more (potentially non normative) text explaining the rough concept.

@cabanier
Copy link
Member Author

cabanier commented May 4, 2021

My concern is more that this does not really define "foveation" in any way, I'd expect more (potentially non normative) text explaining the rough concept.

Would you like to see that as a non-normative note? If not, I don't think the WebXR spec is the correct spot to define foveation.

@Manishearth
Copy link
Contributor

Yes, a note is fine

@cabanier
Copy link
Member Author

cabanier commented May 5, 2021

Yes, a note is fine

I added a note.

@cabanier
Copy link
Member Author

Any thoughts on this @thetuvix @toji ?

Co-authored-by: Brandon Jones <tojiro@gmail.com>
@Manishearth
Copy link
Contributor

Discussed this in a call with @toji.

We both feel that in general the core WebXR spec is very close to CR (we're blocked on a single document being written) and we should not be adding new functionality so late in the process. Rather, new functionality should be developed in existing or new modules. In general, the implementation status of a module is not a sufficient argument for upstreaming a feature.

However, in this case we do feel like it makes sense to merge this. We're reluctant, and don't wish to set too much of a precedent, however in this specific scenario foveation is a feature that's useful outside of layers, and is something frameworks will likely wish to use without having to build a full fallback path where they try to use layers and use the core api if it doesn't exist.

@Manishearth Manishearth merged commit cbd417a into immersive-web:main May 11, 2021
@cabanier
Copy link
Member Author

Thanks @Manishearth and @toji !

@cabanier cabanier deleted the foveation branch May 11, 2021 18:56
@thetuvix
Copy link
Contributor

Just got to this now!

We don't have this sort of knob for HoloLens and Windows Mixed Reality, so we'd probably just implement this as a no-op. From a spec perspective moving forward, though, I'm a little concerned at how loose the definition is of the range between 0 and 1. For example, is it meant to be linear in the ramp from 0 to 1 in terms of FPS impact? How would any given web page know to set 0.7 for this value on a given piece of hardware?

In practice, I think we are likely to see apps detecting specific known headsets and setting known values - perhaps that is OK for the stage where WebXR is now, though it feels like there is a more holistic feature missing here to give apps perf metrics each frame to help them close the loop on setting these values dynamically based on device load. (or even better, having the browser do this kind of feedback loop itself to maintain framerate)

I wonder if we want a more explicit default null value here, even for platforms that support setting this, so that apps can ask the UA to handle it. I suspect we could regret feeling that we must listen to the app's request for a specific fixed foveation value in the future if we try to have our WebXR defaults handle more of this out of the gate.

@cabanier
Copy link
Member Author

cabanier commented May 11, 2021

Just got to this now!

We don't have this sort of knob for HoloLens and Windows Mixed Reality, so we'd probably just implement this as a no-op.

I found this post: https://developer.nvidia.com/blog/nvidia-vrss-2-dynamic-foveated-rendering-no-assembly-required/
Does this not apply to WMR?

From a spec perspective moving forward, though, I'm a little concerned at how loose the definition is of the range between 0 and 1. For example, is it meant to be linear in the ramp from 0 to 1 in terms of FPS impact? How would any given web page know to set 0.7 for this value on a given piece of hardware?

We discussed this in the meeting. The author is in control of this and decided how much foveation is acceptable for their scene. So the UA decides that 100% is the maximum allowable amount of foveation for a given device and the author picks the desired amount.
Right now, clients are turning it on using the Quest specific pref and then brute force it off using some GL commands. This at least gives them more control and allows a solution that works across platforms. There will always be some difference between devices.

In practice, I think we are likely to see apps detecting specific known headsets and setting known values - perhaps that is OK for the stage where WebXR is now, though it feels like there is a more holistic feature missing here to give apps perf metrics each frame to help them close the loop on setting these values dynamically based on device load. (or even better, having the browser do this kind of feedback loop itself to maintain framerate)

I agree. I believe we talked about this in the past but it's not easy to come up with a solution.
The UA could report how long it took for the frame to execute, how long to render, how long it was waiting for a frame, etc.

Quest also has dynamic fixed foveated rendering that decreases the amount if the experience can make frame rate. Even though we're not planning to ship that initially, it is allowed with the current approach.

I wonder if we want a more explicit default null value here, even for platforms that support setting this, so that apps can ask the UA to handle it. I suspect we could regret feeling that we must listen to the app's request for a specific fixed foveation value in the future if we try to have our WebXR defaults handle more of this out of the gate.

This value is a hint so UAs can ignore it and do what they think is best.
For instance, if the user turns on foveation in the flags, we will likely honor that instead of what the author sets at runtime.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Allow foveation in WebXR
4 participants