Addressing the Need for Disabled Fallback by Default in ONNX Runtime #6623
thiago4int
started this conversation in
Ideas
Replies: 1 comment
-
Please feel free to create a feature request in the microsoft/onnxruntime repository for the best visibility, thanks! |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I am writing to address a critical concern with ONNX Runtime’s fallback mechanisms, which, while essential for robustness, currently lack the necessary transparency and configurability to meet user expectations in production environments.
The Problem: Silent Fallbacks Impacting Performance and Debugging
ONNX Runtime’s fallback mechanisms—where operations revert to the CPU Execution Provider (EP) or other alternatives when a preferred EP fails—are enabled by default. While this ensures operational robustness, it introduces several issues:
Real-World Impact
In production scenarios, fallback mechanisms have led to:
Example: A user specifies CUDAExecutionProvider but observes degraded performance because ONNX Runtime silently reverts unsupported operations to the CPU EP without logging the transition.
Proposed Solutions
To address these issues, I propose the following actions:
Fallback mechanisms should be disabled by default to prioritize predictable and transparent behavior. Users should explicitly opt in to enable fallback when desired.
Current Approach:
By default, fallback mechanisms are enabled, requiring explicit disabling via the disable_cpu_ep_fallback session option:
Proposed Default Behavior:
Implement the
disable_cpu_ep_fallback = True
option as the default to prevent unexpected fallbacks. Users who require fallback can enable it explicitly.Fallback events should be logged by default with information about:
Example:
[ONNX Runtime Warning] Operation MatMul not supported on CUDAExecutionProvider. Falling back to CPUExecutionProvider.
The official ONNX Runtime documentation should clearly outline:
I am kindly inviting the ONNX community and maintainers to prioritize the following:
Change the Default Behavior: Make fallback mechanisms opt-in rather than opt-out.
Improve Logging: Provide verbose and actionable feedback during fallback events.
Educate the Community: Enhance documentation to ensure users understand how to configure and debug fallback behavior effectively.
These changes will empower ONNX Runtime users to optimize their workflows, reduce debugging time, and maintain confidence in their production environments.
I look forward to collaborating on these enhancements to make ONNX Runtime even more reliable and user-friendly.
Sincerely,
Thiago
Beta Was this translation helpful? Give feedback.
All reactions