Using RDMA CM for wire-up #10762
Replies: 2 comments 2 replies
-
|
OpenMPI is using UCX in a mode that does not utilize rdma_cm. It creates the connections directly to the remote RDMA address (QPN/LID/GID) and will use TCP or UD for auxiliary message passing if the main transport is p2p (such as RC). |
Beta Was this translation helpful? Give feedback.
-
|
Ah, makes sense :) I guess my next question is, do you know of any way to get around (what I believe is) the TCP port exhaustion issue when using TCP for wire-up on larger scale tests? For example, during an alltoall (16 node, 192 PPN) test, it brings up around 30k QPs per node then eventually fails with: |
Beta Was this translation helpful? Give feedback.
-
|
Why not use UD for wire-up? |
Beta Was this translation helpful? Give feedback.
-
|
When UD is used for wire-up, it's able to bring up QPs for a while, but then eventually fails with either an assertion error: or an "unhanded timeout" error, or sometimes some "endpoint ID mismatch" error. I'm trying to characterize it a little better and ensure there's nothing wrong on my end before filing a bug report though. That said, the environment that I am running in has somewhat lossy UD and I suspect that it is related to UD drops. With smaller scale tests, it works fine. BTW, I was able to get TCP wire-up to work for 500K+ QPs per node by commenting out the call to |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
Does anyone know how to use RDMA CM for wire-up? I'm running Open MPI+UCX with an RDMA NIC and can't seem to get it to use RDMA CM. By default, setting
UCX_TLS=rcseems to use UD for wire-up. If I setUCX_TLS=^ud,ud:aux, it will use TCP for wire-up.Setting
UCX_SOCKADDR_TLS_PRIORITY=rdmacmdoesn't seem to do it either.Any tips would be appreciated. Thanks!
Beta Was this translation helpful? Give feedback.
All reactions