DAOS-17446 common: Chunk dRPC messages larger than 1MB #16479
Conversation
In certain contexts it is possible to exceed the 1MB buffer size limitation for dRPC messages. This is rare, but it renders the payload impossible to read. This patch refactors dRPC to send and receive messages in <1MB chunks by adding a header. With this change, the chunk header is always expected, so client libdaos and daos_agent must be at equivalent versions. This should already be the case with the daos-client RPM package. - Implement chunking as the default behavior in both control and data plane dRPC implementations. - Add low-level integration testing of the dRPC call/response infrastructure for both C and Go implementations, to test the send/recv functionality in each implementation separately from higher-level DAOS tests. Features: control Signed-off-by: Kris Jacque <kris.jacque@hpe.com>
Signed-off-by: Kris Jacque <kris.jacque@hpe.com>
|
Currently based on #16350. When that lands I will have to rebase, so just seeking preliminary reviews at this point. |
|
Test stage Unit Test on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16479/1/execution/node/664/log |
|
Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16479/1/testReport/ |
|
Took a look at the new code... Looks like a nice update. One thing that occurred to me as I was looking at the Go side of it is that it might be better to make a libdrpc and write Go bindings for it. That way we don't have two separate implementations in different languages that have to interoperate. I know that's quite a shift in direction, and you've already written the guts of the Go side. Would kinda suck to throw that out, but might still be useful as a framework for bindings. If we're going to do that, though, now would be a good time to do it, IMO. |
|
Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16479/1/execution/node/892/log |
In certain contexts it is possible to exceed the 1MB buffer size
limitation for dRPC messages. This is rare, but it renders the
payload impossible to read. This patch refactors dRPC to send
and receive messages in <1MB chunks by adding a header.
With this change, the chunk header is always expected, so client
libdaos and daos_agent must be at equivalent versions. This should
already be the case with the daos-client RPM package.
data plane dRPC implementations.
infrastructure for both C and Go implementations, to test the
send/recv functionality in each implementation separately from
higher-level DAOS tests.
Features: control
Signed-off-by: Kris Jacque kris.jacque@hpe.com
Steps for the author:
After all prior steps are complete: