Issue description

DKG SYNC fails occasionally after initialization

Steps to reproduce the problem

1. update node to 8.1.0-rc.1
2. set otnode sync batch to 50

Specifications

• Node version: 8.1.0-rc.1
• Platform: ubuntu 24.04

Error logs journalctl -u otnode --since="1 hour ago" | grep SYNC

May 22 13:27:12 dkgswarm-old node[606855]: [2025-05-22 17:27:12] INFO: [DKG SYNC] SyncService disabled
May 22 13:28:51 dkgswarm-old node[607381]: [2025-05-22 17:28:51] INFO: [DKG SYNC] Initializing SyncService
May 22 13:28:51 dkgswarm-old node[607381]: [2025-05-22 17:28:51] INFO: [DKG SYNC] Initializing sync service for blockchain otp:20430
May 22 13:28:51 dkgswarm-old node[607381]: [2025-05-22 17:28:51] INFO: [DKG SYNC] Adding missing contracts for blockchain otp:20430: 0xcdb28e93ed340ec10a71bba00a31dbfcf1bd5d37, 0xb4c24fc54bc811c2659c477b65da8648e499fd39
May 22 13:28:51 dkgswarm-old node[607381]: [2025-05-22 17:28:51] DEBUG: [DKG SYNC] Setting up sync mechanism for blockchain otp:20430
May 22 13:28:51 dkgswarm-old node[607381]: [2025-05-22 17:28:51] INFO: [DKG SYNC] Sync mechanism initialized for blockchain otp:20430
May 22 13:28:51 dkgswarm-old node[607381]: [2025-05-22 17:28:51] INFO: [DKG SYNC] SyncService initialization completed
May 22 13:29:03 dkgswarm-old node[607381]: [2025-05-22 17:29:03] DEBUG: [DKG SYNC] Starting sync cycle for blockchain otp:20430
May 22 13:29:03 dkgswarm-old node[607381]: [2025-05-22 17:29:03] DEBUG: [DKG SYNC] Running sync for blockchain otp:20430
May 22 13:29:03 dkgswarm-old node[607381]: [2025-05-22 17:29:03] INFO: [SYNC] No missed KC for retry for blockchain otp:20430
May 22 13:29:03 dkgswarm-old node[607381]: [2025-05-22 17:29:03] INFO: [SYNC] No missed KC for retry for blockchain otp:20430
May 22 13:29:15 dkgswarm-old node[607381]: [2025-05-22 17:29:15] DEBUG: [DKG SYNC] Sync mechanism for otp:20430 still running, skipping this interval
May 22 13:29:20 dkgswarm-old node[607381]: [2025-05-22 17:29:20] ERROR: [DKG SYNC] Error in sync mechanism for otp:20430: [SYNC] Unable to Batch GET Knowledge Collection for blockchain: otp:20430, GET result: {"operationId":"07665a1e-7b02-4697-80e1-6b805d76c4fe","data":null,"status":"BATCH_GET_FIND_SHARD_END","minAcksReached":null,"timestamp":1747934943850,"createdAt":"2025-05-22T17:29:03.000Z","updatedAt":"2025-05-22T17:29:03.000Z"}, stack: Error: [SYNC] Unable to Batch GET Knowledge Collection for blockchain: otp:20430, GET result: {"operationId":"07665a1e-7b02-4697-80e1-6b805d76c4fe","data":null,"status":"BATCH_GET_FIND_SHARD_END","minAcksReached":null,"timestamp":1747934943850,"createdAt":"2025-05-22T17:29:03.000Z","updatedAt":"2025-05-22T17:29:03.000Z"}
May 22 13:29:27 dkgswarm-old node[607381]: [2025-05-22 17:29:27] DEBUG: [DKG SYNC] Starting sync cycle for blockchain otp:20430
May 22 13:29:27 dkgswarm-old node[607381]: [2025-05-22 17:29:27] DEBUG: [DKG SYNC] Running sync for blockchain otp:20430
May 22 13:29:28 dkgswarm-old node[607381]: [2025-05-22 17:29:28] INFO: [DKG SYNC] DKG Sync for blockchain otp:20430 Status: 0%
May 22 13:29:28 dkgswarm-old node[607381]: [2025-05-22 17:29:28] INFO: [SYNC] No missed KC for retry for blockchain otp:20430
May 22 13:29:28 dkgswarm-old node[607381]: [2025-05-22 17:29:28] INFO: [SYNC] No missed KC for retry for blockchain otp:20430
May 22 13:29:39 dkgswarm-old node[607381]: [2025-05-22 17:29:39] DEBUG: [DKG SYNC] Sync mechanism for otp:20430 still running, skipping this interval
May 22 13:29:43 dkgswarm-old node[607381]: [2025-05-22 17:29:43] ERROR: [DKG SYNC] Error in sync mechanism for otp:20430: [SYNC] Unable to Batch GET Knowledge Collection for blockchain: otp:20430, GET result: {"operationId":"cd5bc7d4-5132-45e9-a8f6-6f17fe4796b3","data":null,"status":"BATCH_GET_FIND_SHARD_END","minAcksReached":null,"timestamp":1747934968106,"createdAt":"2025-05-22T17:29:28.000Z","updatedAt":"2025-05-22T17:29:28.000Z"}, stack: Error: [SYNC] Unable to Batch GET Knowledge Collection for blockchain: otp:20430, GET result: {"operationId":"cd5bc7d4-5132-45e9-a8f6-6f17fe4796b3","data":null,"status":"BATCH_GET_FIND_SHARD_END","minAcksReached":null,"timestamp":1747934968106,"createdAt":"2025-05-22T17:29:28.000Z","updatedAt":"2025-05-22T17:29:28.000Z"}
May 22 13:29:51 dkgswarm-old node[607381]: [2025-05-22 17:29:51] DEBUG: [DKG SYNC] Starting sync cycle for blockchain otp:20430
May 22 13:29:51 dkgswarm-old node[607381]: [2025-05-22 17:29:51] DEBUG: [DKG SYNC] Running sync for blockchain otp:20430
May 22 13:29:52 dkgswarm-old node[607381]: [2025-05-22 17:29:52] INFO: [DKG SYNC] DKG Sync for blockchain otp:20430 Status: 0%
May 22 13:29:52 dkgswarm-old node[607381]: [2025-05-22 17:29:52] INFO: [SYNC] No missed KC for retry for blockchain otp:20430
May 22 13:29:52 dkgswarm-old node[607381]: [2025-05-22 17:29:52] INFO: [SYNC] No missed KC for retry for blockchain otp:20430
May 22 13:30:03 dkgswarm-old node[607381]: [2025-05-22 17:30:03] DEBUG: [DKG SYNC] Sync mechanism for otp:20430 still running, skipping this interval
May 22 13:30:10 dkgswarm-old node[607381]: [2025-05-22 17:30:10] ERROR: [DKG SYNC] Error in sync mechanism for otp:20430: [SYNC] Unable to Batch GET Knowledge Collection for blockchain: otp:20430, GET result: {"operationId":"ed475820-532f-4b1d-a3a4-5bb0240c1ffd","data":null,"status":"BATCH_GET_FIND_SHARD_END","minAcksReached":null,"timestamp":1747934992909,"createdAt":"2025-05-22T17:29:52.000Z","updatedAt":"2025-05-22T17:29:52.000Z"}, stack: Error: [SYNC] Unable to Batch GET Knowledge Collection for blockchain: otp:20430, GET result: {"operationId":"ed475820-532f-4b1d-a3a4-5bb0240c1ffd","data":null,"status":"BATCH_GET_FIND_SHARD_END","minAcksReached":null,"timestamp":1747934992909,"createdAt":"2025-05-22T17:29:52.000Z","updatedAt":"2025-05-22T17:29:52.000Z"}
May 22 13:30:15 dkgswarm-old node[607381]: [2025-05-22 17:30:15] DEBUG: [DKG SYNC] Starting sync cycle for blockchain otp:20430
May 22 13:30:15 dkgswarm-old node[607381]: [2025-05-22 17:30:15] DEBUG: [DKG SYNC] Running sync for blockchain otp:20430
May 22 13:30:15 dkgswarm-old node[607381]: [2025-05-22 17:30:15] INFO: [DKG SYNC] DKG Sync for blockchain otp:20430 Status: 0%
May 22 13:30:15 dkgswarm-old node[607381]: [2025-05-22 17:30:15] INFO: [SYNC] No missed KC for retry for blockchain otp:20430
May 22 13:30:15 dkgswarm-old node[607381]: [2025-05-22 17:30:15] INFO: [SYNC] No missed KC for retry for blockchain otp:20430

Disclaimer

Please be aware that the issue reported on a public repository allows everyone to see your node logs, node details, and contact details. If you have any sensitive information, feel free to share it by sending an email to tech@origin-trail.com.