Production and Virtual transactions were down
Incident Report for Deriv
Resolved
The DevOps team identified that there was a drop in the transaction in both demo and real for deriv.com and binary.com. Post troubleshooting it was found that the real transaction recovered however the virtual was not. It was observed that the pricer queue was overflowing for virtual only. DevOps team flushed the pricer redis and restarted all services on pricer redis, pricer queue and pricer along with websocket api aon api servers which resolved the issue on few demo products however the transactions were still down for Bot and DerivGo. On further investigation it was found that the feed client service on the RPC servers was connected, however it was receiving no data, but both sides showed that the connection was in place and hence the service was restarted on all the RPC servers, which fixed the issue.
Real transaction on binary/deriv/Dbot/DerivGo = 10 mins (22:08 to 22:18)
Demo transaction on binary/deriv/Dbot/DerivGo = 5 hours 32 mins (22:08 to 3:40)
Posted May 18, 2022 - 14:00 GMT