Everyone thinks it: "Our process would catch it, our testing is sufficient, our team are top-notch." And then it happens to you. No, I'm not talking about cyberattacks, but CDR outages, non-compliance, and data errors. Recently, the Wych CDR monitoring system has identified some significant issues. Today, I'll discuss three notable cases and how they could have been avoided.
The Unknown Outage
In June, a major bank experienced an unexpected CDR outage. Internally, everything seemed fine—performance metrics were green, alerts were silent, and there were no apparent issues. However, the bank was not accepting new consents.
It’s akin to a restaurant only accepting walk-in orders while ignoring phone and online requests. Naturally, the metrics looked good. Given the nature of CDR and its tightly restricted access, the bank couldn’t simply “ping” the system to check if it was functioning, similar to calling your own number from your phone.
Solution: Ensure there’s an external entity capable of initiating consents and alerting when they fail. The bank lacked this, leading to their offline status. To avoid such issues, a system capable of performing all required actions to verify system effectiveness, with real-time alerting integrated into monitoring, is essential.
Malformed Response
The second issue relates to non-conformant data. After planning a release, making changes, testing, and deploying, everything appeared fine. However, the CDR data set contains hundreds of fields, and considering pagination results and PRD, there are thousands of opportunities for errors. The first sign of a problem was ADRs raising tickets post-release.
Suddenly, the data being delivered was malformed and unreadable. Customers couldn’t apply for loans, loyalty apps indicated the bank was offline, and the bank received yet another “please explain.” With a shortage of knowledgeable CDR-skilled personnel and over 6500 tech vacancies, staying on top of these issues is challenging.
Solution: Integrate a tool into the development pipeline to test PRD and Data Holder solutions before going live. This proactive approach helps ensure data conformity and reliability.
Expired Certificate
The third instance involved a two-month expired certificate in production. The system was supposed to use an updated certificate, but it continued using the old one. Similar to the first scenario, this issue appeared straightforward to test but proved complex due to the CDR’s nature. The bank remained unaware of the problem until ADRs reported connection issues due to the expired, insecure certificate.
Solution: Implement a product verification capability that works in both production and non-production environments. Such a tool should analyse the 2000+ fields for CDR compliance, report specific violations, and reference the spec for immediate action. Real-time monitoring is crucial to alert the team before issues escalate.
Your CDR compliance
To avoid these risks, you need a comprehensive product verification system. This tool should validate CDR data holders and product reference data, ensuring compliance and system integrity. If any of these issues sound familiar, it might be time to consider such a solution for peace of mind and operational reliability.
Why not reach out to our team to hear more about how we can help with your CDR conformance, production verification testing and alerting?