In this blog, Simon Whittaker and Greg James discusses the recent outages faced by Smile, an online banking service offered by the Co-operative Bank.
The Co-operative Bank has been no stranger to operational issues affecting its app and web services over the past few years and, yet again, it has suffered issues. The recent outages, with the accompanying customer impact, illustrates common issues and why the regulators are emphasising the importance of building operational resilience in financial services.
What happened?
For context, Smile is an online banking service offered by the Co-operative Bank and, separate to this, the Co-operative Bank offers its own online services.
Over the past fortnight, Smile, has been experiencing significant outages resulting in their customers being unable to access services. Often in the case of extended outages, a cyber attack is the key suspect however it is much less nefarious in this case. The public reason given for the outage was a database error during routine maintenance, a reminder of the importance of development controls. This issue did not affect the Co-operative Bank app.
More positive messages started being presented by Smile from 11 July and as of the 14 July, the issue appears to be resolved.
Separate to this issue, and arguably less significant from a customer perspective, Smile and the Co-operative Bank apps have suffered from a ‘certificate’ issue. According to the Co-operative Bank, an error with a third party supplier caused the apps to require an update in order to fix its ‘certificate’ issue. By putting on our detective hats we can say that this issue is likely related to the recent Digicert ICA SSL certificate revocation. A brief review of the Co-operative Bank website all but confirms that this is the issue as the SSL certificate, at least on the website, is one of the newly issued certificates by Digicert. The Digicert issue was previously identified by the FCA as being a possible supplier chain issue for the financial services sector.
The certificate issue is resolved with a new version of the app now available to users both on Android and iOS and with the publication of a relatively comprehensive FAQ. Although, from a technical perspective, the issue has been resolved customers still need to install the update which will cause friction in an already strained customer relationship.
Why does it matter to me?
In recent weeks we have flagged the importance of operational resilience, especially in relation to the vital importance of third-party suppliers. At the beginning of the year, we identified operational resilience as one of the key issues for 2020 and in May, we hosted a webinar on the subject.
More specifically, the issues with Smile and the Co-operative Bank speak to two separate issues that resulted in operational incidents and will be relevant to most firms: change management and supplier chain risk.
The new ICT and security risk management guidelines call out supplier risk in relation to operational resilience, guideline 3.7.3 (line 86) and the FCA is currently consulting on imposing new requirements to establish impact tolerances for outages of services that ultimately impact on customers. The certificate issue in the Co-operative Bank and Smile apps is a good example of how supplier chain risk can materialise in the least expected areas. Supplier chain risk is a common issue with 17% of incident reported by firms to the FCA being related to third-party suppliers.
Looking at the Smile-specific incident, it stemmed from routine maintenance that culminated in a database issue affecting the online service for all customers. The ICT guidelines noted above make clear that change management is a key control expected of financial services firms to avoid incidents like this. Smile’s customers were unable to access the app for a significant period of time. We don’t know what Smile’s Recovery Time Objective (RTO) in the mandatory Business Impact Analysis was but we can be sure that it was not almost 10 days for a business service as important as the app… for an online-only service.
The moral of the story
In the case of Smile, the moral is that change management must be duly considered for any amendments due to be rolled out to a live system, even for routine maintenance.
Although the supplier chain risk is a major issue in the financial services industry, the issue that affected the Co-operative Bank and Smile apps was relatively minor and was dealt with swiftly. However, due to the recent technical issues plaguing the Co-operative Bank, and especially Smile, this event has created a reputational effect that far outweighs its related operational effect. Therefore, it is still an important reminder to really understand the supply chains that your business relies on.
Especially during these strange times, there are some important questions we must ask ourselves and our suppliers.
- Do you maintain a change management policy and process?
- Is the change management process that you apply consistent and robust for all related systems and entities?
- Have you separated your development environments appropriately, for instance into development, testing, staging and production environments?
- Do you have a supplier register and is it up to date?
- Have you defined your minimum supplier requirements, should they have specific certificates?
- Have you shared your RTOs with your suppliers?
- Do you have supplier contracts in place listing key service level agreements
- How would you cope if they were to experience a significant outage?
- How does your firm deal with certificates and cryptographic requirements?
If you would like to speak to a member of our team about any of the issues raised in this blog, please do not hesitate to get in touch.
This post contains a general summary of advice and is not a complete or definitive statement of the law. Specific advice should be obtained where appropriate.
