“Serious and systematic defects in Huawei’s software engineering and cyber security competence”
A sobering new report by British security officials paints a picture of desperately poor software and hardware build practices at Chinese telecommunications behemoth Huawei. These pose a “significant risk” to UK telecommunications infrastructure, with several hundred vulnerabilities needing to be reported to UK operators in 2018.
The fifth annual report by the Huawei Cyber Security Evaluation Centre (HCSEC) said analysis of Huawei’s wider software component lifecycle management revealed flaws that cause “significant cyber security and availability risks.” HCSEC is “not confident” Huawei can remediate these “significant problems”, it concluded.
These are Computer Business Review’s Top 10 takeaways from the Huawei security report [pdf].
1: Huawei’s Build Processes are Dangerously Poor
Huawei’s underlying build process provides “no end-to-end integrity, no good configuration management, no lifecycle management of software components across versions, use of deprecated and out of support tool chains (some of which are non-deterministic) and poor hygiene in the build environments” HCSEC said.
2: Security Officials Don’t Blame Beijing
The National Cyber Security Centre (NCSC) which oversees HCSEC, said it “does not believe that the defects identified are a result of Chinese state interference.”
3: Pledges of a $2 Billion Overhaul Mean Nothing, Yet…
Huawei promises to transform its software engineering process through the investment of $2 billion over five years are “currently no more than a proposed initial budget for as yet unspecified activities.” Until there is “evidence of its impact on products being used in UK networks” HCSEC has no confidence it will drive change.
4: The Vulnerabilities are Bad…
Vulnerabilities identified in Huawei equipment include unprotected stack overflows in publicly accessible protocols, protocol robustness errors leading to denial of service, logic errors, cryptographic weaknesses, default credentials and many other basic vulnerability types, HCSEC reported.
“Despite Huawei mandating application of its secure coding standards across R&D, extensive use of commercial static analysis tools and Huawei’s insistence that risky code has been refactored, there has been little improvement in the objective software engineering and cyber security quality of the code delivered for assessment by HCSEC and onward to the UK operators.”
5: Last Year’s Issues Haven’t Been Fixed
“No material progress has been made by Huawei in the remediation of the issues reported last year” HCSEC said.
6: Managing the Risk is Going to Get Harder
“It is highly likely that security risk management of products that are new to the UK or new major releases of software for products currently in the UK will be more difficult”, the report noted, pointing to shortfalls in good software engineering and cyber security practice and the “currently unknown trajectory of Huawei’s R&D processes”.
7: UK Operators May Need to Replace Hardware
The “significant risk” in the UK telecommunications infrastructure brought about by Huawei’s equipment mean “significant work will be required from all parties involved to reduce that risk in existing equipment over time.” In some cases, remediation will also require hardware replacement (due to CPU and memory constraints) which may or may not be part of natural operator asset management and upgrade cycles.
8: Huawei is Using Old Operating Systems
Huawei continues to use an “old and soon-to-be out of mainstream support version of a well-known and widely used real time operating system supplied by a third party” HCSEC said (without naming it…)
“Long-term reliance on this operating system in the UK is unacceptable and an upgrade path must be created. NCSC has not seen a credible plan from Huawei for the mitigation of this issue and an upgrade path to a supportable operating system with a security model appropriate for a modern carrier-grade telecommunications system.”
Operators will continue to have to do “extraordinary work” to mitigate the ongoing risk until a credible plan is enacted, the report notes.
9: Did We Mention Build Processes are Really Bad?
Build processes are very poor, the report notes, detailing VMs containing irrelevant source code, artefacts of previous builds and other detritus; tools installed multiple times in a build environment, or in environments where they are not needed. (“Many tools are significantly out of support and have undesirable properties, for example non-deterministic compilation or optimisation based on environment variable values.”)
Configuration management of source code is also poor and applied inconsistently between development teams: “Product code is managed differently to platform code and both are managed differently to third-party components. Secondly, the integration into the overall product architecture is very poor, with multiple copies and versions of components, apparently identically versioned components containing significant differences, circular dependencies between components and some components regressing in version between overall product increments.”
(You Want an Example?)
Analysis by HCSEC of a Huawei LTE eNodeB solution (an LTE base station component) found “inappropriate suppression of warnings from static analysis tools, potentially hiding vulnerabilities” and ” extensive use of inherently insecure and prohibited memory manipulation functions”. The later, “improved” version contained code that is vulnerable to 10 publicly disclosed OpenSSL vulnerabilities, some dating back to 2006.
That product had been singled out not for risk but as a simple benchmark of efforts by Huawei to remediate a single product, following concerns raised in 2018.
These efforts were inadequate…
10: Lack of Progress has Become Critical
“The lack of progress in creating a credible plan to mitigate the significant installed base of unsupportable software in the UK over the previous 12 months had become critical” the report concludes: “At the time of writing, NCSC has seen no credible plan from Huawei for remediation of the eNodeB or any other Huawei product in use in the UK.”
The NCSC will advise the Oversight Board that it can continue to provide only limited assurance in the security of the currently deployed equipment in the UK, it said.
“A single ‘good’ build will provide no confidence in the long-term security and sustainability of the productin the real world. Huawei’s public statements about their transformation plan state that it will take five years… The UK expects “industry good practice software engineering and cyber security development and support processes as a basis. Huawei currently does not meet that basic expectation” it concluded.
A political decision on the ongoing use of Huawei’s solutions in the UK is expected this spring.