ERP Security and Technical Debt with Saša Zdjelar

Saša Zdjelar is Software Security Group (SSG) Supervisor and Software Security Design Lead at ExxonMobil Corporation. In the first installment of this two-part interview, he discusses the challenges of securing Enterprise Resource Planning (ERP) systems and the consequences of “technical debt.”

Question: You’re an expert on ERP – Enterprise Resource Planning – security. What are the challenges with securing these systems?

Zdjelar: My concerns in this area have to do with a combination of what is often referred to as “technical debt” and the criticality of data that resides in these systems.

ERP generally involves internal, “back office” systems, the sorts of systems that run a company, such as payroll, financials, financial planning, treasury, logistics, product movement, mergers and acquisitions and the like, but can also include external facing systems which interface with banks, suppliers, customers, etc. and are often subject to compliance and regulatory standards such as PCI and SOX due to the nature of the data they store and process.

These systems started being broadly adopted in the early 1980s and they didn’t fit the traditional norms around how you develop or operate software, so they tended to be given to operational teams that worked side by side but operationally separate from other typical application support teams. Thus they often fell outside the scope of people who manage traditional application security and these systems significantly predated application security as a domain.

These systems were designed to be stood up and not touched for three to seven years until the next upgrade cycle. Fifteen years ago that may have worked, when the only risk being considered was to availability and operations – you simply didn’t want to take the system down. That also meant, however, that in those intervening years, you haven’t patched anything or placed emphasis on security.

Fast forward to today. In one instance, a prominent ERP brand has issued over 3,500 patches in the last two years. But most companies have only processed about two percent of those patches. That means that, surprisingly often, those systems are running with vulnerabilities, and not just recent ones. Vulnerabilities from years ago are being used actively by attackers to compromise and breach companies with issues that are several years old but remain unpatched. For operational risk reasons, and due to the relative immaturity of tools to help facilitate testing and patching, businesses typically don’t want to risk taking these systems down to apply those patches.

Question: Would you explain the term “technical debt”?

Zdjelar: What’s the cost tomorrow of a shortcut from yesterday? Each time you make a sub-optimal technology decision, you’re incurring debt for future re-engineering and refactoring of the cost of that decision. Think of it like stretching a rubber band a bit further each time. The more sub-optimal the decision and the longer you keep doing it, the more energy you’re storing for when the rubber band finally snaps back. It can happen due to premature adoption of a technology, inadequate integration, tight code coupling, misunderstood implications of middleware dependency, poor design and engineering, lack of sustainment model, etc.

Question: Who is vulnerable in this scenario?

Zdjelar: The vulnerable companies are those with legacy ERP systems designed and implemented many years ago without the appropriate, upfront engineering and sustainment processes related to security. Today’s ERP systems can be very secure if properly hardened, patched, and engineered with appropriate governance and sustainment which includes business and IT alignment on upgrade cycles, development standards, monitoring, etc. But many companies continue to operate those legacy systems for the reasons stated: they are awaiting a predetermined upgrade cycle due to the operational risk of taking those systems down to apply security patches and funding by the business.

In fact, though today’s ERP platforms are designed with appropriate security mechanisms, the lead times to implement these new platforms are still so long and expensive, and most companies won’t be running these new, secure versions for perhaps another five to ten years. The systems are often highly customized to meet specific business needs which also contributes to the technical debt. The problem with technical debt is that companies often have hundreds of millions of dollars invested in legacy platforms which, because of business cycles, won’t be retired for many years.

Question: What is it about legacy ERP systems that make them vulnerable?

Zdjelar: Legacy ERP systems typically lack a foundational principle known as “open design.” Open design systems are secure because their design has been subject to open review and is based on common industry protocols and standards. In open design, the security also isn’t dependent on the obscurity or secrecy of some part of the design. Legacy ERP systems, in contrast, are often highly proprietary in nature. They tend to fly under the radar because not much is known about them, including in the infosec community. But as soon as security researchers figure out how they work, they quickly figure out how the protocols work and can develop exploits. For instance, rather than encrypting data in transit, very often these systems’ data traffic is compressed. To the casual observer it looks like unreadable traffic, which may lead you to believe it’s encrypted, but in fact, it’s easily uncompressed. Once a tool is developed to decompress that traffic, it effectively becomes the equivalent of clear-text traffic, and those tools exist today. They’re free and they’re easy to use.

Of course, the concept championed by the IEEE Center for Secure Design, and others in industry such as Microsoft and the BSIMM framework, essentially says that software should have security built-in from the start and continuing all the way throughout the application lifecycle until disposal. Too often, these legacy ERP systems are not secure by design or by default. When they are installed with default configuration, they’re installed very permissively, with several static encryption keys, with privileged accounts turned on versus being turned off, default passwords, unnecessary and vulnerable services running, port ranges which include common services, etc. These systems need hardening, to say the least.