We recently completed a project involving architecture and implementation of a public key infrastructure (PKI) with disaster recovery (DR) at Deutsche Post DHL (450,000 employees, logistics industry) that over time evolved into the foundation of cryptographic and trust services (cryptographic key management, cloud and database encryption, PKI, Document Signing and Timestamping, Multi-Factor Authentication and hardware security). The following few paragraphs aim to highlight the importance of a central component to the implementation, a certificate lifecycle management platform and illustrate the evolution of such services in a corporate environment.

The Evolution of the Public Key Infrastructure Over the Last Five Years

The story goes back to 2012 when I came to Deutsche Post DHL and assumed the responsibility for the PKI infrastructure that had been set up long before I started working with it. A 'relatively small' and 'business critical' PKI deployment at the time consisting of three certification authorities (CAs) – a combination of publicly trusted GlobalSign-signed issuing CAs and a self-signed private CA, no clustering or high-availability, two infrastructure integrations, managed by an in-house developed certificate lifecycle management platform responsible for approximately 300,000 Digital Certificates (SSL, S/MIME and VPN authentication). The following architectural diagram represents the state of the PKI infrastructure in 2012,

As the one man responsible, I quickly became both intrigued by the potential of the infrastructure and surprised how little the business had leveraged it. Whether it was because the implementation was sufficient to cope with the business demands at the time or because it simply lacked the attention, I set out to change it. The goal was to turn a small and 'business critical' infrastructure into a profitable business critical service with possibilities to attract new customers and business opportunities, a foundation for a cluster of services that the business could benefit from long after I have gone. The greatest challenge right off the bat was attracting the business attention to it, people kept asking me: “why are you doing it?”, “who needs it?” and “what’s the point?” but despite all of that, perseverance, passion and vision became the guides in the years ahead.

During the years in Deutsche Post DHL, it took a lot of patience, verbal communication and time to emphasize the potential of PKI, its numerous applications, and added value to the business and senior management. Most importantly, we learned that a practical demonstration is infinitely more effective than a colorful marketing.

We tried many times to highlight the importance of security, digital trust and reputation to the management, to explain to security teams why is it important to have a centralized certificate lifecycle management, why is it important that cryptographic keys should be protected and managed,

 A cryptographic key is as valuable as the aggregate value of the assets the encryption is protecting.

what is the business impact of identity theft and man-in-the-middle and how a loss of reputation and digital trust is far worse than a monetary loss in the long run, but in the end, words are a dime a dozen and a picture is worth a thousand words. Too often an investment into security abides by the following equation,

Profit = Return on investment – investment

If Profit > Investment, it is good; if Profit >> Investment, it is very good. If you are a provider of PKI services, such as GlobalSign, it’s easy to calculate the profit. If, however, you are a consumer of PKI services, estimating the monetary return on investment of using, for example, SSL – digital trust, security - becomes an unknown variable or at the very least not so easily quantifiable and that’s the sticking point.

Whenever we tried to justify an investment to the management with words, we hit a dead end, “…you are not the first to tell us that this newly proposed solution or product is better and more secure; we have had the current solution for many years and we don’t see the added value in this investment…”. We found the truly effective method was demonstration, plain and simple.

Once we showed what happens when a company’s signature website has a weak SSL configuration with an expired or revoked certificate and instead of a seamless content loading the users have to struggle with erroneous or untrusted/insecure messages, along with explaining the repercussions of a compromise of a cryptographic key, it became much easier to explain the risks, the mitigations, the added value to the business and in turn to receive the so much needed investment. The above does not advocate a willful destruction of a production environment but rather how to effectively translate a technological demonstration into a business understanding within a controlled environment for the purposes of “what would happen if…and how to therefore minimize the risks.”

  A demonstration is much more powerful than explication; nobody can argue with a demonstrated fact.

By 2014, Deutsche Post DHL had become sufficiently acquainted with PKI services and how it related to its day-to-day business and the topic of Digital Certificates and PKI became a part of everyone’s agenda. Because of the business critical nature of the infrastructure and the growing demand for PKI services, the business realized the need for redundancy to guarantee business continuity and API automation to keep up with the growing demand.

That was the time the PKI 'team' grew by one and the PKI DR project began.While the goal from the 2012 remained, the challenge changed to implementing a new generation of the PKI with redundancy, automation and innovation as main drivers.

Before my colleague and I left Deutsche Post DHL in January 2017, there were two major success stories to tell, the first was that the people who years before had kept asking me “Why are you doing it?”, “Who needs it?” and “What’s the point?” suddenly started asking me “How did you achieve it?”. The second was that the infrastructure had grown far beyond the project’s original scope – PKI DR – into what I could now in retrospect call a cryptographic and trust services department with:

• Six clustered CAs - a combination of publicly trusted GlobalSign-signed issuing CAs, self-signed private CAs and a GlobalSign Managed PKI platform,
• Gemalto HSM– hardware security appliances to securely store cryptographic signing keys of the CAs,
• 4,000,000 managed certificates (SSL, S/MIME, mobility and IoT, Multi-Factor Authentication, Document Signing, machine certificates [including TPM-protected for high-value systems]) with various infrastructure integrations,
• and much more…

Click the image to see it in full size. 

The Opportunities Multiply as They Are Explored: The Need for Certificate Lifecycle Management and Automation

To borrow from Wikipedia,

  A public key infrastructure (PKI) is a set of roles, policies, and procedures needed to create, manage, distribute, use, store, and revoke Digital Certificates and manage public-key encryption.

Which is 100% correct, however there are two major areas that are of the utmost importance. The first is the secure management of private keys and their lifecycle:

• “Where are the keys generated and stored?”
• “Who has access to them?”
• “How are the keys archived and disposed of?”

The second is the management of the certificate lifecycle:

• “How are the certificate signing requests generated?”
• “How are certificates installed?”
• “How is the certificate validity monitored?”
• “How are the certificates renewed?
• “Is the certificate enrollment process compliant?”

We found the areas of private key and certificate lifecycle management to be very important and overlooked components of a PKI deployment.

The certificate lifecycle management platform is both a bridge that connects the client-side of PKI with the server-side PKI and a gateway through which users, administrators, customers and businesses see the PKI. It is also the one place where the support teams can centrally manage what goes on under the hood, while at the same time allowing for automation through interesting API integrations to attract new business opportunities and make day-to-day PKI activities more effective

Due to the nature and sensitivity of the PKI infrastructure coupled with strict German laws and internal security policies, making use of a 3^rd party controlled management platform to manage the infrastructure was out of the question. Therefore, after considering a few on-premises solutions and the advantages and disadvantages of each one with the respect to the environment we were building on, we chose the CMS from CSS. 

The homogeneous nature of the infrastructure and the immense volume of infrastructure devices, mobile devices, users, SSL endpoints and potential integrations justified a need for automation. The deployment of a centralized certificate lifecycle management provided an elegant method of balancing between deploying new services and being able to support them, subsequently it catalyzed a number of projects that followed, to name a few,

• Mobility and internet of things.
• SSL automation.

The following two examples were two of many PKI DR integrations in Deutsche Post DHL where the certificate lifecycle management played a pivotal role due to the sheer diversity and volume of endpoints involved. It does not talk about how we did it, but what challenges we faced and why the certificate lifecycle management was the weapon of choice.

Mobility and the Internet of Things

With mobility and Internet of Things, the most challenging aspect was controlling a secure generation, storage, deletion and enrolment of cryptographic material on(to) the device. Additionally, the volume and the diversity of the mobile PKI deployment,

• 20,000+ mobile devices,
• Apple iOS, Android, Windows, Blackberry; both legacy and new mobile devices,
• Different cryptographic parameters (signing algorithms, certificate purposes, assurance levels, validity),
• Various compliance requirements and device categories (BYOD, COPE),
• Multiple mobile-device-management platforms,
• Infrastructure integrations.

made it even more challenging to set up a reasonable degree of control over the lifecycle of certificates enrolled onto the mobile devices.

• A basic mobile PKI setup consists of a mobile device, mobile device management (MDM) platform, and PKI, however there are multiple caveats,
• Mobile devices do not use hardware security to protect private keys and the current software controls represent zero-ish security,
• Mobile device management platforms make use of a SCEP protocol, which was originally not designed for mobile devices and therefore lacks any measurable security controls,
• No one mobile device management platform supports every mobile operating system and therefore enforcement policy and “security controls” are inconsistent,
• The PKI in the world of mobile devices is definitely valuable, however unless endpoint protection and anti-malware detection is deployed and religiously enforced, the PKI is the least of your problems.

There is also a difference between what users want (BYOD) and what a company wants (COPE).

From the perspective of a user (BYOD):

• Multiple mobile devices; different operating systems, typically Android and iOS.
• Enforcing corporate settings on non-corporate devices is a legal minefield.
• Certificate enrollment over “quarantine” WIFI, external VPN, manual .PFX enrollment.
• Short-lived certificates; user certificates.

From the perspective of a company (COPE):

• One or two mobile devices, typically Blackberry, iOS or Windows Phone.
• Enforced corporate settings; allowed applications through mobile device management.
• Pre-enrolled certificates; device certificates + user certificates.

A lot of effort needs to be done to improve the situation, a replacement of SCEP is underway - EST and Metrarc to innovate on the field of cryptographic key security.

A mobile PKI deployment of this scale is definitely not a small feat and requires a constant vigil and stringent control when onboarding and managing mobile devices whether it is BYOD or COPE.

SSL Automation

The SSL automation was an important step towards converting a manual process of installing SSL Certificates and configuring SSL endpoints into a semi-automated and standardized process. Due to a mix of legacy and modern systems, we could not deploy one configuration profile to fit all because of issues with backward compatibility, however even in its semi-automated settings, the automation was efficient enough to yield considerable risk reduction and resource savings. A genericconfiguration would contain the following settings and perform the following operational tasks;

• cipher suites and supported SSL/TLS version(s),
• SSL session or TLS ticket configuration, renegotiation settings,
• HSTS or HPKP and OCSP stapling,
• SSL Certificate and optionally a certificate chain,
• monitor the validity of the SSL endpoint and notify as appropriate,
• renew with a new key or an existing one (automatically or manually).

With 50 SSL endpoints, such a task is not a problem and can be handled manually, but in an environment with,

• 100,000+ SSL endpoints with a mix of legacy and modern systems.
• Different infrastructure integrations (F5, Citrix, Juniper, Cisco, Bluecoat, Apache, Windows, Linux).

a manual approach becomes ineffective and risky due to the possibility of a human error. It is true a downtime is particularly damaging to business critical financial SSL-enabled applications where every minute of a downtime translates into a substantial loss. However when push comes to shove, every downtime is 'critical'; it just depends on who is breathing down your neck.

With the recent outbreak of SSL vulnerabilities (e.g. Heartbleed, BEAST, BREACH, CRIME, SSL renegotiation, weak keys and hashing algorithms, insecure SSL/TLS stacks), more and more attention has been given to security reviews of existing SSL implementations, hardware security, TLS 1.3 development and cultivating awareness on identity theft and man-in-the-middle repercussions. However, it all falls on deaf ears unless businesses learn to think of certificates not just as files sitting on a server somewhere, but as assets that protect digital trust, reputation and security that we all depend on every day, whether in the digital world or the real world.

To summarize, the added value of a certificate lifecycle management:

• A centralized place to manage the lifecycle of Digital Certificates,
• Increased visibility into the public key infrastructure for the support teams,
• Reporting capabilities for the business and 3^rd parties (if any),
• Stronger issuance compliance checking,
• Automation through API integration with security infrastructure services and managed platforms,
• Integration with a ticketing system workflows (e.g. Service Now).

In Conclusion

When business grows, so does the demand for PKI and digital certificates. With abstract definitions such as authentication, integrity, encryption, digital trust and reputation, it is sometimes difficult to translate their meaning to business because in and of itself they do not represent anything tangible. They become quite tangible, however, when data breach occurs, and therefore the core objective is for the business to understand their added business value beforehand.

And because faster, better, cheaper, more secure, more convenient, more customer-centric, no downtime, no impact - even though some at times mutually exclusive - represent a life in a corporation, the certificate lifecycle management should be a mandatory component of a large-scale PKI implementation simply because it allows for the roster of the support team to support it and for the business to have a control over a core business infrastructure service. In the end, it all stands or falls on the perseverance, passion and vision of one man or woman with an idea.

About The Author

Lukáš is a former security engineer responsible for cryptographic services and public key infrastructure at Deutsche Post DHL and a Co-Founder at PKI TECHNOLOGY where he specializes in cryptographic technologies such as cryptocurrencies and blockchain, PKI, cryptographic key management and authentication.

He is also interested in reverse engineering, operating system security, digital forensics and contributing to cyber security by speaking at security conferences and publishing.

Connect on LinkedIn.  Connect on Twitter. 

Bio of PKI TECHNOLOGY

The PKI TECHNOLOGY was founded in 2016 by former employees of Deutsche Post DHL to continue work in the area of cryptographic services, virtualization and infrastructure security.

Note: This blog article was written by a guest contributor for the purpose of offering a wider variety of content for our readers. The opinions expressed in this guest author article are solely those of the contributor and do not necessarily reflect those of GlobalSign.