Since its inception, the secure shell (SSH) data-in-transit protocol has been utilized by organizations of all types and sizes as a secure method to move data from machine to machine and provide remote administrator access. A version of SSH is shipped with every version of Linux, Unix and Mac OS, and its use is growing rapidly in the Windows universe as well. Approximately half of all of the world’s websites use a version of SSH. The exact number of SSH deployments worldwide is impossible to pinpoint, but can easily be estimated in the millions – making SSH an asset ubiquitous in the network security world.

SSH has fulfilled its role as a hardened security solution for over 17 years. Since its creation, it has secured billions of business transactions without suffering any major security breaches caused by vulnerabilities in the protocol itself. While the protocol itself is highly secure, today’s rapidly-changing threat landscape is forcing organizations to reconsider how they manage their SSH environments.

The Evolving Threat Landscape Is Now At Your Doorstep

Traditionally, SSH has been used to transfer massive amounts of sensitive business information, including credit card numbers, personally identifiable information, healthcare records and classified intelligence. From the perspective of an attacker or malicious insider, SSH is an artery that carries vital organizational data.

Yet since the protocol itself is secure, how would a bad guy get access to sensitive information protected by SSH? In this case, the keys are key.

When users connect via SSH, a trust relationship between a computer and the server is established using a cryptographic key pair. These trust relationships are created and managed internally, sometimes on systems dating back to the mid ’90s. None of these systems have the ability to search for – let alone find – where the company’s trust relationships exist. Tracking trust relationships must therefore be done manually. When a network has potentially hundreds of thousands of keys, trust relationships are inevitably lost. If a malicious actor – internally or externally – gains access to one of these keys, he or she can mimic an authorized user with impunity.

Improper management of SSH keys, therefore, presents a prominent vulnerability available for exploitation by attackers looking to gain access to sensitive information. After a study was performed on the management operations of some of the largest organizations in the world, a disturbing trend appeared:

• About 10 percent of all SSH user keys provide root access, creating a major security and compliance issue
• Organizations often share the same SSH host keys across thousands of computers, leaving the network vulnerable to man-in-the-middle attacks
• Enterprises rarely know what each key is used for, presenting not only a security risk, but which poses both a security risk but also a business continuity risk
• Many SSH keys that grant access to critical servers are orphaned and no longer in use
• Some organizations permit administrators to create or delete SSH user keys at will – without approvals or control – essentially granting unfettered, permanent access to systems and people
• Very few organizations ever rotate SSH user keys, or even remove them when a user leaves or an application is decommissioned
• Organizations often share the same SSH host key across thousands of computers, leaving the network vulnerable to man-in-the-middle attacks
• Very few organizations remove keys when a user leaves or an application is decommissioned
• Key-based access grants are essentially permanent, in direct violation of SOX, PCI (News - Alert) and FISMA requirements for proper termination of access, leaving the network vulnerable to attack

With advanced threat vectors becoming more commonplace, the risks faced by organizations without proper SSH key management protocols in place are very real. The greater the variance from a best practices approach to SSH key management, the greater the risk is to the organization.

In addition to the obvious security implications of SSH key mismanagement, organizations need to be aware that federal compliance standards – such as PCI, SOX, NIST and HIPAA – require organizations to maintain a high degree of control over access to sensitive network information, or risk costly fines. This isn’t even getting into the economics argument. Many organizations today have in excess of 20,000 servers, which pegs the cost of manual SSH key management at $40 million over 10 years.  Add in the significant reputation damage that follows a security breach, and organizations are staring at a whole host of incentives to remediate their SSH key management practices.

Key Management Practices Must Change

Fortunately, issues with access control in secure shell environments are not a result of any vulnerabilities or flaws in the SSH protocol itself. Rather, the security and compliance risks identified are caused by:

• Years of lack of clear guidelines or policies relating to SSH key management
• Lack of understanding of the scope and implications of the problem
• Insufficient time and resources to dig into the issue to gain understanding or develop solutions
• A lack of good tools and guidelines early on for solving key management issues
• A reluctance on the part of auditors to flag issues for which they don’t have effective solutions
• The focus of the access management field on interactive users without addressing automated access

It’s easy to wonder why this problem has remained hidden for so long, given the ramifications of exploitation. The answer is simply that SSH key management is so deeply technical, it has remained hidden and somewhat opaque in the domain of system administrators. Each system administrator typically only sees a small corner of the IT environment, and does not have the full picture. Administrators are so busy – especially with staff reductions in recent years – that they may not recognize that there is a problem. Since management may be several steps removed from the problem – and its potentially devastating consequences – the end result is that no action is taken.

But the risk remains.

Best Practices for SSH Key Management Remediation

Because the vulnerability is usually found in all Unix/Linux servers and many Windows servers, the steps needed to fix the issue will involve several teams within IT operations. The potential liability and compliance issues demand the awareness and buy-in from executive management as well.

Some best practices to remedy the problem include:

• Discovering all existing users, public and private keys, and mapping trust between machines and users
• Monitoring the environment to determine which keys are actually used, and removing keys no longer in use
• Enforcing proper approvals for all key setups
• Automating key setups and key removals; eliminating manual work and human errors. This step slashes the number of administrators needed for key setups from potentially several hundred to only a few highly trusted administrators
• Rotating keys regularly, so that copied keys cease to work and proper termination of access can be ensured
• Restricting where each key has access and what commands can be executed using the key

To further reduce risk, proper key management should involve the establishment of internal boundaries within the organization. The organization should strictly control what key-based trust relationships can cross which boundaries, while enforcing iron-clad IP address and “forced command” restrictions for all authorized keys involving trust relationships crossing such boundaries.

While SSH is widely considered the benchmark for data-in-transit security, the current threat landscape requires organizations to rethink how they are managing access to their encrypted networks. The SSH protocol has done a great job in protecting data-in-transit at a tactical level, but an ever-increasing number of threat vectors means effective management of the SSH environment is critical to secure network operations. Best security practices like the ones identified above will position your enterprise to prepare for security threats and new compliance mandates before they occur. 

About the Author:

Jason Thompson is director of global marketing for SSH Communications (News - Alert) Security. Mr. Thompson brings more than 12 years of experience launching new, innovative solutions across a number of industry verticals. Prior to joining SSH, Mr. Thompson worked at Q1 Labs where he helped build awareness around security intelligence and holistic approaches dealing with advanced threat vectors. Mr. Thompson holds a BA from Colorado State University and an MA for the University of North Carolina at Wilmington.