This post is the second of two related to the insider threat attack suffered by Ubiquiti, Inc. (NYSE:UI). The first post provided a background to the investigation including how it was discovered and actions taken by Ubiquiti and law enforcement.
LESSONS LEARNED AND INSIDER THREAT MITIGATION
Unfortunately, Ubiquiti learned a very public and expensive lesson on insider threats. Between lost future revenue and the $4 billion in lost stocks due to this event, Ubiquiti will be feeling this attack for years. Based on publicly available information, it appears there were some missed opportunities to catch this attack before it progressed to the level it eventually did.
Natsar has significant experience in setting up monitoring solutions to detect insider threat activity and has been involved in the investigation of insider threat cases. In more than one occasion, detection capabilities implemented by Natsar’s staff led to the identification of insider threats, termination of employment, and in some cases, criminal investigations. If this is something you need help with, contact us to discuss Natsar’s services in this area.
Here are some areas that could have prevented this insider attack, or at a minimum, detected it early enough that it would have prevented the theft of data and reputational harm.
Separation of Duties
Separation of duties is the principle that no user should have enough privileges to misuse a system on their own. This is common in the financial world, where the person that authorizes paychecks cannot also be the same person who writes them. In IT, separation of duties is often neglected because of time pressures or lack of staffing. In this case, had the employee not been able to access all the systems by themselves, this could have been avoided. They clearly had significant access, including to root administrative privileges for multiple systems across the Ubiquiti enterprise.
User Entity Behavior Analytics (UEBA)
Plenty of security platforms exist in the UEBA space that could have alerted personnel to the actions of the employee in real-time. For example, most of the illegal activity occurred after business hours. A UEBA tool would have identified the employee’s access of systems after work hours as potentially suspicious and flagged it for an analyst’s review. Additionally, the access to the root administrative SSH key and the download of this key should have immediately triggered an alert.
Blocking External IP Access and Foreign IPs
One security control that would have stopped this attack before it started was blocking access to any Ubiquiti systems and data from IP addresses outside of the United States. Simple firewall rules could have been added to Ubiquiti’s AWS infrastructure that would have stopped any known VPN, TOR, or foreign IP address from even being able to login to the backend of their systems. Even better, Ubiquiti could have required that all administrative logins must come from a corporate IP space, meaning that administrator’s would have to either be onsite at an Ubiquiti office, or connecting in via a VPN or virtual desktop.
Data Exfiltration Monitoring
A very simple report to create within a Security Information & Event Management (SIEM) tool is netflow information. That is, information that is aggregated over a period of time (say 24 hours) that shows the source and destination of traffic, the protocols used, and the amount of data sent. Many organizations have identified intrusions, both from insiders and outside adversaries, only by looking at traffic patterns and seeing an unusually high amount of data leaving their systems or networks. In this case, using GitHub Actions to send alerts of repository access and downloads to a SIEM could have almost immediately notified a security operations center (SOC) analyst of abnormal activity worthy of investigation.
Monitoring Administrative Access
One of the most important items to monitor in an enterprise is the use of privileged (administrative) credentials. It is important for cybersecurity staff to see when administrators are accessing systems, what IP address they are coming from, and what they did while being logged in. This can be done extremely easily by sending Windows, Linux, and application event logs to a SIEM and creating alerting to show this activity. Had this been monitored by Ubiquiti, someone should have noticed the foreign IP addresses coming into their system with the root SSH key and immediately began investigating.
Monitoring System Changes
One item that caught our eye in this incident is that the employee allegedly changed the logging retention of Ubiquiti systems in an attempt to cover their tracks. This change is security significant and should have been captured in an event log and sent to Ubiquiti’s security team. Natsar always ensures that system events such as clearing event logs or changing log retention get sent to our customer’s SIEM tool for review. As the old saying goes, “you can’t wipe the wiper”, meaning that there is always some artifact left behind, even if the artifact is evidence of other artifacts being deleted.
Multifactor Authentication (MFA)
Using MFA to access the GitHub repository or AWS in addition to SSH could have also stopped this attack or at least made it much easier to find the suspect faster. In most cases, implementing MFA is free and can drastically reduce cyberattacks that are using stolen credentials. In this case, the employee had the private SSH key, however if Ubiquiti would have added MFA in addition to the private key by using something like Google Authenticator, they would have had to use their mobile device to obtain the one-time passcode (OTP) before logging in. This would have tied their login to their mobile device, eliminating any defense that someone else had accessed the Ubiquiti systems.
Not Allowing Community Credentials
We get it, having common accounts for administrative actions is simple and easy. Unfortunately, simple and easy is typically the antithesis of cybersecurity. This case involved the employee logging in to Ubiquiti systems using their personal account, then accessing shared administrative SSH keys to get into other systems. These shared SSH keys are essentially common accounts that don’t allow for non-repudiation. Best practices would dictate that every account is always named and assigned to a single person, so all activity that happens from that account can be attributed to a particular person. Having a common account like in this case makes it harder to identify who was responsible for particular actions since so many people could have had access to the account.
Just in Time (JIT) Credentials
There are several vendors today that have credential vaults for JIT access. Instead of employees always having these administrative accounts and risk them being misused or stolen, administrator’s must “check out” credentials at the time they need them. At a high level, an administrator makes a request to a system and if they are allowed to access what they are requesting, they are provided a temporary set of credentials that will work for a limited duration. After the administrator completes their task, the credentials are destroyed and will no longer work for that system. An audit log is created that shows all requests and provides sufficient detail to identify what person was responsible for certain activities. If the employee in this case was required to checkout credentials, it’s highly likely they would not have then used those credentials to commit a variety of federal felonies.
Building insider threat programs and conducting investigations into insider threats is a core capability of Natsar. Whether you need written policies, technical and administrative controls, a second set of eyes to review your environment, or training for your staff in this area, let us know.
At Natsar, we are continuously improving our products and services. Please let us know how we did and if this information and resources were helpful to you.