This post is the first of two related to the insider threat attack suffered by Ubiquiti, Inc. (NYSE:UI). Ubiquiti develops technologies such as network switches and routers, wireless networking equipment, phones, security cameras, physical access control devices, and more. The attack began by a former employee of Ubiquiti in December of 2020 and became public knowledge in March of 2021. After an extensive investigation by the FBI, the former employee was indicted by a Grand Jury on four federal felonies.
This attack against Ubiquiti should be a wakeup call to organizations. We have seen plenty of insider threat attacks before that have had significant repercussions (Chelsea Manning, Edward Snowden, and others). This particular attack highlights several missed opportunities to altogether prevent or at least quickly detect the employee’s malicious activity. It also shows the financial impact of such an attack. Ubiquiti lost over $4 billion in market capitalization and an untold amount of future revenue. To some businesses, an insider attack like this one would put them completely out of business. As with so many other cyberattacks, the reputational damage and financial impact is far greater than what would have ever been spent on cyber talent and defensive capabilities.
While the employee’s name is public knowledge, Natsar has chosen not to name them in our posts. At the time of this series, the employee has only been accused of these crimes. As such, the employee is presumed innocent unless and until they are proven guilty.
- The employee allegedly used their credentials to steal gigabytes of private company data and attempted to hide their identity by using a virtual private network (VPN) service
- It is reported that the employee sent a demand to Ubiquiti for $2 million in ransom after stealing the data
- A technical glitch of the VPN service temporarily exposed the residential IP address of the employee, which was logged and discovered during the FBI’s investigation
- Not knowing that the employee was actually behind the attack, Ubiquiti assigned them as part of the incident response team to investigate the intrusion
- After the FBI served a search warrant at the employee’s residence, they allegedly wrote posts on the Internet posing as a whistleblower within Ubiquiti and stating that the cause of the breach was vulnerabilities in Ubiquiti’s cyber defenses
- Following these public statements, Ubiquiti’s stock fell by 20%, costing the company over $4 billion
The former employee and now criminal defendant was a senior software developer for Ubiquiti and employed by them between August, 2018 and April, 2021. As part of their duties as a developer, the employee maintained credentials to access various accounts used by Ubiquiti, including to their Amazon Web Services (AWS) infrastructure and GitHub. The FBI learned that the employee used their personal PayPal account to subscribe to a VPN service called Surfshark VPN on 7 July 2020. Surfshark VPN is based in the Virgin Islands and offers a VPN service for $2.49 per month and advertises that the service will allow users to browse the Internet privately and change their IP address via their network of over 3200 servers in 65 countries.
The Internet Protocol (IP) address of a computer is like a person’s telephone number. In telephone numbers, two different people cannot have the exact same phone number, otherwise there would be no guarantee that a call would arrive at the correct destination. Like phone numbers, IP addresses can be looked up to see geographically where the system exists, what company “owns” the IP address, and who the subscriber is that has that IP assigned. IP addresses are used to identify the source and destination of Internet traffic and by following certain legal processes, the end user (or subscriber) can be identified.
By using a VPN, the suspect in this case was able to bounce their Internet traffic through the Surfshark service and then to their desired destination. For instance, if the employee was in Portland, Oregon and wanted to access AWS servers used by Ubiquiti, they would only have to enable the Surfshark VPN client on their device, then navigate to AWS. By enabling the Surfshark client, the employee’s Internet traffic would go from Portland to a destination server operated by Surfshark. The Internet traffic would then be routed from the Surfshark server to AWS. Surfshark acts as a middle-man, masking the originating IP address of the user. As far as AWS is concerned (and what their logs will reflect) is the IP address of the Surfshark VPN server. Many individuals will select VPN services outside of the United States because it makes it more complex, if not impossible, for US law enforcement to serve the VPN providers with legal process in an attempt to identify the original source of the traffic.
It should be noted that not all VPN usage is nefarious. In fact, VPNs can be an excellent way to protect privacy and increase security and are frequently used by businesses.
Additionally, many VPN providers intentionally do not keep logs, so even if they receive legal process, they have nothing to provide law enforcement.
Below is a very simple overview of how a VPN works. In this example, a suspect at home with an IP address of 220.127.116.11 connected to a Surfshark VPN server with a final destination of AWS. The Surfshark VPN server masked the original IP address and gave the user a new IP address of 18.104.22.168. Surfshark VPN then sent the traffic to AWS, making AWS believe the traffic is all originating from the 22.214.171.124 IP with no idea of the 126.96.36.199 IP.
According to the FBI investigation, the employee logged in to Ubiquiti servers from their home IP address on 9 December 2020 at around 6:55pm local time, and accessed a Secure Shell (SSH) key that would allow them to access other resources within Ubiquiti’s AWS infrastructure. An SSH key is essentially an encryption key that allows users that have that key to log in to systems and acts as a username and password. Two SSH keys are required to make a successful connection, called a public and private key pair. Private keys are to be protected just like passwords and must be stored carefully.
Only two minutes after the employee’s home IP address connected to Ubiquiti and accessed the private SSH key to AWS, an IP address belonging to Surfshark VPN connected to Ubiquiti’s AWS infrastructure and used the same SSH key that the employee’s IP address had just downloaded.
On 21 December 2020, the employee accessed Ubiquiti’s GitHub repositories using their personal credentials and residential IP address. Then just one minute later, another connection was made to GitHub using a Surfshark VPN IP address and an SSH key the employee had. They then allegedly used the SSH key to access and make copies of GitHub repositories (company data and code) to the employee’s personal home computer.
The indictment against the employee describes a momentary glitch in the connection between their computer and the Surfshark VPN service during the exfiltration of Ubiquiti’s GitHub data. This glitch resulted in the employee’s residential IP address being logged in GitHub logs in the middle of the Surfshark VPN traffic. In all, 155 GitHub repositories were downloaded from Ubiquiti’s GitHub account to the employee’s home computer.
To cover their tracks, the employee then reportedly accessed AWS systems and applied a one-day lifecycle retention policy to logs. By applying this policy, logs would only survive on the server for one day and then be deleted. By removing logs of IP address connections, command ran, user logins, etc. this would make a forensic investigation extremely difficult.
On 28 December 2020 employees at Ubiquiti discovered some sort of unauthorized intrusion had occurred and began investigating. When Ubiquiti learned of the intrusion, they assigned the former employee to be part of the incident response team, not knowing that the employee was the alleged perpetrator.
On 7 January 2021 some senior employees of Ubiquiti received a ransom email from someone purporting to be the attacker. When the ransom email was analyzed, it came from an IP address associated with the Surfshark VPN service. The ransom email reportedly said that the attacker would return the stolen data and not publish it in exchange for 25 Bitcoin (approximately $2 million). Also in the email was a threat that a “backdoor” still existed in Ubiquiti’s systems and unless the ransom was paid, further exploitation would occur. The attacker also provided examples of stolen data from Ubiquiti.
Ubiquiti decided not to pay the ransom and on the deadline of the ransom payment (9 January 2021), another message was sent to Ubiquiti management that said in part, “No BTC. No talk. We done here.” The message contained a link to a publicly available repository which contained Ubiquiti private data. Ubiquiti contacted the company hosting this proprietary data and had it taken down almost immediately.
After the FBI served legal process on the Internet Service Providers who maintained the IP addresses found in the logs related to this attack, the FBI realized that in addition to Surfshark VPN, the other IP address returned to the employee’s residence. On 24 March 2021, the FBI served a search warrant at the employee’s home and seized a variety of digital devices for forensic analysis. According to the FBI, the employee gave several false statements to FBI agents.
Several days after the search warrant was served, the employee allegedly contacted media outlets claiming to be an Ubiquiti employee with insider knowledge of the breach. The employee then told the media that Ubiquiti had been hacked by an unidentified suspect and the suspect had obtained root administrator access to Ubiquiti’s AWS accounts. Media outlets began reporting on this whistleblower’s claims on 30 March 2021. One quote that was reported on that apparently came from the defendant was, “It was catastrophically worse than reported, and legal silenced and overruled efforts to decisively protect customers,” and “the breach was massive, customer data was at risk, access to customers’ devices deployed in corporations and homes around the world was at risk.”
Our next post in this series will examine the lessons learned from this insider threat attack and some mitigation strategies that could have been used to prevent and detect the malicious activity.
Building insider threat programs and conducting investigations into insider threats is a core capability of Natsar. Whether you need written policies, technical and administrative controls, a second set of eyes to review your environment, or training for your staff in this area, let us know.
At Natsar, we are continuously improving our products and services. Please let us know how we did and if this information and resources were helpful to you.