Overview – Ripple20

The list of vendors has been assembled carefully by different means and represents vendors that may be affected. The list only contains vendors that CISA ICS-CERT has listed in an internal document as having been contacted.

The status of each vendor internal investigation is supplied by CISA ICS-CERT according to vendor response. Vendors stating they are not affected are also recorded for accountability purposes and future inquiries if they are raised again as potentially having affected products.

The list of affected products can be found in the vendor advisory linked above or directly from the vendor.

The list will be updated from time to time. Any vendor that would like to report a different status or believes there is a mistake can do so through a coordination agency or by emailing [email protected]

The following vendors are affected or might be affected.
Latest update of vendors list: August 25, 2020; 03:50 am ET

Supply Chain

To effectively address the dangers of Ripple20, the widespread distribution pattern must be painstakingly traced through the complex supply chain. During the disclosure process, JSOF tracked the supply chain in creative ways, in an attempt to understand the scale of distribution and the scope of the problem. That was when we started to appreciate just how extensive the supply chain picture really was. During the disclosure process we were also notified that there are 2 additional vulnerabilities disclosed anonymously at the same time. These are part of the 19 Ripple20 vulnerabilities, and all parties involved have been handling these vulnerabilities as part of the same disclosure process, and there are no further actions for vendors to take.

We could track the supply chain trails, but we needed to work with international organizations to extend our reach within organizations and domains for which we had no access. This is why the Ripple20 disclosure process is being coordinated and overseen by multiple national computer emergency response team (CERT) organizations and regulators. We have also been collaborating with other security vendors. As can be understood from the scale of the problem, teamwork here is essential, and benefits all players.

Once Ripple20 is publicized, we are hoping to increase awareness, enabling more companies to use the information to determine for themselves if their products, or products they are using, are vulnerable.

We will be releasing a blog post dedicated to the methods we used to track the supply chain and discover its complexities.

Vulnerability: dependent on the affected industry or sectors

In the case of Ripple20, the starting point was embedded into Treck’s TCP/IP low-level Internet protocol suite library. The library could be used as-is, configured for a wide range of uses, or incorporated into a larger library. The user could buy the library in source code format and edit it extensively. It can be incorporated into the code and implanted into a wide range of device types. The original purchaser could decide to rebrand, or could be acquired by a different corporation, with the original library history lost in company archives. Over time, the original library component could become virtually unrecognizable. This is why, long after the original vulnerability was identified and patched, vulnerabilities may still remain in the field, since tracing the supply chain trail may be practically impossible.

JSOF has conducted extensive, in-depth analysis, over many months, of the vendors affected by the Treck Internet protocol library vulnerability. The first challenge we experienced was simply being able to identify the relevant vendors Vendor identity could be obscured through the intricacies of the supply chain. Even when the vendors are identified, patch implementation is complex and not always possible. Over the course of the disclosure process we found that while patching was difficult for some vendors, it could potentially be even more difficult or close to impossible for some end users to install the patches. (For example, if the library is on a separate physical component or the company that produced the component has ceased operations.)

The number of devices that contain the vulnerable code base library is only a preliminary estimate; the number may realistically be in the billions.

Disclosue From the beginning

The story of Ripple20 begins September 2019, when we decided to begin some part-time research into the Treck TCP/IP library. Within a few months, we realized there was some troubling data on potential vulnerabilities and contacted Treck to share our information as well as kick off a coordinated vulnerability disclosure (CVD) process. We understood from the beginning that Treck worked with many different vendors, with the potential to impact a large number of users. However, we simply had no idea of the scale and sheer magnitude of the situation, nor how complex the supply chain had become.

Our outreach to Treck was initially challenging. Treck is a small company serving a niche clientele, and not generally available for individual consumer approach. They also never appear to have been the target of independent security research. Eventually, we managed to track down an email contact at Treck and received an introduction. We notified them that we found a critical security vulnerability in their TCP/IP stack library, and it was our preference to work directly with them. Initially, we were faced with minimal communication. We later learned that Treck was assessing the information internally and with their legal advisors. (We also noticed litigation lawyers checking us out on LinkedIn and elsewhere).

Subsequently, we started to contact a few select vendors (such as Digi, HP, Intel, and Quadros) that we knew used Treck products and its vulnerable TCP/IP stack library, and asked them to help us make the appropriate connection with Treck. After this vendor outreach, the ball started rolling with Treck.

Working with Treck

Once Treck understood the gravity of the vulnerabilities, we worked cooperatively together, and JSOF provided them with descriptions of the vulnerabilities and some other information.

It was important to us that Treck would make sure their clients were notified of the vulnerabilities at hand. Due to NDAs and other complexities, Treck was not able to provide us with a list of its clients and users of the code library. This was our first brush with some of the supply chain challenges. As a result, Treck ended up taking the lead on the disclosure process, as they were best suited to notify their clients and provide the appropriate patches.

Interestingly, the process of remediating the vulnerability led some of Treck’s clients to renew support contracts, thus Treck seems to have ended up profiting from the situation. When these companies were notified about Ripple20 and well understood the potential risks, they soon realized the necessity of ongoing maintenance and importance of access to patches. In the end, many of the companies worked with Treck to either renew their contracts or make other arrangements. This is a thought-provoking lesson for both small and large vendors wary of facing security issues. A proactive security response approach means clients have a reason to pay for maintenance and support in order to keep the software up to date.

Standard industry practice is not to publicize a vulnerability until there is a patch available to fix it. We agreed on a standard 90-day period within which Treck would fix the vulnerabilities and notify their clients of the patch. Treck had the patch available around the end of March – 45 days before the 90-day deadline, and informed us that they would be reaching out to all affected clients to inform them of these vulnerabilities.

The disclosure was postponed twice after requests for more time came from some of the participating vendors, with some of the vendors voicing Covid-19-related delays. Out of consideration for these companies, the time period was extended from 90 days to over 120 days. Even so, some of the participating companies became difficult to deal with, as they made extra demands and some, from our perspective, seemed much more concerned with their brand’s image than with patching the vulnerabilities.

After some deliberation, June 16 was the date chosen for publicizing Ripple20. In the interim, once we understood the extent of the vulnerabilities and the sheer numbers involved, we focused on how to best use our time (120 days of patience) to identify and help to address all of the parties that could be at risk.

An international effort

Since Treck was unable to supply us with a comprehensive list of their clientele, we decided to create our own, in order to find out who was affected and ensure that everyone was notified and would have the vulnerability patched. We used a creative supply chain tracking approach, in an attempt to understand the scale of distribution and the extent of the problem. We soon began to realize and appreciate just how complex the supply chain picture really was.

It became clear to us that tracking the extent of Treck library distribution was too large for just one small team. We could track the supply chain trails, but we needed to work with international organizations to extend our reach within organizations and domains for which we had no access.

This is why the Ripple20 disclosure process is being coordinated and overseen by multiple national computer emergency response team (CERT) organizations and regulators. All are collaborating in order to reach as many affected vendors as possible before the vulnerabilities became public. Our collaborators initially included:

• The CERT Coordination Center (CERT/CC), the worldwide center for coordinating information about Internet security at Carnegie Mellon University. This is the first (and most well-known) CERT.
• The Cybersecurity and Infrastructure Security Agency (CISA), part of the Department of Homeland Security (DHS)

CERT groups focus on ways to identify and mitigate security risks. For example, they can reach a much larger target group of potential users with blast announcements, “mass-mailings” that they broadcast to a long list of participating companies to notify them of the potential vulnerability. Once users are identified, mitigation comes into play. While the best response might be to install the original Treck patch, there are many situations in which installing the original patch is not possible. CERTs work to develop alternative approaches that can be used to minimize or effectively eliminate the risk, even if patching is not an option.

JSOF assisted with coordinating the disclosure process, including providing proof of concept scripts for some of the vulnerabilities, suggesting mitigations, and providing additional lists of users of the Treck library. We tracked these users down in a variety of creative ways, including working with partners as well as some interesting open-source intelligence collections .

Working with international groups was essential in this case for another reason. Back in the 1990s, Treck collaborated with a Japanese company named Elmic Systems. Published information suggests the two companies co-developed the TCP/IP stack, though we are not sure exactly how this collaboration evolved over the years. Treck and Elmic later split apart and went their separate ways. Treck now markets this stack as Treck TCP/IP in the U.S., while Elmic Systems, now called Zuken Elmic, markets it as Kasago TCP/IP in Asia. This resulted in two separate branches of the TCP/IP stack devices – one managed by Treck and one managed by Elmic Systems – marketed in totally separate geographic areas, with no cooperation between them to the best of our knowledge, until we reported the vulnerabilities. We had tried to contact Zuken Elmic early on, but were not successful; they stopped replying after a short email correspondence. (We even tried to send some emails in Japanese!) Later on, CERT/CC was able to coordinate with JPCERT/CC, a Japanese national CERT organization, who is handling the follow-up with Elmic Systems and other affected companies in the Japanese/Asian supply chain. We have confirmed with Zuken Elmic that their TCP/IP stack is affected by some of the Treck vulnerabilities. Initial research shows Kasago to be in widespread use, providing the beginning of a completely different supply chain.

Teamwork: in everyone’s best interest

We have been collaborating with security vendors CyberMDX (medical device cybersecurity solution) and Forescout (device visibility and control). In our lab, we worked with the Treck TCP/IP library to identify the relevant network signatures. We then shared the network signatures with CyberMDX and Forescout. Each checked their own extensive client networks for these signatures, to identify more affected devices and components.

Note that this type of collaboration in the cybersecurity community benefits all participants, and is the first of its kind to our knowledge. We expanded the range of companies and devices that can be identified, while CyberMDX and Forescout are able to already provide their clients with mitigations and visibility, even before the issue is publicized. Additional companies were involved that have asked not to be named. These collaborations proved effective even though they were begun quite late in the disclosure process. We have no doubt that future collaborations of this type can be started earlier and provide immense value for all parties.

We also developed a script that companies can run themselves, to identify Treck products in their own networks. This will not be 100% effective at this stage, but can be an efficient, effective complementary approach, since it addresses the difficulty in identifying relevant users in a cloudy supply chain trail.

The bottom line is that teamwork benefits all players.