Film-Tech Forum ARCHIVE: Supermicro Motherboards hacked?

This is topic Supermicro Motherboards hacked? in forum Digital Cinema Forum at Film-Tech Forum ARCHIVE.

To visit this topic, use this URL:
https://ft-forum.com/ft/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic;f=16;t=003453

Posted by Harold Hallikainen (Member # 5405) on 10-04-2018, 10:32 AM:

I know Supermicro motherboards are commonly used in digital cinema servers. Here's an article about them.

https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies

The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies

The attack by Chinese spies reached almost 30 U.S. companies, including Amazon and Apple, by compromising America’s technology supply chain, according to extensive interviews with government and corporate sources.

In 2015, Amazon.com Inc. began quietly evaluating a startup called Elemental Technologies, a potential acquisition to help with a major expansion of its streaming video service, known today as Amazon Prime Video. Based in Portland, Ore., Elemental made software for compressing massive video files and formatting them for different devices. Its technology had helped stream the Olympic Games online, communicate with the International Space Station, and funnel drone footage to the Central Intelligence Agency. Elemental’s national security contracts weren’t the main reason for the proposed acquisition, but they fit nicely with Amazon’s government businesses, such as the highly secure cloud that Amazon Web Services (AWS) was building for the CIA.

To help with due diligence, AWS, which was overseeing the prospective acquisition, hired a third-party company to scrutinize Elemental’s security, according to one person familiar with the process. The first pass uncovered troubling issues, prompting AWS to take a closer look at Elemental’s main product: the expensive servers that customers installed in their networks to handle the video compression. These servers were assembled for Elemental by Super Micro Computer Inc., a San Jose-based company (commonly known as Supermicro) that’s also one of the world’s biggest suppliers of server motherboards, the fiberglass-mounted clusters of chips and capacitors that act as the neurons of data centers large and small. In late spring of 2015, Elemental’s staff boxed up several servers and sent them to Ontario, Canada, for the third-party security company to test, the person says.

Nested on the servers’ motherboards, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community. Elemental’s servers could be found in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships. And Elemental was just one of hundreds of Supermicro customers.

During the ensuing top-secret probe, which remains open more than three years later, investigators determined that the chips allowed the attackers to create a stealth doorway into any network that included the altered machines. Multiple people familiar with the matter say investigators found that the chips had been inserted at factories run by manufacturing subcontractors in China.

This attack was something graver than the software-based incidents the world has grown accustomed to seeing. Hardware hacks are more difficult to pull off and potentially more devastating, promising the kind of long-term, stealth access that spy agencies are willing to invest millions of dollars and many years to get.

There are two ways for spies to alter the guts of computer equipment. One, known as interdiction, consists of manipulating devices as they’re in transit from manufacturer to customer. This approach is favored by U.S. spy agencies, according to documents leaked by former National Security Agency contractor Edward Snowden. The other method involves seeding changes from the very beginning.

One country in particular has an advantage executing this kind of attack: China, which by some estimates makes 75 percent of the world’s mobile phones and 90 percent of its PCs. Still, to actually accomplish a seeding attack would mean developing a deep understanding of a product’s design, manipulating components at the factory, and ensuring that the doctored devices made it through the global logistics chain to the desired location—a feat akin to throwing a stick in the Yangtze River upstream from Shanghai and ensuring that it washes ashore in Seattle. “Having a well-done, nation-state-level hardware implant surface would be like witnessing a unicorn jumping over a rainbow,” says Joe Grand, a hardware hacker and the founder of Grand Idea Studio Inc. “Hardware is just so far off the radar, it’s almost treated like black magic.”

But that’s just what U.S. investigators found: The chips had been inserted during the manufacturing process, two officials say, by operatives from a unit of the People’s Liberation Army. In Supermicro, China’s spies appear to have found a perfect conduit for what U.S. officials now describe as the most significant supply chain attack known to have been carried out against American companies.

One official says investigators found that it eventually affected almost 30 companies, including a major bank, government contractors, and the world’s most valuable company, Apple Inc. Apple was an important Supermicro customer and had planned to order more than 30,000 of its servers in two years for a new global network of data centers. Three senior insiders at Apple say that in the summer of 2015, it, too, found malicious chips on Supermicro motherboards. Apple severed ties with Supermicro the following year, for what it described as unrelated reasons.

In emailed statements, Amazon (which announced its acquisition of Elemental in September 2015), Apple, and Supermicro disputed summaries of Bloomberg Businessweek’s reporting. “It’s untrue that AWS knew about a supply chain compromise, an issue with malicious chips, or hardware modifications when acquiring Elemental,” Amazon wrote. “On this we can be very clear: Apple has never found malicious chips, ‘hardware manipulations’ or vulnerabilities purposely planted in any server,” Apple wrote. “We remain unaware of any such investigation,” wrote a spokesman for Supermicro, Perry Hayes. The Chinese government didn’t directly address questions about manipulation of Supermicro servers, issuing a statement that read, in part, “Supply chain safety in cyberspace is an issue of common concern, and China is also a victim.” The FBI and the Office of the Director of National Intelligence, representing the CIA and NSA, declined to comment.

The companies’ denials are countered by six current and former senior national security officials, who—in conversations that began during the Obama administration and continued under the Trump administration—detailed the discovery of the chips and the government’s investigation. One of those officials and two people inside AWS provided extensive information on how the attack played out at Elemental and Amazon; the official and one of the insiders also described Amazon’s cooperation with the government investigation. In addition to the three Apple insiders, four of the six U.S. officials confirmed that Apple was a victim. In all, 17 people confirmed the manipulation of Supermicro’s hardware and other elements of the attacks. The sources were granted anonymity because of the sensitive, and in some cases classified, nature of the information.

One government official says China’s goal was long-term access to high-value corporate secrets and sensitive government networks. No consumer data is known to have been stolen.

The ramifications of the attack continue to play out. The Trump administration has made computer and networking hardware, including motherboards, a focus of its latest round of trade sanctions against China, and White House officials have made it clear they think companies will begin shifting their supply chains to other countries as a result. Such a shift might assuage officials who have been warning for years about the security of the supply chain—even though they’ve never disclosed a major reason for their concerns.

How the Hack Worked, According to U.S. Officials

Back in 2006, three engineers in Oregon had a clever idea. Demand for mobile video was about to explode, and they predicted that broadcasters would be desperate to transform programs designed to fit TV screens into the various formats needed for viewing on smartphones, laptops, and other devices. To meet the anticipated demand, the engineers started Elemental Technologies, assembling what one former adviser to the company calls a genius team to write code that would adapt the superfast graphics chips being produced for high-end video-gaming machines. The resulting software dramatically reduced the time it took to process large video files. Elemental then loaded the software onto custom-built servers emblazoned with its leprechaun-green logos.

Elemental servers sold for as much as $100,000 each, at profit margins of as high as 70 percent, according to a former adviser to the company. Two of Elemental’s biggest early clients were the Mormon church, which used the technology to beam sermons to congregations around the world, and the adult film industry, which did not.

Elemental also started working with American spy agencies. In 2009 the company announced a development partnership with In-Q-Tel Inc., the CIA’s investment arm, a deal that paved the way for Elemental servers to be used in national security missions across the U.S. government. Public documents, including the company’s own promotional materials, show that the servers have been used inside Department of Defense data centers to process drone and surveillance-camera footage, on Navy warships to transmit feeds of airborne missions, and inside government buildings to enable secure videoconferencing. NASA, both houses of Congress, and the Department of Homeland Security have also been customers. This portfolio made Elemental a target for foreign adversaries.

Supermicro had been an obvious choice to build Elemental’s servers. Headquartered north of San Jose’s airport, up a smoggy stretch of Interstate 880, the company was founded by Charles Liang, a Taiwanese engineer who attended graduate school in Texas and then moved west to start Supermicro with his wife in 1993. Silicon Valley was then embracing outsourcing, forging a pathway from Taiwanese, and later Chinese, factories to American consumers, and Liang added a comforting advantage: Supermicro’s motherboards would be engineered mostly in San Jose, close to the company’s biggest clients, even if the products were manufactured overseas.

Today, Supermicro sells more server motherboards than almost anyone else. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order server setups at banks, hedge funds, cloud computing providers, and web-hosting services, among other places. Supermicro has assembly facilities in California, the Netherlands, and Taiwan, but its motherboards—its core product—are nearly all manufactured by contractors in China.

The company’s pitch to customers hinges on unmatched customization, made possible by hundreds of full-time engineers and a catalog encompassing more than 600 designs. The majority of its workforce in San Jose is Taiwanese or Chinese, and Mandarin is the preferred language, with hanzi filling the whiteboards, according to six former employees. Chinese pastries are delivered every week, and many routine calls are done twice, once for English-only workers and again in Mandarin. The latter are more productive, according to people who’ve been on both. These overseas ties, especially the widespread use of Mandarin, would have made it easier for China to gain an understanding of Supermicro’s operations and potentially to infiltrate the company. (A U.S. official says the government’s probe is still examining whether spies were planted inside Supermicro or other American companies to aid the attack.)

With more than 900 customers in 100 countries by 2015, Supermicro offered inroads to a bountiful collection of sensitive targets. “Think of Supermicro as the Microsoft of the hardware world,” says a former U.S. intelligence official who’s studied Supermicro and its business model. “Attacking Supermicro motherboards is like attacking Windows. It’s like attacking the whole world.”

The security of the global technology supply chain had been compromised, even if consumers and most companies didn’t know it yet

Well before evidence of the attack surfaced inside the networks of U.S. companies, American intelligence sources were reporting that China’s spies had plans to introduce malicious microchips into the supply chain. The sources weren’t specific, according to a person familiar with the information they provided, and millions of motherboards are shipped into the U.S. annually. But in the first half of 2014, a different person briefed on high-level discussions says, intelligence officials went to the White House with something more concrete: China’s military was preparing to insert the chips into Supermicro motherboards bound for U.S. companies.

The specificity of the information was remarkable, but so were the challenges it posed. Issuing a broad warning to Supermicro’s customers could have crippled the company, a major American hardware maker, and it wasn’t clear from the intelligence whom the operation was targeting or what its ultimate aims were. Plus, without confirmation that anyone had been attacked, the FBI was limited in how it could respond. The White House requested periodic updates as information came in, the person familiar with the discussions says.

Apple made its discovery of suspicious chips inside Supermicro servers around May 2015, after detecting odd network activity and firmware problems, according to a person familiar with the timeline. Two of the senior Apple insiders say the company reported the incident to the FBI but kept details about what it had detected tightly held, even internally. Government investigators were still chasing clues on their own when Amazon made its discovery and gave them access to sabotaged hardware, according to one U.S. official. This created an invaluable opportunity for intelligence agencies and the FBI—by then running a full investigation led by its cyber- and counterintelligence teams—to see what the chips looked like and how they worked.

The chips on Elemental servers were designed to be as inconspicuous as possible, according to one person who saw a detailed report prepared for Amazon by its third-party security contractor, as well as a second person who saw digital photos and X-ray images of the chips incorporated into a later report prepared by Amazon’s security team. Gray or off-white in color, they looked more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment. Depending on the board model, the chips varied slightly in size, suggesting that the attackers had supplied different factories with different batches.

Officials familiar with the investigation say the primary role of implants such as these is to open doors that other attackers can go through. “Hardware attacks are about access,” as one former senior official puts it. In simplified terms, the implants on Supermicro hardware manipulated the core operating instructions that tell the server what to do as data move across a motherboard, two people familiar with the chips’ operation say. This happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow. Deviously small changes could create disastrous effects.

Since the implants were small, the amount of code they contained was small as well. But they were capable of doing two very important things: telling the device to communicate with one of several anonymous computers elsewhere on the internet that were loaded with more complex code; and preparing the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

This system could let the attackers alter how the device functioned, line by line, however they wanted, leaving no one the wiser. To understand the power that would give them, take this hypothetical example: Somewhere in the Linux operating system, which runs in many servers, is code that authorizes a user by verifying a typed password against a stored encrypted one. An implanted chip can alter part of that code so the server won’t check for a password—and presto! A secure machine is open to any and all users. A chip can also steal encryption keys for secure communications, block security updates that would neutralize the attack, and open up new pathways to the internet. Should some anomaly be noticed, it would likely be cast as an unexplained oddity. “The hardware opens whatever door it wants,” says Joe FitzPatrick, founder of Hardware Security Resources LLC, a company that trains cybersecurity professionals in hardware hacking techniques.

U.S. officials had caught China experimenting with hardware tampering before, but they’d never seen anything of this scale and ambition. The security of the global technology supply chain had been compromised, even if consumers and most companies didn’t know it yet. What remained for investigators to learn was how the attackers had so thoroughly infiltrated Supermicro’s production process—and how many doors they’d opened into American targets.

Unlike software-based hacks, hardware manipulation creates a real-world trail. Components leave a wake of shipping manifests and invoices. Boards have serial numbers that trace to specific factories. To track the corrupted chips to their source, U.S. intelligence agencies began following Supermicro’s serpentine supply chain in reverse, a person briefed on evidence gathered during the probe says.

As recently as 2016, according to DigiTimes, a news site specializing in supply chain research, Supermicro had three primary manufacturers constructing its motherboards, two headquartered in Taiwan and one in Shanghai. When such suppliers are choked with big orders, they sometimes parcel out work to subcontractors. In order to get further down the trail, U.S. spy agencies drew on the prodigious tools at their disposal. They sifted through communications intercepts, tapped informants in Taiwan and China, even tracked key individuals through their phones, according to the person briefed on evidence gathered during the probe. Eventually, that person says, they traced the malicious chips to four subcontracting factories that had been building Supermicro motherboards for at least two years.

As the agents monitored interactions among Chinese officials, motherboard manufacturers, and middlemen, they glimpsed how the seeding process worked. In some cases, plant managers were approached by people who claimed to represent Supermicro or who held positions suggesting a connection to the government. The middlemen would request changes to the motherboards’ original designs, initially offering bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery of the chips to the factories.

The investigators concluded that this intricate scheme was the work of a People’s Liberation Army unit specializing in hardware attacks, according to two people briefed on its activities. The existence of this group has never been revealed before, but one official says, “We’ve been tracking these guys for longer than we’d like to admit.” The unit is believed to focus on high-priority targets, including advanced commercial technology and the computers of rival militaries. In past attacks, it targeted the designs for high-performance computer chips and computing systems of large U.S. internet providers.

Provided details of Businessweek’s reporting, China’s Ministry of Foreign Affairs sent a statement that said “China is a resolute defender of cybersecurity.” The ministry added that in 2011, China proposed international guarantees on hardware security along with other members of the Shanghai Cooperation Organization, a regional security body. The statement concluded, “We hope parties make less gratuitous accusations and suspicions but conduct more constructive talk and collaboration so that we can work together in building a peaceful, safe, open, cooperative and orderly cyberspace.”

The Supermicro attack was on another order entirely from earlier episodes attributed to the PLA. It threatened to have reached a dizzying array of end users, with some vital ones in the mix. Apple, for its part, has used Supermicro hardware in its data centers sporadically for years, but the relationship intensified after 2013, when Apple acquired a startup called Topsy Labs, which created superfast technology for indexing and searching vast troves of internet content. By 2014, the startup was put to work building small data centers in or near major global cities. This project, known internally as Ledbelly, was designed to make the search function for Apple’s voice assistant, Siri, faster, according to the three senior Apple insiders.

Documents seen by Businessweek show that in 2014, Apple planned to order more than 6,000 Supermicro servers for installation in 17 locations, including Amsterdam, Chicago, Hong Kong, Los Angeles, New York, San Jose, Singapore, and Tokyo, plus 4,000 servers for its existing North Carolina and Oregon data centers. Those orders were supposed to double, to 20,000, by 2015. Ledbelly made Apple an important Supermicro customer at the exact same time the PLA was found to be manipulating the vendor’s hardware.

Project delays and early performance problems meant that around 7,000 Supermicro servers were humming in Apple’s network by the time the company’s security team found the added chips. Because Apple didn’t, according to a U.S. official, provide government investigators with access to its facilities or the tampered hardware, the extent of the attack there remained outside their view.

American investigators eventually figured out who else had been hit. Since the implanted chips were designed to ping anonymous computers on the internet for further instructions, operatives could hack those computers to identify others who’d been affected. Although the investigators couldn’t be sure they’d found every victim, a person familiar with the U.S. probe says they ultimately concluded that the number was almost 30 companies.

That left the question of whom to notify and how. U.S. officials had been warning for years that hardware made by two Chinese telecommunications giants, Huawei Corp. and ZTE Corp., was subject to Chinese government manipulation. (Both Huawei and ZTE have said no such tampering has occurred.) But a similar public alert regarding a U.S. company was out of the question. Instead, officials reached out to a small number of important Supermicro customers. One executive of a large web-hosting company says the message he took away from the exchange was clear: Supermicro’s hardware couldn’t be trusted. “That’s been the nudge to everyone—get that crap out,” the person says.

Amazon, for its part, began acquisition talks with an Elemental competitor, but according to one person familiar with Amazon’s deliberations, it reversed course in the summer of 2015 after learning that Elemental’s board was nearing a deal with another buyer. Amazon announced its acquisition of Elemental in September 2015, in a transaction whose value one person familiar with the deal places at $350 million. Multiple sources say that Amazon intended to move Elemental’s software to AWS’s cloud, whose chips, motherboards, and servers are typically designed in-house and built by factories that Amazon contracts from directly.

A notable exception was AWS’s data centers inside China, which were filled with Supermicro-built servers, according to two people with knowledge of AWS’s operations there. Mindful of the Elemental findings, Amazon’s security team conducted its own investigation into AWS’s Beijing facilities and found altered motherboards there as well, including more sophisticated designs than they’d previously encountered. In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips. That generation of chips was smaller than a sharpened pencil tip, the person says. (Amazon denies that AWS knew of servers found in China containing malicious chips.)

China has long been known to monitor banks, manufacturers, and ordinary citizens on its own soil, and the main customers of AWS’s China cloud were domestic companies or foreign entities with operations there. Still, the fact that the country appeared to be conducting those operations inside Amazon’s cloud presented the company with a Gordian knot. Its security team determined that it would be difficult to quietly remove the equipment and that, even if they could devise a way, doing so would alert the attackers that the chips had been found, according to a person familiar with the company’s probe. Instead, the team developed a method of monitoring the chips. In the ensuing months, they detected brief check-in communications between the attackers and the sabotaged servers but didn’t see any attempts to remove data. That likely meant either that the attackers were saving the chips for a later operation or that they’d infiltrated other parts of the network before the monitoring began. Neither possibility was reassuring.

When in 2016 the Chinese government was about to pass a new cybersecurity law—seen by many outside the country as a pretext to give authorities wider access to sensitive data—Amazon decided to act, the person familiar with the company’s probe says. In August it transferred operational control of its Beijing data center to its local partner, Beijing Sinnet, a move the companies said was needed to comply with the incoming law. The following November, Amazon sold the entire infrastructure to Beijing Sinnet for about $300 million. The person familiar with Amazon’s probe casts the sale as a choice to “hack off the diseased limb.”

As for Apple, one of the three senior insiders says that in the summer of 2015, a few weeks after it identified the malicious chips, the company started removing all Supermicro servers from its data centers, a process Apple referred to internally as “going to zero.” Every Supermicro server, all 7,000 or so, was replaced in a matter of weeks, the senior insider says. (Apple denies that any servers were removed.) In 2016, Apple informed Supermicro that it was severing their relationship entirely—a decision a spokesman for Apple ascribed in response to Businessweek’s questions to an unrelated and relatively minor security incident.

That August, Supermicro’s CEO, Liang, revealed that the company had lost two major customers. Although he didn’t name them, one was later identified in news reports as Apple. He blamed competition, but his explanation was vague. “When customers asked for lower price, our people did not respond quickly enough,” he said on a conference call with analysts. Hayes, the Supermicro spokesman, says the company has never been notified of the existence of malicious chips on its motherboards by either customers or U.S. law enforcement.

Concurrent with the illicit chips’ discovery in 2015 and the unfolding investigation, Supermicro has been plagued by an accounting problem, which the company characterizes as an issue related to the timing of certain revenue recognition. After missing two deadlines to file quarterly and annual reports required by regulators, Supermicro was delisted from the Nasdaq on Aug. 23 of this year. It marked an extraordinary stumble for a company whose annual revenue had risen sharply in the previous four years, from a reported $1.5 billion in 2014 to a projected $3.2 billion this year.

One Friday in late September 2015, President Barack Obama and Chinese President Xi Jinping appeared together at the White House for an hourlong press conference headlined by a landmark deal on cybersecurity. After months of negotiations, the U.S. had extracted from China a grand promise: It would no longer support the theft by hackers of U.S. intellectual property to benefit Chinese companies. Left out of those pronouncements, according to a person familiar with discussions among senior officials across the U.S. government, was the White House’s deep concern that China was willing to offer this concession because it was already developing far more advanced and surreptitious forms of hacking founded on its near monopoly of the technology supply chain.

In the weeks after the agreement was announced, the U.S. government quietly raised the alarm with several dozen tech executives and investors at a small, invite-only meeting in McLean, Va., organized by the Pentagon. According to someone who was present, Defense Department officials briefed the technologists on a recent attack and asked them to think about creating commercial products that could detect hardware implants. Attendees weren’t told the name of the hardware maker involved, but it was clear to at least some in the room that it was Supermicro, the person says.

The problem under discussion wasn’t just technological. It spoke to decisions made decades ago to send advanced production work to Southeast Asia. In the intervening years, low-cost Chinese manufacturing had come to underpin the business models of many of America’s largest technology companies. Early on, Apple, for instance, made many of its most sophisticated electronics domestically. Then in 1992, it closed a state-of-the-art plant for motherboard and computer assembly in Fremont, Calif., and sent much of that work overseas.

Over the decades, the security of the supply chain became an article of faith despite repeated warnings by Western officials. A belief formed that China was unlikely to jeopardize its position as workshop to the world by letting its spies meddle in its factories. That left the decision about where to build commercial systems resting largely on where capacity was greatest and cheapest. “You end up with a classic Satan’s bargain,” one former U.S. official says. “You can have less supply than you want and guarantee it’s secure, or you can have the supply you need, but there will be risk. Every organization has accepted the second proposition.”

In the three years since the briefing in McLean, no commercially viable way to detect attacks like the one on Supermicro’s motherboards has emerged—or has looked likely to emerge. Few companies have the resources of Apple and Amazon, and it took some luck even for them to spot the problem. “This stuff is at the cutting edge of the cutting edge, and there is no easy technological solution,” one of the people present in McLean says. “You have to invest in things that the world wants. You cannot invest in things that the world is not ready to accept yet.”

Bloomberg LP has been a Supermicro customer. According to a Bloomberg LP spokesperson, the company has found no evidence to suggest that it has been affected by the hardware issues raised in the article.

Posted by Leo Enticknap (Member # 534) on 10-04-2018, 12:48 PM:

I wonder if DCI will be worried by this, given that China is frequently accused of being behind a large proportion of movie piracy.

Posted by Carsten Kurz (Member # 5396) on 10-04-2018, 03:05 PM:

As long as no one super-secretly puts super-secret rice grains on media block boards, we should be safe. If that story wasn't so longish, it would sound as if Donald Trump had made one up again.

- Carsten

Posted by Mark Gulbrandsen (Member # 72) on 10-04-2018, 04:06 PM:

I think it has way more to do with pirating Eye phones than it does server boards. Eye phone clones are all the rage in Asia.

Mark

Posted by Leo Enticknap (Member # 534) on 10-04-2018, 04:31 PM:

Whoever planted these chips is more interested in the data that passes through the motherboards to which they are attached, or is on drives connected locally to them, rather than the motherboards themselves. As Mark points out, that could very well include iPhones, given that Apple was targeted.

And agreed, unless something similar has been done to media blocks, IMBs, or IMSes (which presumably are manufactured in the west under tight security conditions, not outsourced to Chinese factories), decrypted movie data cannot be compromised by the use of a hacked motherboard in a server.

Furthermore, the compromised motherboards presumably still wouldn't pose a security risk if used within a properly secured network infrastructure, because they would try to "phone home" for instructions on a port that would be closed at a firewall downstream from the server.

Still, the potential for paranoia among those who don't understand these concepts remains significant.

Posted by Steve Guttag (Member # 268) on 10-05-2018, 07:12 AM:

My suspicion is that there is a path for most servers to get to the internet in our industry. Typical systems have two networks. The media and the management and, as such, most servers have, at least, two NICs to allow this scheme. The management NIC allows control, monitoring and TMS commands to be carried out. It is this port that I suspect has internet access in MOST systems (not all and certainly is not required to have internet access).

I don't know just what harmful information can be had on the server activity from that and the content is encrypted so there is no compromise there.

Perhaps they can tell all of the servers to shutdown remotely and cause us to lose movies for a day?

Posted by Mark Gulbrandsen (Member # 72) on 10-05-2018, 08:05 AM:

Newer Dell servers starting with the 710 actually have four networks on the mommie board. I believe all HP's do as well.

A device like the one in the article would be way more advantageous planted on the I/O of a major network switch than on a single server. On a network switch they could literally mine gobs of data per hour, or everything passing through the switch.

Mark

Posted by Marcel Birgelen (Member # 6801) on 10-05-2018, 08:38 AM:

The potential backdoor has been placed on the BMC/IPMI component, as far as I can account for, none of the currently common SuperMicro boards being used in DCI do have a BMC module installed, so I guess the risk in this field is pretty minimal.

Like Carsten already noticed, even if the "chip-tuning" did affect DCI servers, chances of them getting hold of a key are still pretty minimal. The key should only ever exist in unencrypted form inside the mediablock.

Still, it's pretty shocking to see how far our governments are willing to go those days. Ironically, it will be new ammo for another round of finger pointing. In the end, those things almost never stay secret forever and too much of this might or actually should hurt China's bottom line...

Posted by Dave Macaulay (Member # 813) on 10-05-2018, 09:37 AM:

Everyone involved is denying this ever happened, using pretty specific language that would lead to serious repercussions if they're lying. Amazon and Supermicro would be making false statements that affect their share prices and the SCC does not like that at all - look at Musk.
I'm not saying this could not happen or that it didn't. But there are easier ways to do it without adding secret hardware. The BMC and pre-boot software can be modified.
Also, these boards are not what any cinema server uses. None currently available even have a BMC network jack, this is a network vital to datacenters to manage server farms - not useful in a cinema really. With a few hundred or thousand servers - even if one has crashed, since the BMC is outside of the OS - you can reboot, update, and manage them from a quiet office rather than go out with a ladder and KVM, find the one with issues, and work on it there.

Posted by Mark Gulbrandsen (Member # 72) on 10-05-2018, 03:37 PM:

Note that not ALL Supermicro stuff is manufactured in China.

Posted by Marcel Birgelen (Member # 6801) on 10-06-2018, 02:31 AM:

AFAIK most of their prints, including the motherboards are made in Taiwan, the chassis and PSUs are mostly built in China.

They also have manufacturing plants in California and the Netherlands, not sure what they build there though.

It's an American company, it would be quite a big deal if they really followed suit with a secret order of the Chinese government to backdoor their equipment. Making it work effectively in a chip the size of a rice grain is also pretty interesting and by far not as trivial as it might sound.

Posted by Harold Hallikainen (Member # 5405) on 10-06-2018, 12:57 PM:

https://www.eetimes.com/document.asp?doc_id=1333831&_mc=RSS_EET_EDT&utm_source=newsletter&utm_campaign=link&utm_medium=EETimesWeekInReview-20181006

Apple, Amazon Refute China Chip Hacking Story
By Dylan McGrath, 10.04.18

SAN DIEGO — Tech giants Apple and Amazon are denying a report by Bloomberg Businessweek that they are among nearly 30 companies that had their hardware compromised by Chinese spies that allegedly implanted tiny microchips for the purpose of accessing their networks.

Both Apple and Amazon, as well as server vendor Super Micro Inc., issued statements Thursday that strongly refuted the report, which said the attacks were first discovered in 2015.

The Bloomberg report — based on interviews with 17 people, including two Amazon Web Services insiders, three Apple insiders and six U.S. government officials — the U.S. is still conducting a top-secret investigation of the incidents more than three years later. Investigators have determined that the chips let attackers create a stealth doorway into any network that included the compromised servers, according to the report.

Sources told Bloomberg that the chips — about the size of a grain of rice and not part of the servers' original design — were inserted at factories run by manufacturing subcontractors in China. The attacks were made on servers sold by Supermicro, according to the report.

The report comes at a time when tensions between the U.S. and China are high. The world's two largest economies are currently embroiled in a trade war, and the U.S. has accused China of involvement in hacking and cybercrimes, including the recent suggestion by U.S. President Donald Trump that China is interfering in the U.S. election process.

Most allegations of espionage-related security breaches involve remote hackers gaining access to networks and systems through operating systems and other software. The allegations in the Businessweek story are unusual because they involve the physical placement of an IC on a board by a government agency.

Super Micro, based in San Jose, Calif., said it "strongly refutes reports that servers it sold to customers contained malicious microchips in the motherboards of those systems."

Super Micro went on to say it "has never found any malicious chips, nor been informed by any customer that such chips have been found."

Apple said it has been contacted by Bloomberg multiple times with claims of alleged security incidents. The company said it has conducted rigorous internal investigations which have not yielded any evidence to support the claims.

"On this we can be very clear: Apple has never found malicious chips, 'hardware manipulations' or vulnerabilities purposely planted in any server," Apple said in a statement "Apple never had any contact with the FBI or any other agency about such an incident. We are not aware of any investigation by the FBI, nor are our contacts in law enforcement."

Amazon said there were "many inaccuracies" in the Bloomberg article. The article stated that Amazon discovered the implanted chips when it was working with Elemental Technologies — which it later acquired — in 2015, after Elemental sent several servers to a third-party tech security firm, which discovered the tiny chips.

Amazon said Thursday that the third-party's report did not identify any issues with modified chips or hardware, but instead made some typical recommendations for shoring up security that Amazon implemented prior to finalizing the deal to buy Elemental. Bloomberg, Amazon said, has admittedly never seen the results of the report from the third-party vendor.

"Amazon employs stringent security standards across our supply chain – investigating all hardware and software prior to going into production and performing regular security audits internally and with our supply chain partners," Amazon said in a statement. "We further strengthen our security posture by implementing our own hardware designs for critical components such as processors, servers, storage systems, and networking equipment."

Posted by Leo Enticknap (Member # 534) on 10-06-2018, 02:15 PM:

quote: Dave Macaulay
I'm not saying this could not happen or that it didn't. But there are easier ways to do it without adding secret hardware. The BMC and pre-boot software can be modified.

I've heard of approaches that involve modifying UEFI code, too.

quote: Marcel Birgelen
It's an American company, it would be quite a big deal if they really followed suit with a secret order of the Chinese government to backdoor their equipment.

My understanding is that Supermicro themselves are not alleged to have done that, but rather one of their Chinese subcontractors.

Posted by Mark Gulbrandsen (Member # 72) on 10-06-2018, 02:38 PM:

quote: Leo Enticknap
My understanding is that Supermicro themselves are not alleged to have done that, but rather one of their Chinese subcontractors.

I think that goes without saying. NO company is going to set themselves up for this can of worms!

Posted by Mark Gulbrandsen (Member # 72) on 10-10-2018, 04:26 PM:

More new evidence of this occurance...

https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found-in-u-s-telecom

A major U.S. telecommunications company discovered manipulated hardware from Super Micro Computer Inc. in its network and removed it in August, fresh evidence of tampering in China of critical technology components bound for the U.S., according to a security expert working for the telecom company.

The security expert, Yossi Appleboum, provided documents, analysis and other evidence of the discovery following the publication of an investigative report in Bloomberg Businessweek that detailed how China’s intelligence services had ordered subcontractors to plant malicious chips in Supermicro server motherboards over a two-year period ending in 2015.

Yossi AppleboumSource: Yossi Appleboum
Appleboum previously worked in the technology unit of the Israeli Army Intelligence Corps and is now co-chief executive officer of Sepio Systems in Gaithersburg, Maryland. His firm specializes in hardware security and was hired to scan several large data centers belonging to the telecommunications company. Bloomberg is not identifying the company due to Appleboum’s nondisclosure agreement with the client. Unusual communications from a Supermicro server and a subsequent physical inspection revealed an implant built into the server’s Ethernet connector, a component that's used to attach network cables to the computer, Appleboum said.

Posted by Leo Enticknap (Member # 534) on 10-12-2018, 11:59 AM:

Our IT guy at work pointed out something interesting: that none of the coverage of this story has cited any actual model numbers of motherboards said to be affected.

Posted by Marcel Birgelen (Member # 6801) on 10-13-2018, 02:32 AM:

Maybe I didn't really look good enough, but I even haven't seen a single picture yet of how this chip is supposed to look like.

Since it's apparently a microchip with quite some capabilities, it should have more connections than just a simple SMD diode or resistor...

I do have a few SuperMicro machines with BMC module at my disposal with varying build dates. Would be interesting to take a look at them, if I knew what I was looking for...

Posted by Carsten Kurz (Member # 5396) on 10-13-2018, 04:57 AM:

It still appears to be a bogus story to me. Everyone involved denies this happened - except for Bloomberg. The only way to lead out traffic on this machines is by ethernet. That is traffic that can usually be recognized by established security tools.

Marcel - it does not necessarily need to be a chip with a higher function. It could be as simple as an SMD resistor enabling a special function in the network chip.

- Carsten

Posted by Randy Stankey (Member # 64) on 10-14-2018, 10:22 AM:

When I first heard this story, I was pretty shocked but, as time goes on and I have had a chance to think about it, this story just doesn't pass the smell test with me.

It would be a pretty huge undertaking to pull off an operation like this:

First, somebody would have to obtain the schematic diagrams for the circuit boards in question. That information would not ordinarily be released to a semiconductor assembly plant. The whole board would have to be reverse engineered if you don't have the schematics. That's not something that "just anybody" can do.

Then, the board would have to be redesigned. It might be as simple as adding one component but somebody would have to test the new design to be sure that it works as intended and remains undetectable. Again, "just anybody" would not be able to do this.

Next, somebody would have to get inside the PC Board manufacturing plant ("Board House") and substitute all the faked printing/etching masks for the real ones and get them into production. Most electronics assembly plants don't design and build their own boards. They usually get shipped in from a separate plant.

Then, somebody has to get inside the SMT Assembly plant and redesign ALL of the tooling to make these boards. The SMT (paste) stencils, the SMT "Pick-and-Place" machines and several Automated Optical Inspection programs would need to be changed... all undetected.

You would need to test all of these boards once them are made and all of the testing procedures would have to be re-created, too. Plus, the added chips/circuits would all have to be tested in secret.

You would have to do all of this, do it in secret while hundreds of people are involved in the manufacturing process and it all has to be done in a time frame that meets the customer's schedule.

That's a HUGE undertaking and I believe it would be a daunting task even for a government to do in the time frame we are talking about and it's difficult for me to believe that even the Chinese Government could pull this off and keep it secret for so long.

Posted by Harold Hallikainen (Member # 5405) on 10-14-2018, 12:58 PM:

It IS interesting that only Bloomberg is reporting this. Further, just because something is very difficult does not mean someone will not do it. Here's an article discussing ways this could be accomplished ( https://www.electronicdesign.com/embedded-revolution/how-hack-server-motherboard ):

How to Hack a Server Motherboard

Bloomberg Businessweek reported on an alleged hardware attack by China that impacted server motherboards. This is how it might work.

William Wong | Oct 10, 2018

Bloomberg Businessweek’s “The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies” article alleges Chinese-based corruption of the supply chain by adding a chip to the motherboard (see figure). This is likely replacing or intercepting communication from the baseboard management controller (BMC) and the serial flash-memory chip that holds the BMC’s code. One of the most common BMC chip families comes from Aspeed, a Chinese vendor. A number of chips populate this family, and they’re used on the motherboards in question.

The BMC is a typical Arm-based system-on-chip (SoC). It has on-board peripherals and some memory, but it can utilize off-chip memory as well. Often the boot code is contained in a serial flash device or a parallel NAND flash device.

What’s the Best Method to Fix the Problem?

In theory, it would be easy to replace the application in flash memory with compromised code, but this approach has a problem. The flash memory is often reprogrammed by the vendor when it receives the motherboard to allow for delivery of the hardware with the latest code. This is often done using a connection directly to the memory rather than booting the system and programming in new software, although that’s a possible method as well. However, it could be circumvented by a compromised system.

Another approach would be to replace the memory chip with a custom version that included compromised code. The challenge with this approach it twofold. First, creating such a chip is a major undertaking. Second, the size and capacity of the chip would be limited by the chip it replaced. This may be less of an issue given the size reductions available using new technology.

The baseboard management controller (BMC) is an SoC that can boot from off-chip flash or serial memory. Interposing a chip between the serial memory would be one way to take control of the system.

The third, and evidently the chosen approach, is to add a custom chip between the serial memory and the BMC. This has the advantage of not changing the flash memory, but it does mean creating a new chip which, as noted, isn’t an easy task. It could be a relatively simple chip, and it’s even possible to use an existing microcontroller.

The challenge is that the traces on the motherboard for the serial memory would have to be altered. According to Bloomberg, contractors building the boards and populating them were coerced to make these additions. The small size of the chip and minimal alterations would prevent a casual observer from noting a change. Hiding the chip near or under other hardware makes that task even more difficult.

The advantage of this approach is it would work with a wider range of motherboards. The attacker’s chip could be programmed to handle different BMC chips. Also, the amount of code that the attacker needs to include on their chip doesn’t have to be large since it can take advantage of the code that the BMC chip would load from the flash memory on boot. Attackers simply need to modify the code to suit their needs. The existing code is likely to include a small RTOS, communication support, etc., that could be exploited. Such an approach isn’t easy to implement but very practical.

There have been more details revealed lately and it appears that the motherboard circuit board did not have to be modified. Likewise, the additional chip may simply be a standard serial memory chip that was added to a location designed for the chip and left unpopulated. This is a common design approach to provide more options. For example, a TPM security chip is often an option for a server motherboard. The chip is simply left out if the motherboard will not provide that option.

Leaving out a single chip is common, but so is leaving entire sections of a printed circuit board (PCB) unpopulated. It would be impossible for someone without a circuit diagram and bill of materials to determine what should or should not be on a PCB.

The hack was supposedly caught, not by observing the changes to the motherboard, but by network traffic that was abnormal. A more sophisticated implementation might delay compromised communication until much later making it much harder to detect.

BMC at the Center of It All

The BMC is normally tied into one or two Ethernet ports. Normally one is designed to be connected to a dedicated administration network that’s often isolated so that traffic can’t get on the internet. Network managers are able to use this network to manage a server farm; the operating systems and applications that run on the server can’t even detect that this is being done. A BMC typically provides simulated serial ports and disk drives that are indistinguishable from the real thing. They can also control and modify the boot memory for the main processor.

A second BMC port is often piggybacked onto a network port used by the main processor. It would be connected to a more public network or the internet. This “feature” allows the public network to be used for administration, simplifying the operation of small networks by not requiring a parallel network for each server. Encrypted communication can prevent attacks even in this case, but it’s not the same thing as an isolated network. Of course, the compromised BMC could also gain control of the main processor, though that’s a much harder programming chore. Compromising the BMC is much easier.

The BMC has control of both ports; therefore, a compromised system could use the public connection even while the BMC application was using the administration network in the normal fashion. The BMC will need an IP address on the public network in addition to the one needed for the processor, so that could be a way of detecting a problem assuming the compromised system would operate in such a way.

Some SoCs are designed to address this type of attack, but most are not. Essentially, each SoC would need to have its own crypto identification that’s used to verify or decrypt boot code. This would prevent booting of a compromised system. It typically requires a signed version of code that’s unique to an SoC.

The advantage of this approach is that the SoC creation process needs to be secure, which tends to be much easier to manage than the assembly of boards, chips, and other hardware to create a server or other device. Unfortunately, this approach is relatively new, not available to most OEMs, and not used by most vendors—yet.

Antivirus software, even something implemented in the processor’s boot code, is incapable of detecting or preventing an attack like this from compromising the system.

If the attack winds up being real, then it could force many companies to reevaluate all aspects of their supply chain. It might be a good idea to do that anyway, because even if this instance was a hoax, the next one may not be. The method of attack is valid, although it’s difficult to implement unless one has influence over part of the supply chain. Still, there are many ways to do this.

System security is getting much better in general with features like secure boot—if they’re used. Attackers will likely continue to exploit holes or attack targets such as the BMC that bypass normal security measures. It simply means that security needs to be applied to all aspects of system design, deployment, and even during the building of a system.

Posted by Harold Hallikainen (Member # 5405) on 10-23-2018, 11:46 AM:

https://www.reuters.com/article/us-china-cyber-super-micro-comp/super-micro-to-review-hardware-for-malicious-chips-idUSKCN1MW1GK

Super Micro to review hardware for malicious chips
Sonam Rai

(Reuters) - Computer hardware maker Super Micro Computer Inc said on Monday it would review its motherboards for any proof of malicious chips as alleged in a recent media report.

“Despite the lack of any proof that a malicious hardware chip exists, we are undertaking a complicated and time-consuming review to further address the article,” the server and storage manufacturer said in a letter to its customers, dated Oct. 18.

Shares of the San Jose, California-based company rose 4.3 percent to $14.70 on Monday.

A Bloomberg report on Oct. 4 cited 17 unidentified sources from intelligence agencies and businesses that claimed Chinese spies had placed computer chips inside equipment used by about 30 companies, including Apple Inc and Amazon.com Inc and multiple U.S. government agencies, which would give Beijing secret access to internal networks.

Super Micro denied the allegations made in the report.

The company said the design complexity makes it practically impossible to insert a functional, unauthorized component onto a motherboard without it being caught by the checks in its manufacturing and assembly process.

It is entirely plausible that a malicious chip can be placed on a motherboard but it will be at a very high cost, and the risk of detection increases with every such chip in the field, said Jake Williams, a former National Security Agency analyst and founder of the cyber security firm Rendition Infosec.

“This technique would only be used for high value targets that couldn’t be easily compromised via another attack vector,” Williams said.

The Bloomberg report also said Apple in 2015 had found malicious chips on Super Micro motherboards and added that Amazon uncovered such chips the same year while examining servers made by Elemental Technologies, which Amazon eventually acquired.

Both Apple and Amazon have denied the allegations. Apple Chief Executive Officer Tim Cook told online news website BuzzFeed on Friday that Bloomberg should retract the story.

Amazon Web Services CEO Andy Jassy also joined Cook in asking Bloomberg to retract the report.

“Bloomberg story is wrong about Amazon, too ... Reporters got played or took liberties. Bloomberg should retract,” Jassy said in a tweet on Monday.

Bloomberg had previously said it stood by its report and was confident of its reporting, which was conducted for more than a year.

Security experts as well as the U.S. and U.K. authorities have said they had no knowledge of the attacks.

Reporting by Sonam Rai in Bengaluru; Editing by Arun Koyyur and Anil D'Silva