Is An American Company’s Technology Helping Turkey Spy On Its Citizens?

Founded in 2002, Procera’s headquarters are in Fremont, though large chunks of its development work is done in Canada and Sweden, the latter serving deep packet inspection to Europe and the Middle East. In mid-2015, Francisco Partners, a private equity firm with $10 billion in assets, acquired Procera for $240 million. A new CEO, Lyndon Cantor, was installed at the top to drive Procera through the “next chapter in its strategic development,” according to a company press release, as the executive team was given a refresh. The changes rankled some of Procera’s left-leaning employees. One former employee told FORBES that the acquisition by Francisco Partners led to greater focus on “regulatory compliance… mostly bulk surveillance.” Another claimed: “When Francisco Partners took control it was business ethics that mattered, not human ethics.”

In August, already-suspicious engineers grew more concerned. Researchers from the University of Toronto’s Citizen Lab and mobile security firm Lookout raised questions about the ethics of another Francisco Partners portfolio company, NSO Group, a government spyware provider founded by an alum of Israel’s vaunted intelligence agencies. (Francisco Partners bought its stake in the company for $120 million in 2014). Citizen Lab uncovered NSO’s Pegasus malware targeting iPhones of a Mexican journalist and a UAE activist. The same day, FORBES reported that Francisco Partners added Circles to its roster of investments, another Israeli-founded surveillance firm, which sold contentious gear to hack a part of global telecoms networks, known as SS7. That cost the private equity firm $130 million, a source close to the deal told FORBES.

In a statement, a Francisco Partners spokesperson refuted the criticisms: “Having invested in over 80 technology companies, we have demonstrated in our role as board members a nearly 20-year history of working with management teams on practicing appropriate corporate social responsibility and adhering to legal and ethical standards. This includes supporting management teams’ commitments to make every effort to ensure their products are used legally, responsibly and ethically by their customers.”

‘Shocking’ surveillance in Turkey

After they learned of the username and password feature shipping through Sekom to Turk Telekom, Procera engineers feared they would in effect be supporting Turkey’s surveillance state, whose actions have come under increased criticism from human rights groups. There have been plenty of disturbing cases: a 14-year-old in prison after criticizing Erdoğan in a Facebook post, a doctor on trial after a meme he produced compared Erdoğan to Lord of the Rings character Gollum. After the failed coup this summer, the assault on dissents has only intensified as Turkey enacted what Reporters Without Borders (RSF) called “draconian” state of emergency laws. Any individual or organization deemed to have any connection to Fethullah Gülen, a Turkish cleric in exile in the US whom Erdoğan believes masterminded the coup, faces persecution. In July, 15,200 Ministry of Education personnel were suspended and faced investigation, 1,577 university deans were asked to resign, and 2,277 judges and prosecutors were detained, all because of alleged connections to Gülen. Amnesty reported credible sources as claiming some of those detained were subjected to torture and rape. The state of emergency was, this October, extended for a further three months.

“These are executive orders that should be under scrutiny, but they are rubber stamped by judges and there’s no practical way to appeal these decisions,” said Andrew Gardner, Amnesty International researcher for Turkey. Gardner was the target of an unsuccessful order to force Twitter to block his account — Twitter’s most recent transparency report showed Erdoğan’s regime lodged 2,493 requests for content to be removed between January and June this year, more than any other country.

It’s no wonder then that the idea of an American company supplying services that appeared to support Turkish surveillance caused so much concern, not only inside Procera but also among human rights and privacy advocates. “To have the power for password extraction at the network level is a quite shocking capability for any government to have, let alone Turkey where the respect for fundamental rights has taken a stark downturn recently,” said Matthew Rice, advocacy officer at Privacy International, a not-for-profit organization. “Everyone should be concerned not only that this capability was requested, but that it was provided… This work was unprecedented for not only Procera, but for the surveillance industry as a whole.”

‘NSA-grade tech’

Two security experts compared the feature Procera sold to a weapon in the National Security Agency (NSA) arsenal. Nicholas Weaver, senior staff researcher focusing on computer security at the International Computer Science Institute in Berkeley, told FORBES Procera’s capabilities were similar to those of a core function of the intelligence agency’s XKEYSCORE software. According to files leaked by NSA whistleblower Edward Snowden, XKEYSCORE kept a constant monitor on internet traffic and siphoned off data of interest, including usernames and passwords. “That’s XKEYSCORE 101,” Weaver said of the Procera sale.

Morgan Marquis-Boire, an ex-Google security staffer, Citizen Lab senior researcher and the man in charge of protecting First Look Media, said he expected nations to buy into NSA-esque tech. “It’s not surprising to me that Procera has been found adding this capability to their existing solutions given what XKEYSCORE can accomplish,” he told me after reviewing some of the leaked documents from Procera. “It stands to reason that the intelligence operations of many countries would see this as desirable.”

Sources’ description of the Procera Turk Telekom project noted the former’s PacketLogic tool would monitor connections and redirect traffic of interest – e.g. unencrypted logins – to another product, the Network Application Visibility Library (NAVL). That would probe the data packets further to retrieve usernames and passwords across Turk Telekom, which boasts 18 million mobile and 8.3 million broadband customers. It’s one of the largest telecoms providers in Turkey, said to run 80 per cent of the country’s fibre optics network and run the biggest ISP in the form of TTNet. Once an entirely state-owned asset, it’s now private, though the Turkish Treasury still holds a 30 per cent stake.

It could be argued the export of such a service would not cause much harm; most social media, email and security-critical sites are run over HTTPS, where connections between a computer or smartphone and a website are encrypted. But according to Google data, of the most-visited 100 non-Google sites on the web, 60 do not run HTTPS by default. Most news sites – from FORBES, to The New York Times, the BBC and Turkey’s two most-read publications Hurriyet Daily News and the Daily Sabah – do not use HTTPS. At the same time, password re-use is prevalent. In a LastPass survey of 2,000 adults conducted this year, 61 per cent said they either use the same or similar passwords across websites. Thanks to those security weaknesses, hackers, be they government or criminal, can harvest passwords from unencrypted traffic and attempt to re-use them, or slightly different ones, on any site, and have a high chance of gaining access. Weaver also said that by capturing usernames, the Procera technology could be used to deanonymyze web surfing and more easily track what millions of Turk Telekom customers are doing.

‘A really bad idea’

Procera employees raised such concerns with CEO Cantor throughout the first half of 2016. “Capturing passwords feels like a red line in the sand that we should not cross,” co-founder and CTO Alexander Havang wrote on the company’s internal social network, Confluence, extracts of which were obtained by FORBES. “Lawful intercept is not our key competency. If this is a regulatory requirement and not a business requirement from the operator, we should try to help them advocate why this is a really bad idea.” (Havang declined to comment for this article after the executive team asked all staff to refer all press enquiries to the PR department).

Another employee on the same thread asked: “Why do we want to extract password? What is the use case? This feels pretty bad.” A Procera EMEA solutions engineer followed up, suggesting there was a fraud detection use case. In response, Andsten added: “There’s no fraud detection use case that I’m aware of that’d require the password, and the entire use case smells way more like a social graph thing than a fraud thing. Either they make a pretty bad job of requirements or there’s something else going on.” He later added: “Even if we discount the whole business of extracting passwords from the equation, what they are asking for is normally associated with a totally different market. I’m concerned about what the real ask is here and what brand risk exposure we’d be taking on.” The thread ended in late March, with Andsten saying the feature was “outside the scope of product features and requirement tickets.”

After learning the work with Turk Telekom was going ahead anyway, Andsten quit. His valediction opened a can of worms. “He essentially exposed the issue to the whole company,” a former colleague said. Two days later, on April 6th, another company-wide email sent from a disposable, anonymous email address went out, signed La Resistance. It called on all who opposed the Turk Telekom deal to protest directly to Francisco Partners. “We have absolutely no reason to do unethical deals. Procera is a great company that could do good in the world. We used to be all about improving network quality. That’s why we’re here… Your email absolutely matters. Make it anonymous if you want. If a few voices are heard, it will sound like we have a few vocal people, but if a lot of voices are heard, there must be actions taken.”

On April 11th, Cantor held an emergency meeting in Malmo to hear employees out and, in light of the brewing discontent, revamp the ethics committee; in internal communications, employees had previously expressed frustration at the lack of transparency from the group, set up in late 2014 after deals to provide deep packet inspection for operators in the Middle East had proven sticky subjects. “I don’t want blood on my code,” complained one engineer. “Is it even possible to do ethical business in the Middle East?” asked another.

Sources recalled a particularly awkward Malmo moment. “All the developers proposed to stand up and give applause to [Andsten] because he had taken a stand,” one source said. “With red ears the management team on stage had to participate in the applause of the person who’d caused all their problems.”

Procera didn’t force unwilling staff to do the work. Current and former employees said the username and password extraction was partly outsourced to Canadian firm Northforge, claiming this was done to avoid exacerbating the relationship between engineers and execs. Procera did not comment on that aspect of the work. Northforge did not respond to requests for comment.

Since Andsten’s departure, another five engineers have quit, according to sources. The work continues, said current and former employees, who also said Cantor sent out further company-wide emails advising staff not to speak with press.

Turkey’s surveillance regime

Though employees remain concerned about the Turkish government’s access to usernames and passwords of millions of its citizens, Procera wasn’t contracted by the Erdoğan regime or even Turk Telekom. The contract was with Sekom, a systems integrator who worked to install the technology at the operator. (Sekom had not returned requests for comment.) A Turk Telekom spokesperson wrote in an email: “As Turkey’s leading communication and entertainment technologies company Turk Telekom, we always work to deliver products and services with cutting edge technology to our customers. Towards this goal, we are upgrading and renewing outdated devices in our network infrastructure. During this renewal process, we have recently replaced some equipment in our network with updated versions through an auction. We will continue our offerings related to quota based services and parental control services to our customers with this new equipment.

“At Turk Telekom, we are highly sensitive regarding protection and confidentiality of our customers’ personal data and we fully comply with the rules and regulations we work within.” The company declined to comment on the username and password extraction capabilities.

Like any operator, Turk Telekom is at the mercy of government legislation. One of the more invasive regulations – Law No. 5651 – requires each telecom company to log and store user activity for up to two years and submit that data to the government when requested by a court. The Homeland Security Act, passed in 2015, allows the Turkish government to spy on suspects’ telecoms connections for 48 hours without the need to get permission from a judge.

The Turkish embassy in London, the foreign office and the Prime Minister’s office had not responded to repeated requests for comment on this article.

Deep packet inspection

And like any company working in deep packet inspection (DPI), Procera has had to tread a thin line between providing useful networking technology and dangerous surveillance gear. Procera would never describe itself as a surveillance company, but rather a QOE and QOS (quality of experience and quality of service) vendor for telecom operators.

But DPI, whilst most often benign, is inherently invasive. “Deep packet inspection enables surveillance at the outset,” noted Citizen Lab’s senior legal advisor Sarah Mckune. Its very purpose is to open up “packets” of data flying across networks and inspect them to check if they should pass. Just like someone searching a package at the post office to determine if there’s any illegal contraband inside, DPI scans every packet and logs what’s inside, before deciding where to route the data or just trash it. Certain kinds of traffic might be prioritized if they need to be shifted faster, making DPI a core technology in the net neutrality debate. Or DPI can uncover malware and stop it from spreading.

DPI has made headlines for controversial use cases. China, for instance, likes to use DPI in its infamous censorship and surveillance systems. Sunnyvale, California-based Blue Coat Systems, in which Francisco Partners was a significant investor, saw its DPI technology censoring the internet in Syria in 2011, just as the civil war was erupting. Human rights activists looked on agog, but Blue Coat later said resellers were to blame and that it had not given permission for the technology to be shipped to the country. One reseller was later slapped with a maximum fine of $2.8 million by the Bureau of Industry and Security (BIS). (Francisco Partners also has stakes in Barracuda Networks and Dell Software, which both ship DPI products).

Procera employees remained apprehensive about their paymaster’s other DPI shipments. In particular, a deal signed through systems integrator Giza Systems for Egypt’s National Telecom Regulatory Authority (NTRA), which had asked to use Procera’s ScoreCard service to evaluate the network performance and subscriber experience for different operators, according to a leaked “scope of work” document.

Legitimate work on the face of it, said one former staffer. But some employees remained perturbed about the potential for abuse of the product, so widely could the technology be deployed. The former staffer said: “It’s an unusual and quite expensive way to accomplish it by tapping all traffic in the country and indexing it… when it would be simpler to just require your operators to give you quality metrics.” They noted that the ScoreCard product keeps a searchable database of a subscriber ID, what website they visited, and the location ID associated with their IP address. “That database is searchable for the end customer if they know how.” But a person familiar with the company’s business in the Middle East said that Procera has not sold or deployed any license capability for database access to any operator or regulator in Egypt, and that the ScoreCard product does not conduct any type of surveillance.

Meanwhile, Turkey continues to expand its control over the web. Just this month, more than 150 police personnel were reportedly arrested for using an encryption tool, ByLock, which the government believed Gülenists used to plot the putsch. Cloud services Microsoft OneDrive, Google Drive and Dropbox, as well as code repository Github were also blocked. The blackout was enforced in response to a leak of 50,000 emails of Turkey’s energy minister and Erdogan’s son-in-law Berat Albeyrak by communist hacktivist crew RedHack. Did Turk Telekom and Procera help enforce that blackout? Neither had provided comment at the time of publication.

Tips and comments are welcome at TFox-Brewster@forbes.com or tbthomasbrewster@gmail.com for PGP mail. Get me on Twitter @iblametom and tfoxbrewster@jabber.hot-chilli.net for Jabber encrypted chat.