CircleCI is feeling the heat of scrutiny after it asked customers to rotate secrets following a security incident disclosure with scant details.
The San Francisco-based company, one of the largest continuous integration and continuous delivery platforms in the industry, has yet to explain any specifics of the incident disclosed Wednesday, however it urged much of the software community to undertake costly and time-consuming mitigation steps to protect their most valuable assets.
“Having to rotate those secrets is a fire drill for organizations to identify what those secrets are and where they live, in the wake of an attack,” Matthew Rose, field CISO at ReversingLabs, said via email.
CircleCI is very well known and widely used across the industry, with about 200,000 DevOps teams using the platform across industries, Rose said, citing the company’s data.
“We take customer security and privacy extremely seriously,” a spokesman told Cybersecurity Dive via email on Thursday. “We are committed to sharing a full incident response when we can do so, while preserving the integrity of our investigation.”
CircleCI CTO Rob Zuber, who has regularly updated a blog post on the incident, on Saturday announced the company completed the process of rotating GitHub OAuth tokens on behalf of customers.
The company previously announced that it had removed personal and project API Tokens created before Jan. 5 and that its partners at Atlassian expired all OAuth Tokens for Bitbucket users.
Last week, Zuber said the company was confident it had eliminated the risk that led to the incident and assured customers the platform was safe to build.
Customers were advised to review their logs from Dec. 21 to Jan. 4, when CircleCI originally disclosed the incident. However Zuber denied any connection to his Dec. 21 post about prior reliability issues at the company, saying that was pure coincidence.
CircleCI warned in November about attempts to launch phishing attacks against organizations by attackers pretending to be from the company.
The company warned it would not have much in the way of additional substantive details about the cause of the most recent incident until it completed a forensic investigation with a third-party firm.
Security researcher Daniel Huckmann posted on Twitter that he had been investigating a CircleCI incident over the holiday break involving a Thinkst Canary AWS token. The CircleCI spokesperson said the company was aware of the claim, but did not comment further.
Keeping secrets
Rotating secrets in the software build environment generally refers to any credential that needs to be protected, including passwords, API keys, auth tokens and public and private keys, according to Tom McNamara, CEO of Hopr.
McNamara said the reaction from CircleCI customers speaks to the costly impact this incident – and how the company is handling it – is having on the developer community.
“This is very bad for software engineering teams,” McNamara said via email. “It is costly and time consuming and the support comments revealed that there is a lot of frustration among developers.”
McNamara said most of the remediation steps appear to be very manually intensive for developers and security engineers.
“Just trying to get an audit of what secrets exist and what has been stolen appears very difficult,” McNamara said. “Some security staff have even suggested the ‘break glass’ option of completely disconnecting the service, but not everyone can accept this option.”
The CircleCI incident is just further evidence that developers and development infrastructure continue to be the front line of new cyberattacks, according to Brian Fox, co-founder and CTO of Sonatype.
“Developers need to assume that any CI/CD system that runs build against code contributed from untrusted parties, such as a pull request from a contributor, could be compromised in some way,” Fox said via email.
When the CI/CD system is executing code, whether it be a unit test or a new plugin, the code could do something nefarious, including snatching secrets the CI system has access to, Fox said.