FreeBSD.org intrusion announced November 17th 2012
From: FreeBSD Security Officer <security-officer@FreeBSD.org>
On Sunday 11th of November, an intrusion was detected on two machines within the FreeBSD.org cluster. The affected machines were taken offline for analysis. Additionally, a large portion of the remaining infrastructure machines were also taken offline as a precaution.
We have found no evidence of any modifications that would put any end user at risk. However, we do urge all users to read the report available at http://www.freebsd.org/news/2012-compromise.html and decide on any required actions themselves. We will continue to update that page as further information becomes known. We do not currently believe users have been affected given current forensic analysis, but we will provide updated information if this changes.
As a result of this event, a number of operational security changes are being made at the FreeBSD Project, in order to further improve our resilience to potential attacks. We plan, therefore, to more rapidly deprecate a number of legacy services, such as cvsup distribution of FreeBSD source, in favour of our more robust Subversion, freebsd-update, and portsnap models.
More information is available at http://wwww.freebsd.org/news/2012-compromise.html
Saturday November 17th, 2012
On Sunday 11th November 2012, two machines within the FreeBSD.org infrastructure were found to have been compromised. These machines were head nodes for the legacy third-party package building infrastructure. It is believed that the compromise may have occurred as early as the 19th September 2012.
The compromise is believed to have occurred due to the leak of an SSH key from a developer who legitimately had access to the machines in question, and was not due to any vulnerability or code exploit within FreeBSD.
To understand the impact of this compromise, you must first understand that the FreeBSD operating system is divided into two parts: the "base" maintained by the FreeBSD community, and a large collection of third-party "packages" distributed by the Project. The kernel, system libraries, compiler, core command-line tools (e.g., SSH client), and daemons (e.g., sshd(8)) are all in the "base". Most information in this advisory refers only to third-party packages distributed by the Project.
No part of the base FreeBSD system has been put at risk. At no point has the intruder modified any part of the FreeBSD base system software in any way. However, the attacker had access sufficient to potentially allow the compromise of third-party packages. No evidence of this has been found during in-depth analysis, however the FreeBSD Project is taking an extremely conservative view on this and is working on the assumption that third-party packages generated and distributed within a specific window could theoretically have been modified.
If you are running a system that has had no third-party packages installed or updated on it between the 19th September and 11th November 2012, you have no reason to worry.
The Source, Ports and Documentation Subversion repositories have been audited, and we are confident that no changes have been made to them. Any users relying on them for updates have no reason to worry.
We have verified the state of FreeBSD packages and releases currently available on ftp.FreeBSD.org. All package sets for existing versions of FreeBSD and all available releases have been validated and we can confirm that the currently available packages and releases have not been modified in any way.
A package set for the upcoming FreeBSD 9.1-RELEASE had been uploaded to the FTP distribution sites in preparation for 9.1-RELEASE. We are unable to verify the integrity of this package set, and therefore it has been removed and will be rebuilt. Please note that as these packages were for a future release, the standard
pkg_add -r tools to install packages could not have downloaded these packages unless they were requested explicitly.
We unfortunately cannot guarantee the integrity of any packages available for installation between 19th September 2012 and 11th November 2012, or of any ports compiled from trees obtained via any means other than through svn.freebsd.org or one of its mirrors. Although we have no evidence to suggest any tampering took place and believe such interference is unlikely, we have to recommend you consider reinstalling any machine from scratch, using trusted sources.
We can confirm that the freebsd-update(8) binary upgrade mechanism is unaffected, as it uses an entirely separate infrastructure. We have also verified that the most recently-available portsnap(8) snapshot matches the ports Subversion repository, and so can be fully trusted. Please note that as a precaution, newer portsnap(8) snapshots are currently not being generated.
As soon as the incident came to light, the FreeBSD Cluster Administration team took the following actions:
- Power down the compromised machines.
- Power down all machines on which the attacker may have had access.
- Audit the SVN and Perforce repositories to:
- Verify that there had been no server intrusion.
- Verify that no malicious commits had been made to the repository.
- Verify that the SVN repository exactly matched a known-clean off-site copy.
- Verify that all FreeBSD base release media and install files on the master FTP distribution sites are clean.
- Verify all package sets available have checksums that match known-good copies stored off-site.
- The package set built for the upcoming 9.1-RELEASE did not have an offsite backup to verify against. These have been deleted, and will be rebuilt before 9.1 is released.
- All suspect machines are being either reinstalled, retired, or thoroughly audited before being brought back online.
- If you use the already-deprecated cvsup/csup distribution mechanisms, you should stop now.
- If you were using cvsup/csup for ports, you should switch to portsnap(8) right away. ports developers should be using Subversion already. Further information on preferred mechanisms for obtaining and updating the ports tree can be found at http://www.freebsd.org/doc/handbook/ports-using.html
- If you were using cvs/anoncvs/cvsup/csup for src, you should consider either freebsd-update(8) for signed binary distribution or Subversion for source. Please see the chapter on updating FreeBSD from source in the handbook. Further details on using Subversion and a list of official mirrors can be found at http://www.freebsd.org/doc/handbook/svn.html
- If you use portsnap(8), you should portsnap fetch && portsnap extract to the most recent snapshot. The most recent portsnap(8) snapshot has been verified to exactly match the audited Subversion repository. Please note that as a precaution, portsnap(8) updates have been suspended temporarily.
- Follow best practice security policies to determine how your organization may be affected.
- Conduct an audit of your system that uses FreeBSD.org provided binary packages. Anything that may have been installed during the affected period should be considered suspect. Although we have no evidence of any tampering of any packages, you may wish to consider rebuilding any affected machine from scratch, or if that is not possible, rebuild your ports/packages.
If you have any further questions about this announcement, please contact the FreeBSD-security@FreeBSD.org mailing list, or for questions where public mailing list distribution is inappropriate, please contact the FreeBSD Security Team.