This morning, February 19, 2013 at about 3 AM an email arrived which set off alarms. Monitoring software on one of our servers had discovered a suspicious file: /lib64/libkeyutils.so.1.9.
Investigation revealed that this file is part of a server compromise. How the attacker is able to put this file in place is as yet unknown. What we do know at this point:
- RedHat Enterprise servers including CentOS and Scientific Linux are affected.
- Logins via SSH are recorded, including login name and passwords.
- Other logins, such as to email and cPanel are not affected
- Only 1 IP address has yet been recorded as the recipient of information
- More than 10,000 servers have been affected so far
- The goal of the attacker has so far been limited to sending spam email
To mitigate the threat we have set up scanning to find and remove suspicious files at 5 minute intervals and send alert emails when any suspicious file is found. This will trigger further investigation.
Since the source of the infection is unknown, the only prudent course is to assume the worst. We have set up firewall rules to prevent communication with the single IP address known to be receiving information. However, it would be naive to assume that this walls off the problem.
If you notice that your service is running more slowly than usual, the likely cause is actions we are taking to deal with this threat. It is possible that your service will be interrupted. Some counter measures are disruptive. For example, when server load becomes very high it can appear that a server is down because response is so slow. The fastest way to regain control in this case is a reboot.
If you call or email and do not get an immediate response, the reason is apt to be that we are working on a problem. We sometimes need to choose between solving a problem and explaining to 20 or 30 people that we are working on a problem. Frankly, it makes more sense to fix now and explain later.
If you observe problems with your service while this threat remains active, please be patient. We are all over it.
Thanks for update. Just letting you know that I am receiving alot of spam emails from people that I don’t know and when I put them into the Black List they still are able to send more. Now I know why.
With all due respect, you don’t know why.
We found a suspicious file and removed it. No damage was done. Nothing happened. No spam was sent. No email configuration files were modified. There is no relationship between the file we removed and any email activity you may be experiencing. The removed file was not on your hosting server.
Spammers use faked email headers and constantly use different ones. You could put tens of thousands of email addresses in black lists and continue to get the same emails from the same spammers.
Hi Dennis,
With all due respect I was agreeing with point 6 which states: The goal of the attacker has so far been limited to sending spam email.
So he either can or can’t but I have been with you for many yrs and in the last week or so I have had an unusual amount of junk mail come through so I thought it was inline with what your Blog had said.
OK. Perhaps I should be more clear. Although we did see the suspicious file appear, we did not see any suspicious activity of any description.
There have been reports of servers compromised in this way sending spam. So far we have had no evidence that any of our servers was used in this way. It’s quite likely that we prevented anything illicit by acting quickly.
As a general point of information, it is true that a large percentage of the spam we see is sent from compromised machines. It’s also generally true that if you make it even a little bit difficult for the attacker to do anything he will move on to easier quarters. There’s no shortage of poorly managed servers, admins asleep at the switch.
Dennis,
Thanks SO much for the high alert and the comprehensive information about what has been going on. I really do appreciate your hard work to keep the servers secure and our websites up and running, as I know you recognize that these are our business outlets and vital to us.
I’m rooting for you and your team and trust that you have things well in hand.
Kind regards,
LariAnn Garner