Deerfield Hosting, Inc.

High Performance Web Hosting

dennis

Email Should Never Bounce

by

Don’t be lulled into the idea that great big rich companies automatically have high quality correctly configured Internet service. It is very common for outbound email service to be incorrectly configured.

A big part of the problem is that Microsoft email servers are configured incorrectly out of the box. They work correctly in the common cases and testing frequently stops there. At that point they will not use a backup or secondary email server AND they fail to keep things on a queue for a few hours to work around temporary failures as the standards dictate.

It is virtually impossible for your inbound email service to be completely down. There are 2 servers in different data centers capable of receiving mail. Yet, I commonly get reports of bounced email when our main mail server is off line. There is absolutely no excuse for that because the secondary is available.

Next time anyone, and especially a Great Big Company, tells you they couldn’t get an email to you, demand that they send you a copy of the mail with the failed headers. I can almost guarantee they will reveal a configuration error on their server. It’s only by pointing out their problems that they will become aware of and correct them. They should want to know they have a problem.

Your Domain and Google Search

by

We are often asked about the results when a domain name is typed directly into a Google search box. It’s helpful to understand a little bit about how Google searches work. It’s a gigantic topic and we’re only dealing with one small corner of it here.

What Google tries to do first is find things other people have searched for. When you click on a particular result, they record the click. The idea is that since you clicked on it, the description probably was a match for what you wanted. Next time a similar search is done, your click tends to move that result up to a higher position.

Next, people often make typing mistakes and Google attempts to correct them. In theory, this saves the customer from wasting time and needing to re-type and saves Google from wasting resources on bad searches.

It gets a lot more complicated than that, but that’s the beginning of how it works.

When you type a domain name into a search box, Google is likely to recognize it as a domain name. But the same logic will be applied, with results you may or may not like. I was just asked about the domain name miniatureangels.com. Google returned, “Showing results for miniature-angel.com” – NOT what was wanted. Apparently that’s a popular site.

Unfortunately there is nothing to be done about this. We sometimes hear from customers who are upset that something like this is happening to them. There isn’t anything magical or mystical going on and it wouldn’t matter where or how your site is being hosted. Website content may have some effect. It has nothing to do with hosting at all.

I tend to regard typing a domain name into a search box as a dumb thing to do. After all, if you are looking for the web site for a domain there is no need for a search. Just go there. The trouble is, huge numbers of people where introduced to the Internet by simply sitting down at a computer with a Google search box in front of them. They typed in what they wanted and found it and that is the end of that. The principles of least thought and least resistance have coincided and that is what they will do evermore.

Our customer who owns miniatureangels.com wants to replace that with miniatureangelsfarm.com. Probably a good idea.

“Secure” Email

by

The content of your emails is not as secure as you may think it is.

When you set up an email account on your computer (or tablet or phone) you have the opportunity to specify encrypted connections. The trouble is, email is a store and forward system. Unless you know every server your mail will traverse, you can’t know that it will be transmitted encrypted from server to server.

You are also subject to trusting the mail server administrator of each server it traverses, sometimes as many as 4 or 5. While the transmission from your machine to the first server may be encrypted, subsequent transmissions may not be.  What is more, at each server it will be stored in plain text format.  It is trivial for any mail server administrator to retain a copy of your emails.  It is also trivial to scan plain text files looking for key words such as the word, “password”.

If you want what is in your email to be secure, what you need to do is encrypt the content, not just the connection.

One commonly used tool for this purpose is PGP. That stands for “pretty good protection”.  Encryption is a huge subject, but there tools in the cPanel control panel to help you get this done.  More information about PGP and this subject in general can be found HERE.

Having said all that, there is still 1 good reason to set up secure (encrypted) connections from your email client to your hosting server: passwords.  Secure connections are established before your password is sent and that means it is sent encrypted and not as plain text.

High Alert

by

This morning, February 19, 2013 at about 3 AM an email arrived which set off alarms. Monitoring software on one of our servers had discovered a suspicious file: /lib64/libkeyutils.so.1.9.

Investigation revealed that this file is part of a server compromise. How the attacker is able to put this file in place is as yet unknown. What we do know at this point:

  • RedHat Enterprise servers including CentOS and Scientific Linux are affected.
  • Logins via SSH are recorded, including login name and passwords.
  • Other logins, such as to email and cPanel are not affected
  • Only 1 IP address has yet been recorded as the recipient of information
  • More than 10,000 servers have been affected so far
  • The goal of the attacker has so far been limited to sending spam email

To mitigate the threat we have set up scanning to find and remove suspicious files at 5 minute intervals and send alert emails when any suspicious file is found. This will trigger further investigation.

Since the source of the infection is unknown, the only prudent course is to assume the worst. We have set up firewall rules to prevent communication with the single IP address known to be receiving information. However, it would be naive to assume that this walls off the problem.

If you notice that your service is running more slowly than usual, the likely cause is actions we are taking to deal with this threat. It is possible that your service will be interrupted. Some counter measures are disruptive. For example, when server load becomes very high it can appear that a server is down because response is so slow. The fastest way to regain control in this case is a reboot.

If you call or email and do not get an immediate response, the reason is apt to be that we are working on a problem. We sometimes need to choose between solving a problem and explaining to 20 or 30 people that we are working on a problem. Frankly, it makes more sense to fix now and explain later.

If you observe problems with your service while this threat remains active, please be patient. We are all over it.

Automated Email Checking

by

Setting your email client (MS-Outlook, Windows mail, Thunderbird, etc.) to check email too often is abuse. Sometimes it’s accidental abuse. It’s common for unstable programs like Microsoft Outlook to become corrupted and go wild, checking incessantly.

Recently we implemented a system to track (among many other things) email logins. When too many logins to a specific email account exceed limits, the IP address the logins are coming from is temporarily blocked. The reason for doing this is performance. Our servers typically have 5,000 or more email accounts on them. 20 or 30 abusive accounts is not a large percentage of the total, but can significantly reduce performance for everyone using a server.

If your ability to send and receive email is experiencing intermittent problems, this may be the reason. We can check this for you if you submit a trouble ticket. Our ticketing system automatically includes your public IP address which we need to do this. Click on “Orders – Tickets” above left, then “Submit Ticket”.

All this begs the question, why you are doing this in the first place. All you really need to do is click on the send/receive button in your email client and you will have your email. If you have a broadband connection, which is most often the case, you will almost certainly have your email within 2 seconds. Most people who have automatic checking turned on will do this anyway! If you have a slow connection, by all means do automated checking to save yourself some time.

Otherwise, wouldn’t you rather see faster performance of your server? If you actually think you have another reason for automated checking, let’s hear it!

Hard Drive Failure !

by

100% uptime is impossible. All we can do is get close.

Last week 2 hard disks failed (simultaneously !) in node 2 of cluster 2. Besides being backed up in real time on a second cluster node, each node is running RAID-5 with hot swap drives. A single drive can fail and be replaced with no down time. But if a second drive fails, it’s fatal.

It wasn’t a clean failure and the performance of the fail-over system and failure reporting was less than perfect. Initial symptoms seemed to point at network card failure. The cluster software did fail over properly, but we had to clean up some databases. Some sites were not in good shape for 2 or 3 hours.

The next day, the remaining node was bombarding us with emails about the failed node. I had to shut everything down and power it back up outside the cluster. Total down time with this was probably 10 minutes. This was necessary because otherwise we could easily miss emails about failures of which we are not yet aware. The signal to noise ratio was way too high.

Monday we replaced all the hard drives in the failed node, re-installed the operating system and all the cluster software and began the process of manually syncing the drives from the node which was still in operation. Synchronization completed overnight last night, Tuesday night.

This morning at 5 AM I began the task of moving services back into the cluster. I will spare you the details, but it’s a nasty and error prone process. All the safeguards, checks and balances in the cluster software really get in the way while doing this. Sites were up and down several times. My guess at total down time today was something like 30 minutes.

Everything is completely back to normal now.

This was the first major real world test of the clustered live fail-over system we put in place 18 months ago. I’m not totally happy with it. Previous tests were done by pulling plugs – total failures. In that situation, performance was flawless. Down time was so short no one noticed. Real world failures are usually messy like this one was. The fail-over system worked, but it needed a little help. It was still a big win compared to re-installing a server and restoring backups. That could take a day or more.

There is a recurring pattern with problems like these. There is a period of a few days or a week during which problems come up and quickly or gradually get ironed out. These periods in retrospect feel like they are much longer than they really are because the worry and frustration when a server is down is intense. An hour is remembered as half a day. Related problems recurring a few times over several days is remembered as lasting a week or more. It’s human nature. Problem periods are followed by long periods, many months or a year during which everything runs smoothly.

If you look at our up time in longer periods it’s actually very good. It’s something over 99.99%. My perfectionist nature often makes me lose site of that. But nobody does any better so it’s worth a reminder.