Follow GFI:
Find us on Facebook Follow us on Twitter Find us on Linkedin Subscribe to our RSS Feed Find us on YouTube Find us on Google+
 

57 Simple Sys Admin Mistakes that Someday You’ll Regret

on April 17, 2012

When it comes to IT, simple mistakes can cost you big time. Whether that is immediately or not depends on a number of factors, and sometimes, when something has gone incredibly wrong, you won’t even realize that the root cause was a snap decision or simple oversight you made months ago. If you parse this list, you might just avoid a costly mistake sometime in the future; or, you may see something that makes you realize you weren’t the only one who had that happen.

 

Networking

There are tons of simple mistakes, oversights, omissions, and bad ideas that can bite you in the networking realm. Here are some of our favourites. Whether they caused outages, security issues, poor performance, or just made troubleshooting a pain, we’ve been there, done that, bought the t-shirt.

DNS

1. Forgetting to close a zone name with a trailing dot. The lack of one simple period stopped mail for 30K users once at a major company.

2. Leaving your zone open to perform a full zone transfer to any requestor just makes recon easy for attackers. Never permit zone transfers to any server except the ones you want hosting a copy.

3. Setting week long TTLs may reduce traffic, but when you make a typo and don’t realize it for a couple of hours, explaining that it will take a week before that site is back up globally, is an uncomfortable conversation to have.

4. Keeping all your DNS servers on the same network is a quick path to losing everything when a routing table entry goes bad.

5. Using routing/NAT instead of split DNS is not the way to let your users access DMZ resources using external names; that is, not if you expect everything to work.

6. Circular recursion is when two different servers are configured to recurse to one another for unknown requests. You can take out an entire Ethernet segment if the DNS servers are robust enough as the request ping-pongs between them.

7. It may seem like a pain to create all the in-addr.arpa records so you can use PTR records internally, but when you are trying to find the source of some rogue traffic, you’ll be glad you did.

Other networking issues

8. Yes, I know Microsoft told you 10 years ago that NetBIOS was going away. Guess what. It’s still here. Not configuring WINS is a bad idea and can cause poor network performance.

9. Clear-text protocols should never be used, no matter how simple they are to set up or how limited you think their use will be. Ignore this rule and soon you will see domain creds crossing the wire.

10. Using non-standard ports is just a bad idea all around. You may think it lets you do more on your server, or provides some security, but all it really does is keep your customers from accessing the services. Way too many potential customers could use your web app if it was on 80 or 443, but their network blocks outbound access to anything else.

11. Legacy rules on the firewall must be cleaned up immediately. Open ports to unused ip.addrs are a disaster waiting to happen when a new box uses an old address.

12. NAT internal traffic out to the Internet, and Internet traffic to your DMZ for applications that support it, but never NAT internally. You will break stuff left and right, and some of that cannot be fixed. There once was a major corporation who ignored that, and wound up having to migrate 10,000 user accounts, twice, because they couldn’t get a trust working through a NAT’d connection.

13. Using long xlate timeouts means that bad connections stick around for hours. Think of xlates in terms of seconds to minutes, but never in hours.

14. Bypassing the firewall is never the right answer. If you are going to allow systems to span the external and internal network, and they themselves are not firewalls, you may as well just get rid of the firewall entirely. Think of the money you’ll save on annual support. You’ll need that to pay for incident response!

15. Never use a domain name that you do not own. Register anything you want to use, even if you think it will never be used externally.

16. Never use bogon addresses. They won’t be bogons forever. RFC 1918 addresses and registered addresses you control are the only ones you should use.

17. Using a HOSTS file to get around something, and then never going back to fix the issue. It’s never the HOSTS file, until you spend a few hours troubleshooting DNS before you finally remember that it actually is the HOSTS file.

User accounts and passwords

18. Go Bing Search default password lists, and then go change all the default passwords you never thought were a risk.

19. Never set a user’s password to never expire. We expire passwords so that when they are compromised, we limit the time the attacker can use the compromised credentials. Never is a mighty long time.

20. Don’t forget to set a service account’s password to not expire, and then schedule a regular reset. If a service account’s password expires, the next time that service needs to start, it won’t.

21. Setting too complex a password requirement just leads users to write things down.

22. Requiring too many things in password (must start with a capital, must end in number, must be no more than 11 characters long, etc.) leads to passwords that will be written down.

23. Using the same password for multiple accounts means that an attacker has the master key as soon as he compromises one service. Use different passwords for your work, personal email, Facebook account, etc.

24. Never use a User account as a service account. You change your password, you break the service.

25. Leaving sAMAccountName blank may be fun and make scripting easier, but there are some systems that cannot use UPN.

26. Using non-routable UPNs works great internally, but the more services that move to the cloud, the more important a routable UPN will be.

27. Never set the local admin account password to be the same on all workstations. Someday, it will get out, and you’ll have to change it on every single machine in the company. Don’t use a pattern that can be easily figured out either.

28. Stale accounts should be disabled and then deleted.

29. Weak passwords are worse than strong ones written down. An attacker has to find the piece of paper you wrote a password on; they can break a weak password from anywhere.

30. If you have a web service you offer to external users, like your customers, let them use their email address as their username. It’s unique, you don’t have to maintain it, and they will be much happier for it.

 

Speaking of Email…

31. As much as they are spam magnets, you still need to establish the RFC 2142 addresses and monitor them. Comply with the RFCs.

32. The last thing you should do to a mail server before opening port 25 on the firewall is to check it for open relay.

33. The first thing you should do to a mail server after opening port 25 on the firewall is to check it for open relay.

34. Emailing a bunch of people, some of whom don’t work for you? Use BCC. CC’ing hundreds of strangers happens by accident all the time. That they immediately drop your service and block your emails after this happens is not by accident.

35. Spellchecking is not optional! Enable it on your client, and check the spelling of your email, and your subject line!

36. Reply to all is almost never a good idea. Consider how much noise you deal with every day in your inbox, and do unto others as you wish they would do unto you.

 

Active Directory

37. Never set up a DC to suffer DNS islanding. A DC should never use itself for DNS; always point it at another DC.

38. Keep your admin accounts separate and use a second, normal account for your workstation use. Too many things like inheritance filtering can cause administrative accounts all kinds of problems with other services.

39. Monkeying with inheritance is a bad idea. I’ve never seen a case where it was a good idea, but I’ve seen literally hundreds where someone turned off inheritance years ago, and it caused major issues for the poor team that ‘inherited’ the mess.

40. Repeat after me. You create a new subnet, you define it in AD and assign it to a site. It takes two extra minutes. Not keeping up with your sites will cause logon and other performance issues, can impact replication, and will be a multi-month project to clean up when it finally becomes the blocker to a major project.

Antivirus

41. Disabling antivirus should be a capital offense.

42. Not configuring proper exemptions based on vendor recommendations is a sure way to crush application performance, and will cause others to disable antivirus protection.

43. Not investigating failed updates immediately is asking for trouble, since usually the reason an update failed is either because the antivirus was disabled, or your machine is already infected.

Teach your Users well

44. Asking for their password is never acceptable. The first time you do, you can never again expect them to safeguard their credentials from potential attackers.

45. Using self-signed certs just trains your users to ignore certificate warnings and makes them click-through-happy. The last thing you want users to do is ignore warnings.

Other

46. Certificates expire. Not monitoring certificates will lead to outages, and mad scrambles to replace them. Set up a reminder for a couple of weeks ahead of expiration so you have plenty of time to renew.

47. Pings are not the enemy. Not allowing ICMP echo just makes troubleshooting and simple monitoring more difficult, and it violates RFC 792 and 4443. Don’t violate the RFCs.

48. Not using NTP is just lazy. Pick a good time source and sync everything to it. It makes your logging correlation much easier, makes users happy, and prevents authentication from breaking.

49. Not verifying backups is playing Russian Roulette. You must perform regular test restores to ensure you have a good backup.

50. Deploying an untested patch is only  fractionally worse than…

51. Skipping a patch.

52. Encrypt everything that can leave the four walls of your network. That means laptop hard drives, USB keys, and network traffic. Unless it’s data you would post on your website, it should never go out the door (physical or virtual), unless it’s encrypted.

53. Using unsupported configurations may be cheaper or faster, but unless you really are willing to forgo the option of vendor support, it’s just not worth it. You’ll spend more money trying to troubleshoot an issue, or waste more time getting things back into a supported configuration than you ever saved by trying to cut corners.

54. I have personally seen at least three instances where letting support/maintenance lag or expire wound up costing the companies in question millions of dollars. Never let your software support expire until you have formatted every machine that was running that software.

55. Staying on legacy software versions may save you upgrade fees, but eventually you will find yourself in an unsupported configuration, or at a point where you want to do X, but you cannot because it won’t support legacy Y. You’ve just found that proverbial spot between the rock and the hard place.

56. Hardcoding anything – just don’t do it.

57. Documentation is a pain. We all hate it, we never do it, and yet we curse the guy who came before us that didn’t keep good documentation. Pay it forward.

 

So there you have 57 far too common mistakes, omissions, shortcuts, bad ideas, and lapses in judgment that might seem like a good idea today, but will cost someone pain and suffering in the future. Don’t be that guy. Do things right, the first time, and you’ll be much better for it, and so will your network.

Have any other tips or experiences you’d like to share? Leave us a comment below!

Like our posts? Subscribe to our RSS feed or email feed (on the right hand side) now, and be the first to get them!

Discover how cloud-based network monitoring can help you to maximize your IT resources, save money, be proactive and how it can give you a fast return on a small investment – Download your FREE ebook today

 

About the Author:

Christina is Web Marketing Content Specialist at GFI Software. She is a keen blogger and has contributed content to several IT sites, besides working as an editor and regular contributor to Talk Tech to Me. Christina also writes for various publications including the Times of Malta and its technology supplement.

 
Comments
dustin April 18, 20129:35 am

netbios..still using the crap out of that on a dos network with netbeui

:( oh legacy software, y u no has modern alternatives

Binaryspiral April 18, 20124:02 pm

#18, is the mistake using Bing or relying on its results to locate default password lists?

 
Christina Goggi April 18, 20125:23 pm

Hah! Good one!

Erik April 18, 20126:19 pm

The only reason to use Bing, is to search for Google. Much the same: the only reason to use IE is to download Firefox / Chrome.

Roberto April 19, 20125:26 pm

The only reason to use Google, is to be phished and data mined.

lewax00 April 24, 20121:09 am

And you think Bing doesn’t? I have news for you: if a service is free, you aren’t the customer, you’re the product.

 
Bill Gates April 18, 20126:27 pm

Writing all your passwords in a little book and then accidentally throwing it away with the junk mail on your desk.

Dude April 18, 20128:45 pm

nice list!

50. Deploying an untested patch is only fractionally worse than…

51. Skipping a patch.

I have heard just deploy all patches as soon as possible, because exploits are developed in record time now a days. Finding the time to test every patch for every software for every configuration is impossible. If you have good nightly backups you can roll back updates in most cases and issues with patches are few and far between, at least for major vendors.

Or am I way off?

 
Christina Goggi April 19, 20123:01 pm

Hi Dude, there are two schools of thought on this controversial topic, but my own experiences have taught me that it is less risky to apply an untested patch than to not patch at all. The vast majority of incidents I have worked came down to someone exploiting an unpatched vulnerability.

Often, when a patch does cause a problem, it causes that problem with third party software, or non-standard configs, so if your servers are fairly vanilla, the risk is pretty low. Even with testing, if there is a problem with a patch, it may not show up until someone performs a task you didn’t test for, like a month end closing, or a full backup, or an alignment of the flux capacitor, or a future update of your antivirus software, so at the end of the day, testing may reduce the risk of a patch introducing a problem but it cannot eliminate it.

 
Richard Hobbs April 19, 201211:13 am

Point 41 is very true. However I would be poorer as I spend a lot of time fixing virus attacks when a user has disabled the anti-virus.

Dave April 23, 20128:09 pm

These tips are great. However; if you work for more than a small organization where you are the admin, engineer, help desk, and have employees who never receive any training you will break a lot of the rules. Unless of course you like working 16 hour days.

Rob April 25, 20127:15 pm

Nice list, thanks for compiling that! Here are a few other simple, but very, very useful ones:
*Never* use the “root” account when logging into a system remotely, not even when using SSH. SSh is secure and all, but the PermitRootLogin option should always be set to “No”. This way, if you do have someone sniffing on your network and exploiting SSH in some way, they’re still not gonna be able to sniff your root password. Also, just logging in as root “because it’ll allow you to do anything with no trouble” just isn’t an excuse and should be considered very, very bad practice. Also, on my own server, I’ve seen lots and lots of script kiddies establishing a connection to my sshd and trying to brute force my root password. Well, that’s never gonna lead to success once remote root logins are disabled, even if the attacker would guess the correct password for root.
Another no-brainer that’s caused lots of trouble in the past is not checking file permissions. If, say, you edit the config file of a daemon and then put it in its appropriate directory, you’ll most likely want that file to belong to that daemon’s service user. Otherwise, you’ll end up with annoying “permission denied” errors and services not being able to launch directory. This can also be a huge pain when you set up a mail service, but forget to assign the right permissions / ownerships to the directory where mail should be saved… :-)
My last suggestion is especially related to apt-based systems, such as Debian and Ubuntu. The apt-packaging system is sweet and easy to use, and can save a lot of compiling and installation time. Another thing that makes apt so “sweet” is the fact that it makes it very easy to deploy updates and apply security patches. Thus, my simple suggestion is to take advantage of that! Install a tool such as Apticron, which can check your apt repositories for updates on a daily basis, and will conveniently send out an EMail whenever an update is available. Take the time to read through the (mostly brief) changelogs, then log in and install the updates. This will take you about 2 minutes. I do not, however, recommend using any software that will update your system automatically. It’s never good to lose control of what is going on on your system.
Not sure if everyone will agree with me on these points, just my 2 cents really! :-)
Happy hacking everyone.