There is an arms race going on between the dark and white forces; a Sisyphus work of building defenses which are in turn being defeated in a seemingly endless cycle. How can we ever break out of this cycle to finally feel and be safe?

Trust, together with encryption, is the keys to this goal. While most of the internet traffic is unencrypted and untrusted in origin, it is vulnerable to attacks. Obviously encryption by itself is not the silver bullet; it has to be done right, together with trust management and without exceptions.

This can’t be done overnight. Wherever possible, encryption should be used with proper key management. This would close many holes in the system, no longer exposing end user data to the attackers. The end user needs to be educated and forced to use the more secure – encrypted storage and protocols whether it’s HTTPS, SFTP, DNSsec or IPsec. Also email encryption and digital signing has been available for decades, but is rarely used by the general public.

It’s up to us, the IT pros, to set the standards, to configure secure defaults on our systems and in our products. We have to insist on using the most secure options, no compromises.

Many of us use VPNs which are de-facto encrypted by default, but many other services are not! We need to fix this. The best start would be:

• use encrypted storage, internal and external
• use IPsec on your intranet
• force HTTPS/SFTP on your website/webmail
• force SMTPS/IMAPS/POPS on your email server
• introduce email signing/encrypting
• enforce proper key management

More advanced securing can be achieved by employing DNSsec and NTP over SSL. Also a good idea is to pass proprietary/custom/3rd party protocols via SSL/TLS/IPsec tunnels.

When the majority of IT pros start following these basic rules, the situation will improve. It’s going to take time, but I am optimistic that we will get there.

Open source is great, you can learn a lot from looking at the source code of an application, you can even fix a bug here and there, or code in a feature you always wanted. And all for free…

What bothered me with this setup was the excessive amount of custom compiled subsystems to make them all perform in the desired way. To get the system working is a nice achievement, but to keep it running in production would be a nightmare. This is a bad security practice on a binary package based distro, let me explain why.

The applications compiled from source do not integrate with the package manager, and if they do (rpmbuild), it’s just a dirty trick, to compile and build a package to install it. Usually the package is just included in the inventory; versioning is broken, dependencies broken, updates broken…

The administrator would have to track changes to the custom compiled subsystems, pick out the worthwhile updates, and watch for security fixes, patch, compile, reconfigure and test the system while keeping good uptime. That’s not good and you don’t want to do that, unless you are some kind of masochist!

Instead let’s use the resources of the respective distros packaging team. That’s what we have package management for. Use it! Each of the top distros has a dedicated team to keep the packages up-to-date.

If your distro does not natively provide the package you desire, look for optional or 3rd party repositories. Usually your requirements are not that unique, and the application is already prepackaged in one of the optional repositories. There is a good chance that the repositories are maintained well enough, and you’ll have updates available when needed.

Next time when you decide to install something, think – is it also maintainable?

I decided to install Funtoo, a Gentoo based distro or rather fork, by the Gentoo fouder Daniel Robbins. Daniel is no longer active in Gentoo, and Funtoo is his new pet project, introducing some radical new ideas into the Gentoo landscape. Not to stop there, I went for 64bit, core2 optimized compiler flags, Ext4, latest kernel (2.6.30-gentoo-r5), grub2 and git based portage. Oh boy, am I asking for trouble or what!?!

To start off, I needed a 64bit boot cd, just any Linux distro would do, but it must be 64 bit – the bootstrap enviroment must be compatible with the installed enviroment, otherwise the chroot from one to the other won’t work. The choice fell to a Gentoo-MinimalInstallCD-64bit, fresh weekly build, cca 120MB in size.

The install follows mostly the usual Gentoo install routine – fdisk, mkfs, mount, mount proc, mount dev, chroot, env-update, source profile. To jumpstart the network a ‘dhcpcd eth0‘ will do – for now.

The MinimalInstallCD supports ext4 out of the box, so formatting was a non-issue. I downloaded the latest preconfigured and optimized stage3 & portage tarballs for core2 architecture from funtoo.org and extracted them into the new ext4 partition. Some customization is needed to fstab to match the partition layout, and make.conf to make use of more compilation threads on my multicore CPU. The compiler flags and make.profile were already preset by my choice of the coresponding stage3.

Here comes the tricky part, grub2 & kernel. I won’t drag you through the whole kernel building, just make sure you switch on ext4 support in the new kernel. My wrong was the assumption that ext4 was enabled by default; well, we all learn from mistakes… Btw, I used genkernel to build the kernel.

Grub2 is a different cup of coffee.

Grub2 is unstable, and won’t be stable any time soon. To further complicate things, the only usable version (ext4) is from SVN, and my firewall does not allow svn: connections. I tested the 1.96 ebuild, it failed miserably, time for hacking the 9999 aka SVN ebuild.

Grub2, needs a patch to recognize gentoo style kernels, which is included with the 9999 ebuild unlike 1.96, but the path to access SVN via http: is different then via native svn: protocol. A bit of searching, a quick fix and rebuilding the digest, and the SVN ebuild was working now via http. There is also a weird dependency on ruby, which I find to be somewhat overkill.

Now the ebuild worked, grub2-install worked, but still no luck booting. A few sources recommend to be extremely conservative with the build flags, so I chose to set ‘sys-boot/grub static custom-cflags multislot‘ in packge.use.

No harm done, but still no boot. Finally I figured out that grub-mkconfig assumed I used a /boot partition (I didn’t), and generated a wrong path to the kernel in grub.conf. Once fixed, the kernel started booting and I was hit by a problem once more, the boot stalled at the mounting of the / partition. As mentioned before, don’t forget to build in Ext4 support into your kernel!

Rebuilt kernel with Ext4, booting now… login: Damn! Forgot to set the root password. Where is that boot CD again?

While now the system booted and kind of worked, the / partition stayed read only. I had to remount it manually ‘mount -o remount,rw / /‘ to be able to fix anything. After some pondering and unsuccessful attempts to fix the issue, I remembered that I encountered this problem before while installing Gentoo on my SGI O2. It turned out to be the same problem, the kernel parameter root=/dev/hda1 won’t cut it, real_root=/dev/hda1 to the help! It has to do something with the temporary roots in ram during early boot and unmatched filesystem types, who knows…

Next came the screen resolution, where the familiar vga=791 kernel parameter did nothing. This time some Googling helped me to find some Arch Linux guys solution, including ‘insmod vbe‘ in grub.conf did the trick, vga=791 works now!

Well, I hope that this was as much Fun for you as it was for me; more on Funtoo next time.

All product and company names herein may be trademarks of their respective owners.

I noticed this, when I received some strange messages from cron. Basically my backup script started failing when svnadmin dump could not read a file. On closer examination, one of the DIFF files in the svn repository was damaged. This was bad, as the content is stored as a series of diffs, and a particular branch of the repository became unavailable.

The Subversion aka SVN repository is located on /var a separate partition. Since /var receives a lot of writes, I decided to migrate the whole /var to a different partition to avoid further bad blocks.

Armed with rsync and ddrescue I started the recovery process. You know the drill; format the new partition, rsync the content and ddrescue the damaged file. But this time ddrescue let me down. Now it was time for backups to step in.

Satisfied with a quick solution – restore from backup, I stopped all services which were using /var, unmounted it, mounted the new partition as /var and restarted all previously stopped services.

To my surprise the SVN still did not want to cooperate. I was getting a bad feeling about this, which was confirmed after examining all eleven weekly backups. In all of them the file was corrupted.

How did I not notice this earlier? Well, it was time to think about a new strategy.

I tried to run svnadmin recover mode, but that didn’t yield anything.

I don’t know the internal workings of SVN, but what I figured out is:

• content is stored as diffs
• each commit is one big file containing all changes
• there is file containing the number of the current revision

Luckily I knew exactly what files were committed in that particular diff, even better for me, it was a commit of only a few new big files! My best chance would be to recreate the corrupted commit.

When I manipulated the revision counter, the SVN server was tricked into thinking that the ‘current’ version was whatever I set it to be.

So I rolled back the revision counter to the revision before corruption, checked out the branch with TortoiseSVN into a new location, included the ‘new’ files and committed them to the repository. Re-setted the revision counter to original and voila, everything started working!

I guess this time I got more luck than brain. What would you do?

All product and company names herein may be trademarks of their respective owners.

In that case, where you end up with unrecoverable i/o errors, and your OS refuses to cooperate, don’t give up. Here comes ddrescue, well, to the rescue!

ddrescue is a GNU tool, similar to unix dd but better in handling i/o errors. It is multi-platform, but native to Linux. On Windows it’s available via Cygwin.

GNU ddrescue is often confused with dd_rescue, an unrelated and currently unmaintained project with a similar aim.

ddrescue can be used on any block device, including raw disk devices, partitions, or single files. It automatically uses some smart methods of access like direct i/o, different block sizes, reverse advance and repeated retries to maximize the chances of data recovery.

If you got just a few unreadable files (like what happened to me), then run ddrescue just on the damaged files.

# ddrescue -d –R -r 100 /damageddisk/somedir/damagedfile /rescuedir/recoveredfile

-d instructs to use direct I/O

-R retrims the error area on each retry

-r 100 sets the retry limit to 100

In my experience, if it does not succeed in 10 retries, it won’t ever… but you are free to try and hope for miracles

</tips&tricks>

So what does processor affinity do to a process on a multiprocessor architecture? This depends on the system architecture. The two common system architectures today are SMP and NUMA. SMP is the classic multiprocessing used by Intel up to the Core2 generation, and NUMA has been used for a long time by AMD Opterons and descendants, and lately also by the Intel Core i7 family. On SMP it does not matter, which data are processed on which core; however, the contrary is true on NUMA, where memory can be local or remote relative to the core used.

While playing with my workstation (Server 2008 x64 on a quad core Core2 Q6600 @ 3GHz with 8GB of RAM), I noticed a funny thing. The process I was running, a Python script, was jumping from core to core without any recognizable pattern. No other CPU intensive processes were running, only the usual system stuff. The script does some data analysis on quite large datasets – tens of gigabytes, which have to be processed as a stream, because of Python’s incredible memory hunger.

Before you start complaining that the script should have been parallelized in the first place, yes, it is. Parts of it at least are. But certain parts simply must be run sequentially.

So my question was, wouldn’t it be more efficient, if the single threaded process would run on one dedicated processor? Well, unless the process scheduler is über smart, I think it would.

See, the jumps from core to core probably because some serious cache trashing, so if the process runs on one dedicated core, the cache can be utilized much better. Unless, (and here is where my doubts come in), Python is an interpreted scripting language. The interpreter is quite complex with its own infrastructure, taking a lot of CPU power on itself. What if the scheduler is smart enough to split the different parts of the interpreter and the running byte code across multiple cores, and dedicate each piece of code to a core to avoid cache trashing? That would be nice! Not as nice as an auto parallelizing Python, but still, a smart feature to have.

I looked at some documentation about Server 2008 process scheduler, but could not find anything, which would confirm or disprove any of my theories. Apparently there are optimizations related to NUMA architecture, to minimize process core/cpu to memory distance, but nothing about maximizing cache utilization by jumping cores.

Therefore I ran my own tests. I tested on 32 and 64 bit version of Python 2.6.2, as well as Psyco – the python accelerator. Unfortunately, Psyco supports only 32 bit Python . All tests were run on my workstation as mentioned earlier. Times are in minutes and seconds.

Script1:
Processor Affinity                       All Cores      Dedicated Core
Python 2.6.2 32bit                              10:03                10:03
Python 2.6.2 64bit                              10:54                10:42
Python 2.6.2 32bit + Psyco             10:14                 9:38

Script2:
Processor Affinity                       All Cores      Dedicated Core
Python 2.6.2 32bit                              41:58                 39:46
Python 2.6.2 64bit                              40:09                40:00
Python 2.6.2 32bit + Psyco             30:12                29:38

I have run each test only once, because of time constraints, but I made sure that all tests run under equal conditions. As you can see, the affinity gain is minimal, but measurable, and specific to the executed code.

So generally I can say, yes, it makes sense to pin down a process to use a dedicated core, but the gain is dependant on the code used, and probably negligible. Ultimately, there are better ways to speed up your code.

What? How can a supposedly secure environment like a military installation or a hospital catch a worm? Panic everywhere.

Why is everyone so scared of Conficker? The worm basically does nothing! It only tries to dig in and wait. According to a build in the timer something should have happened on April 1st 2009. April 1st came and went, and nothing happened. Everyone was expecting a doomsday scenario, where the worm was expected to do something horrific, but nothing happened apart from an update to a newer version, and more waiting for commands. So far there have been 5 versions of the worm observed, labelled as A, B, C, D and E. They seem to be modifications of the original, as the author tries to get things right and build a bigger bot army.

Only the latest version E was observed as actually doing something: send spam, and install scareware. Technically, it’s not Conficker itself that does this, it’s the payload that it downloads and executes on demand.

So why is it so dangerous?

The payload is the keyword. The infected machine is at the disposal of the attacker to do anything he wants. An unknown payload executed on demand could be anything from DOS attacks to extortion, spamming to spying. It also updates itself, to enhance its own capabilities, and plugs entry points to avoid infection from competing worms. Very flexible isn’t it? All this is secured by hash signatures and encryption.

How does the thing actually spread?

It spreads in two ways:

• Network
• Removable Storage

The worm tries to attack a known vulnerability in the DCE-RPC service running on port 445, also used for various services needed by Windows file sharing. A patch for this hole was released in October 2008 – MS08-067; regardless, the worm still succeeded in spreading. Another way of spreading is old fashioned copying. It places a copy of itself on removable storage, and uses the autorun feature for infection. Also worth mentioning is a dictionary attack on administrative network shares, but this might not have been a very successful infection vector, because it seems to be missing in the latest versions of the worm.

How to spot the infection?

Conficker uses a number of self defense mechanisms, which are a giveaway. It disables a number of services -  Automatic Update, Security Center, Defender and Error Reporting. It also creates its own service to stay resident. The name is constructed from two random words from titles of other services. Another type of self-defense employed is the redirection of domain names related to AV products and the Windows Update.

An infection can also be spotted using a professional product. Most AV vendors offer a stand alone tool to detect a Conficker infection. GFI LANguard 9 is also capable of detecting infected machines remotely as well as detecting missing patches regardless of infection. Once an infection is detected a removal tool should be employed and the systems should be patched to avoid repeated infection.

Back to the question, how can this worm spread into supposedly secure institutions? Well, the problem mostly lies in people – People who are naïve or who do not follow the security guidelines set by the organization.

• Developers: Environment choices, running a whole Windows system on an embedded black box such as an MRI or a heart monitor might not be the best choice.
• Administrators: Not updating systems as required by choice; in some cases not being able to do it because of the black box nature of a system where only the vendor is allowed to update, thus leading to substantial delays in updating that might leave the system vulnerable.
• End users: Not following rules, and trying everything to work around restrictions.

So, what can we expect to see of Conficker in the future? According to some research, there are now only around 200,000 infected machines. However, this might be just the tip of the iceberg, because this includes only the latest version of the worm - version E.

Version E is set to expire on 3rd May 2009, but not the previous versions. Are all versions destined to eventually upgrade to E and retire? Did it serve its purpose? Is it going to be replaced by something else?

More questions than answers. The future will tell. The moral of the story so far – keep your systems updated.