Let us now head on to configuring GFI MailArchiver to make use of SQL Express and to automatically roll-over from one database to another when the 4GB size limit of the SQL Express database is reached.

1.    To start off with, we need to pre-populate SQL Server Express with a number of databases for use by GFI MailArchiver. Roll-over by size limit in GFI MailArchiver 6.3 does not yet support the automatic creation of new databases to use when rolling over – the good news is that this feature will be available in GFI MailArchiver 6.4 which will ship later on this year.

• To create a new database, fire up the SQL Server Management Studio console, expand the node of the SQL instance you’re going to use, right click on ‘Databases’, and click ‘New Database’.

• Set a database name, and set the initial size for the ‘Rows Data’ to 1 GB and for ‘Log’ to 10 MB.

• Set autogrowth for the rows data to 100 MB at a time.

• Set autogrowth for the logs data to 10 MB at a time.

If you’re asking yourself why you should set the initial and autogrowth values so high the answer is that each time SQL Server needs to grow its file, performance is impacted. The less times SQL Server needs to grow its files, the better.

• We now need to set GFI MailArchiver to make use of the new database we’ve created in the active archive store. But before doing that, go ahead and create a few more databases for GFI MailArchiver to make use of when the 4 GB limit is reached on the first database.

2.    We now need to create an archive store based on SQL Express and set it as the active archive store.

• Fire up the GFI MailArchiver configuration console, open the node ‘GFI MailArchiver’ > ‘Configuration’ > ‘Archive Stores’ and click ‘Add Archive Store’.

• Set a name for the new archive store and click ‘Next’.

• Select ‘Microsoft SQL Server database and file system (including Express Edition)’. Please note this is the only option for which automatic roll-over by size limit is supported. Click ‘Next’.

• Select the SQL Server instance in which you created the databases for use with GFI MailArchiver. As for credentials, I suggest you use sa credentials for these operations so as not to get stuck on some permissions related issue. Don’t worry, these credentials will only be used to perform the necessary operations and will then be discarded (they are not stored anywhere). Click ‘Next’.

• Select the database you’re going to use for the active archive store. Click ‘Next’.

• Select the location on disk where you want to store the emails archived to this archive store (and, unlike me, keep in mind the disk I/O recommendations I made in part 1 of this post – in my case it’s a test environment). Click ‘Next’.

• You should now get a dialog indicating that database verification is complete. Click ‘Next’.

• Enter user names and passwords for GFI MailArchiver to create users in SQL Server which it will use to write to the database when emails are being archived, and read from the database when emails are being browsed or searched. Make sure that the password you configure satisfies any password complexity requirements on your machine/domain! Click ‘Next’.

• You should now get a dialog indicating that login creation was successful. Click ‘Next’.

• Select the location on disk where you want to store the search index files for this archive store (again, keeping in mind the disk I/O recommendations I made in part 1 of this post). Click ‘Next’.

• You should now get a dialog indicating that you have completed the archive store wizard. Click ‘Finish’.

• You should now be back on the ‘Archive Stores’ page. In the ‘Archive in the following store’ drop down menu select the new SQL Express based archive store which you have just created. Finally, don’t forget to click ‘Save Settings’.

• You have now configured GFI MailArchiver to make use of a SQL Express based archive store as its active archive store.

3.    Now we need to create a number of archive stores based on SQL Express so as to roll-over to them when the size limit on the active archive store is reached.

• In the GFI MailArchiver configuration console, open the node ‘GFI MailArchiver’ > ‘Configuration’ > ‘Archive Stores Management’ and click ‘Add Archive Store’.

• You should be pretty familiar with the wizard you get at this point, as it’s the same one we went through to create an archive store based on SQL Express and set it as the active archive store.

• Go ahead and create a few archive stores in ‘Archive Stores Management’ based on SQL Express.

4.    At this point all that is left for us to do is to configure GFI MailArchiver such that it will automatically roll-over to one of the newly created SQL Express based archive stores when the active SQL Express based archive store reaches its 4GB size limit.

• Fire up the GFI MailArchiver configuration console.

• Open the node ‘GFI MailArchiver’ > ‘Configuration’ > ‘Archive Stores Management’ and click ‘Add Archive Store’.

• Select the ‘Enable the swap archive store schedule’ check box and make sure the ‘Use a new Archive Store when it is full’ radio button is selected. Don’t forget to click ‘Save Settings’.

• That’s it! You have configured GFI MailArchiver to make use of SQL Express based archive stores and to automatically roll-over to a new SQL Express archive store when the active archive store reaches the 4GB size limit of its SQL Express Database!

• Please keep in mind that you will not be able to select the option ‘Enable the swap archive store schedule’ should you not set a SQL Express based archive store as the active archive store, or have archive stores in ‘Archive Stores Management’ which are not based on SQL Express.

• Furthermore, once you’ve enabled the option ‘Use a new Archive Store when it is full’ you will not be able to set the active archive store to one which is not based on SQL Express.

• As a final note, when the active archive store reaches the 4GB size limit of the SQL Express database it is based on and GFI MailArchiver rolls over to a new SQL Express based archive store, a notification such as the one below will be sent to the administrator’s email address as configured under ‘GFI MailArchiver’ > ‘Configuration’ > ‘General Settings’.

• Should GFI MailArchiver be rolling over to the last available SQL Express based archive store, the email notification will also notify about this. At this point it is important that you create a few more archive stores for GFI MailArchiver to roll over to once the SQL Express database for this last archive store reaches its 4GB size limit – otherwise email archiving will then stop!

In this 2 part series I have briefly covered deployment and configuration of SQL Express for use with GFI MailArchiver, as well as configuring GFI MailArchiver to make use of SQL Express and automatically roll-over from one database to another when the 4GB size limit of the SQL Express database is reached. Admittedly this could and should be a simpler process – and this is precisely one of the things we will be working on in the next release, GFI MailArchiver 6.4, due later on in Q2 2010.

On this vein, I thought about drawing up some guidelines for using SQL Express with GFI MailArchiver on virtual paper.

Deployment of SQL Express for use with GFI MailArchiver

Some of you might already have SQL Express deployed, so I will give you a couple of hints for things to make sure of before trying to use your SQL Express deployment with GFI MailArchiver. I also realize that some of you might not have SQL Express already deployed, so I will also briefly go through deploying it from scratch for our purposes. I will be using SQL Express 2008.

Before covering deployment itself of SQL Express, however, I have one recommendation for you, whether you’re deploying it from scratch or otherwise. Due to the nature of its task, GFI MailArchiver is a disk I/O intensive piece of software – it is continuously reading/writing emails to/from disk, it is continuously reading/writing data to/from database (which can itself be disk I/O intensive), and it is continuously indexing and performing searches on emails, which necessitates reading/writing to/from index files on disk (which also entails disk I/O). Customers tend to fall into the trap of storing emails, databases and indexes on one disk, sometimes even on the same disk on which the Microsoft Exchange Information Store databases are stored (which bring along with them even more disk I/O) – resulting in disk thrashing and severely degraded performance of the product. In order to avoid this, it is important to plan for a disk which is dedicated for the storage of your SQL Express databases. Should you wish to avoid having to grab another disk for this purpose, you might want to look at whether you have another machine on your network with some spare disk and CPU cycles on which you might want to deploy SQL Express instead. Given the flexibility offered by SQL Express you will probably find that indeed you do have such a machine (damn, I even installed it on a XP machine in my test environment!). That is an equally viable option with the added advantage of also offloading your server of some CPU cycles. But enough about that, let’s dive into what you need to do if you already have SQL Server Express deployed somewhere on your network.

If you already have SQL Server Express deployed on your network, it’s quite simple really:

1. Make sure mixed mode authentication is enabled.

• Fire up the SQL Server Management Studio console, right click on the SQL instance you’re going to use, and click ‘Properties’.

• In the Server Properties dialog box, select ‘Security’ and enable ‘SQL Server and Windows Authentication mode’.

• Click ‘OK’.

2. Make sure that the ‘SQL Server Browser’ service is started.

• Fire up the SQL Server Configuration Manager console and select ‘SQL Server Services’. Should the ‘Start Mode’ for the ‘SQL Server Browser’ service not be set to ‘Automatic’, right click on it, click ‘Properties’, and on the ‘Service’ tab set ‘Start Mode’ to ‘Automatic’.

• If the ‘SQL Server Browser’ service is currently stopped, right click on it and click ‘Start’.

3.    If SQL Express is installed on a different machine to your GFI MailArchiver machine, make sure that TCP/IP is enabled for the SQL Server instance you will use with GFI MailArchiver.

• In the SQL Server Configuration Manager console, expand the ‘SQL Server Network Configuration’ node, and select ‘Protocols for SQLEXPRESS’.

• Right click ‘TCP/IP’, and click ‘Enable’.

• You will need to restart the SQL Server service for this change to come into effect. Select ‘SQL Server Services’, right click on the SQL Server service for the instance you’re working on, which will be ‘SQL Server (SQLEXPRESS)’ if you’re using the default instance, and click ‘Restart’.

4.    If SQL Express is installed on a different machine to your GFI MailArchiver machine, and you have the Windows Firewall enabled on the SQL Express machine, you will need to open access to the SQL Server Browser service, as well as configuring the SQL Server Express database engine service to listen on a specified port and opening access to that port.

To open access to the SQL Browser service, you will need to open access to UDP port 1434.

To set the SQL Server Express database engine service to listen on a specified port, follow the instructions at http://msdn.microsoft.com/en-us/library/ms177440.aspx – ‘How to: Configure a Server to Listen on a Specific TCP Port (SQL Server Configuration Manager)’ making sure that you clear ‘Dynamic Ports’ and set ‘TCP Port’ in ‘IPAll’ too like below. You will then need to open access to TCP on the port you configured. More information here: http://msdn.microsoft.com/en-us/library/ms175043(v=SQL.100).aspx – ‘How to: Configure a Windows Firewall for Database Engine Access)’.

You will also need to restart the SQL Server service for these changes to take effect.

5.    That’s it –  your SQL Server Express deployment is ready for use with GFI MailArchiver!

Deploying SQL Express from scratch

I don’t want to make this boring, so I’m not going to post a screen shot of each installation step… but I will give you some useful information and hints on the matter:

1.    To start off with, a comment about prerequisites – if the platform you’re installing on happens not to have them, you’ll need to install the following:

• Microsoft Windows Installer 4.5
• Microsoft .NET Framework 3.5 Service Pack 1
• Microsoft Windows PowerShell

2.    If you’re installing SQL Server Express on an Active Directory Domain Controller machine, or on a machine on which the Windows Firewall is enabled, you might get some warnings in the ‘Setup Support Rules’ dialog as shown below.

• With respect to installing on a domain controller, I will refer you to SQL Server 2008 Books Online (November 2009) > Hardware and Software Requirements for Installing SQL Server 2008 > Installing SQL Server on a Domain Controller: http://msdn.microsoft.com/en-us/library/ms143506.aspx#DC_Support. In short, however, if you’re installing on a domain controller you will need to run SQL Server services under the SYSTEM account or you’ll need to create an account for the purpose – other than that, SQL Server Express Setup will not block you from installing, and you shouldn’t have problems unless you’re planning to change the machine in question from a domain controller to a domain member, you’re installing to a read-only domain controller, or you’re looking at a SQL Server failover cluster.

The SQL Server 2008 Books Online section on the matter also mentions that Microsoft does not recommend installing SQL Server 2008 on a domain controller “for security reasons” – granted, you don’t want anyone who manages to break into your SQL Server machine to also have access to your domain controller… Furthermore, you will probably find that with a larger number of users (my guess being anything more than a 100 users) the usage patterns of active directory and SQL Server on the same machine will be such that they impact each other performance-wise. If any of this is of concern to you, always remember that you can easily install SQL Server Express on pretty much any machine with some spare disk and CPU cycles.

• If you’re installing SQL Express on a different machine to your GFI MailArchiver machine, and you have the Windows Firewall enabled on the machine on which you’re installing SQL Server Express, you will need to open access to the SQL Server Browser service, as well as configuring the SQL Server Express database engine service to listen on a specified port and opening access to that port.

You will be able to do this once SQL Express is actually installed.

3.    Service accounts: When you get to the ‘Server Configuration’dialog, you will need to configure a couple of settings related to SQL Server Express services.

• As per Microsoft recommendation, I chose to run the SQL Server Express services under separate accounts, so I created an account, ‘sqlexpress’, under which to run the ‘SQL Server Database Engine’ service.

• Make sure you set ‘Startup Type’ for the ‘SQL Server Browser’ service to ‘Automatic’. This service makes it possible to discover a SQL Server instance on the network, so it needs to be started for GFI MailArchiver to be able to discover your SQL Server Express instance.

4.    In the ‘Database Engine Configuration’ dialog, make sure you select the ‘Mixed Mode’ authentication mode. You will also need to select a user to act as administrator for your SQL Server Express instance.

5.    If you’re installing SQL Express on a different machine to your GFI MailArchiver machine, once the SQL Express installer has completed successfully make sure that TCP/IP is enabled for the SQL Server instance you will use with GFI MailArchiver.

• Fire up the SQL Server Configuration Manager console.

• In the SQL Server Configuration Manager console, expand the ‘SQL Server Network Configuration’ node, and select ‘Protocols for SQLEXPRESS’.

• Right click ‘TCP/IP’, and click ‘Enable’.

• You will need to restart the SQL Server service for this change to come into effect. Select ‘SQL Server Services’, right click on the SQL Server service for the instance you’re working on, which will be ‘SQL Server (SQLEXPRESS)’ if you’re using the default instance, and click ‘Restart’.

6.    Almost forgot – if SQL Express is installed on a different machine to your GFI MailArchiver machine, and you have the Windows Firewall enabled on the SQL Express machine, you will need to open access to the SQL Server Browser service, as well as configuring the SQL Server Express database engine service to listen on a specified port and opening access to that port.

This can be done in exactly the same way as I described earlier in this post for existing installations of SQL Server Express.

7.    That’s it – you’ve deployed SQL Server Express and it’s ready for use with GFI MailArchiver!

In my next post I will run you through configuring GFI MailArchiver to make use of SQL Express and to automatically roll-over from one database to another when the 4GB size limit of the SQL Express database is reached.

The current incarnation of GFI MailEssentials owes its first design to early 2002 (if my memory does not fail me, we released GFI MailEssentials 7 somewhere around June 2002).

The general architecture was quite simple and straightforward:

• We hooked into the Microsoft IIS SMTP server using SMTP Event Sinks.
• Emails were passed on to a scan engine which was plugin based.
• Different anti-spam plug-ins implemented various spam filters.
• These anti-spam plug-ins called a library of actions (which are effectively what gets done with a spam email, for example, saved to a folder on disk, forwarded to a particular mailbox, moved into a particular folder in a mailbox, etc).

This architecture is illustrated in Figure 1 below – it worked transparently (or almost) and equally well both on Windows 2000 IIS SMTP and Exchange 2000 (and eventually Windows 2003 IIS SMTP and Exchange 2003), and it was around through six revisions of GFI MailEssentials – up to GFI MailEssentials 12.

Figure 1: GFI MailEssentials Architecture, v7 – v12.

One thing to note about this architecture is that the anti-spam scanning components all “live” within inetinfo.exe, which had the disadvantage that any instability in our scanning components would bring down IIS – including the SMTP server and also the IIS web server (i.e. any websites on the machine hosted by IIS would also be taken down…)

With the advent of Microsoft Exchange Server 2007, which was in 64-bit, we took out the scan engine into a separate 32-bit process, gfiscans.exe, which appears as the GFI MailEssentials Scan Engine service in the services console. Making use of out-of-proc COM, we started calling the 32-bit scan engine from the 64-bit transport agents written for Exchange 2007. This brought with it two main advantages:

• First of all, since the anti-spam scanning components now “lived” in a separate process, any instability in our scanning components would only bring down the GFI MailEssentials Scan Engine service (i.e. gfiscans.exe).
• Furthermore, any crash in our scanning components could now be recovered from, because the transport agents could detect that the GFI MailEssentials Scan Engine was no longer responding and start it up again – without any impact on email flow.

These changes were at first only rolled out on Microsoft Exchange Server 2007 installations of GFI MailEssentials 12. The architecture of GFI MailEssentials 12 on 32-bit IIS SMTP and Exchange 2000/2003 installations remained as in Figure 1 above.

Figure 2: GFI MailEssentials Architecture, v12 – v14.

When we released GFI MailEssentials 14.0, we introduced support for 64-bit IIS SMTP, and the same architecture as in Figure 2 above was also introduced on 64-bit IIS SMTP installations. The architecture of GFI MailEssentials 12 on 32-bit IIS SMTP and Exchange 2000/2003 installations remained as in Figure 1 above.

With the upcoming release of GFI MailEssentials 14.1, the architecture of GFI MailEssentials on all installations will be as illustrated in Fig 3 below.

• The scan engine is now always hosted in a separate 32-bit process, gfiscans.exe, which appears as the GFI MailEssentials Scan Engine service in the services console.
• Email sources – be they 32-bit IIS SMTP or Exchange 2000/2003 event sinks, 64-bit IIS SMTP event sinks, or 64-bit Exchange 2007/2010 transport agents – all submit emails for processing by the scan engine via out-of-proc COM.
• The various anti-spam filter plugins do their scanning within the gfiscans.exe process.
• Emails for which an action needs to be executed, such as spam emails to be forwarded to some other mailbox or moved to some folder within the recipient’s mailbox, are removed from the IIS SMTP or Exchange queue and queued for processing to a separate process.
• When an email is queued for an action to be executed, the raw email content is spooled to disk, and some metadata is queued to MSMQ.
• Actions are executed within the GFI MailEssentials Managed Attendant Service, the process name of which is contentsecurity.as.attendant.exe.
• An email which has been queued for an action to be executed is picked up by an actions coordinator which is hosted within the Managed Attendant, and passed to a set of plugins which implement the actions.
   

Figure 3: GFI MailEssentials Architecture, v12 – v14.1.

The new architecture in GFI MailEssentials 14.1 brings about a number of advantages and improvements:

• No matter what platform GFI MailEssentials is installed on, since the anti-spam scanning components now “live” in a separate process, any instability in our scanning components will only bring down the GFI MailEssentials Scan Engine service (i.e. gfiscans.exe).
• No matter what platform GFI MailEssentials is installed on, any crash in our scanning components could now be recovered from, because the sinks or transport agents can detect that the GFI MailEssentials Scan Engine is no longer responding and start it up again – without any impact on email flow.
• As opposed to the previous architecture, in which actions were executed within the scan engine, increasing the time the scan engine was taking to process spam emails, spam emails are now simply queued to the Managed Attendant, reducing the time in which a response is returned to IIS SMTP or Exchange in the case of spam emails, and thereby improving the performance of the scan engine.
• With the new architecture, when an action being executed on a spam email fails, that email is queued to what is called the recovery queue, where emails in the recovery queue are reprocessed so as to execute any non-executed actions. This makes for a more robust actions framework.

One action for which this makes a particular difference is the “Move to Exchange Folder” action, now known as the “Deliver email to mailbox: in Exchange mailbox sub-folder” option. With the previous architecture, when for some reason or another the Exchange store is not operational, the “Move to Exchange Folder” action would fail, and “Global Actions” would kick in, i.e. the spam email would instead be saved to disk or forwarded to another mailbox as configured. With the new architecture, such emails are instead queued to the recovery queue, which is reprocessed when the Managed Attendant is restarted.

• The new architecture also allowed us to support configuring multiple actions for spam emails so, for example, it is now possible to deliver a copy of a spam email to a sub-folder within the user’s mailbox, as well as forwarding a copy to another mailbox and saving another copy to a folder on disk.
• This new architecture also takes us a number of steps in the direction of supporting Exchange 2010, but that is a post in itself for another time.

As can be seen, although it is a minor release, GFI MailEssentials 14.1 is a substantial update to the product. If you haven’t yet tried out the beta, I recommend you check out the GFI MailEssentials 14.1 beta forum at forums.gfi.com, or visit beta.gfi.com for details on how to join the GFI MailEssentials beta mailing list.