FreeBSD: Using Open-Xchange on FreeBSD

If you go looking for a usable webmail application, then you might end up with Open-Xchange (OX for short). Some larger ISP’s are using OX as their webmail application for customers. It has a multitude of options available, using multiple email accounts, caldav/carddav included (not externally (yet?)) etc. There are commercial options available for these ISP’s, but also for smaller resellers etc.

But, there is also the community edition available. Which is the installation you can run for free on your machine(s). It does not have some of the fancy modules that large setups need and require, and some updates might follow a bit later which are more directly delivered to paying customers, but it is very complete and usable.

I decided to setup this for my private clients who like to use a webmail client to access their email. At first I ran this on a VM using Bhyve on FreeBSD. The VM ran on CentOS6 and had the necessary bits installed for the OX setup (see: https://oxpedia.org/wiki/index.php?title=AppSuite:Open-Xchange_Installation_Guide_for_CentOS_6). I modified the files I needed to change to get this going, and there, it just worked. But, running on a VM, with ofcourse limited CPU and Memory power assigned (There is always a cap) and it being emulated, I was not very happy with it. I needed to maintain an additional installation and update it, while I have this perfectly fine FreeBSD server instead. (Note that I am not against using bhyve at all, it works very well, but I wanted to reduce my maintenance base a bit :-)).

So a few days ago I considered just moving the stuff over to the FreeBSD host instead. And actually it was rather trivial to do with the working setup on CentOS.

At this moment I do not see an easy way to get the source/components directly from within FreeBSD. I have asked OX for help on this, so that we can perhaps get this sorted out and perhaps even make a Port/pkg out of this for use with FreeBSD.

The required host changes and software installation

The first thing that I did was to create a zfs dataset for /opt. The software is normally installed there, and in this case I wanted to have a contained location which I can snapshot, delete, etc, without affecting much of the normal system. I copied over the /opt/open-xchange directory from my CentOS installation. I looked at the installation on CentOS and noticed that it used a specific user ‘open-xchange’, which I created on my FreeBSD host. I changed the files to be owned by this user. Getting a process listing on the CentOS machine also revealed that it needed Java/JDK. So I installed the openjdk8 pkg (”pkg install openjdk8”). The setup did not yet start, there were errors about /bin/bash missing. Obviously that required installing bash (”pkg install bash”) and you can go with two ways, you can alter every shebang (#!) to match /usr/local/bin/bash (or better yet #!/usr/bin/env bash), or you can symlink /usr/local/bin/bash to /bin/bash, which is what I did (I asked OX to make it more portable by using the env variant instead).

The /var/log/open-xchange directory does not normally exist, so I created that and made sure that ”open-xchange” could write to that. (mkdir /var/log/open-xchange && chown open-xchange /var/log/open-xchange).

I was able to startup the /opt/open-xchange/sbin/open-xchange process with that. I could not yet easily reach it, on the CentOS installation there are two files in the Apache configuration that needed some attention on my FreeBSD host. The Apache include files: ox.conf and proxy_http.conf will give away hints about what to change. In my case I needed to do the redirect on the Vhost that runs OX (RedirectMatch ^/$ /appsuite/) and make sure the /var/www/html/appsuite directory is copied over from the CentOS installation as well. You can stick it in any location, as long as you can reach it with your webuser and Alias it to the proper directory and setup directory access).

Apache configuration (Reverse proxy mode)

The proxy_http.conf file is more interesting, it includes the reverse proxy settings to be able to connect to the java instance of OX and service your clients. I needed to add a few modules in Apache so that it could work, I already had several proxy modules enabled for different reasons, so the list below can probably be trimmed a bit to the exact modules needed, but since this works for me, I might as well just show you;

LoadModule slotmem_shm_module libexec/apache24/mod_slotmem_shm.so
LoadModule deflate_module libexec/apache24/mod_deflate.so
LoadModule expires_module libexec/apache24/mod_expires.so
LoadModule proxy_module libexec/apache24/mod_proxy.so
LoadModule proxy_connect_module libexec/apache24/mod_proxy_connect.so
LoadModule proxy_http_module libexec/apache24/mod_proxy_http.so
LoadModule proxy_scgi_module libexec/apache24/mod_proxy_scgi.so
LoadModule proxy_wstunnel_module libexec/apache24/mod_proxy_wstunnel.so
LoadModule proxy_ajp_module libexec/apache24/mod_proxy_ajp.so
LoadModule proxy_balancer_module libexec/apache24/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module libexec/apache24/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module libexec/apache24/mod_lbmethod_bytraffic.so
LoadModule lbmethod_bybusyness_module libexec/apache24/mod_lbmethod_bybusyness.so

After that it was running fine for me. My users can login to the application and the local directory’s are being used instead of the VM which ran it first. If you notice previous documentation on this subject, you will notice that there are more third party packages needed at that time. It could easily be that there are more modules needed than that I wrote about. My setup was not clean, the host already runs several websites (one of them being this one) and ofcourse support packages were already installed.

Updating is currently NOT possible. The CentOS installation requires running ”yum update” periodically, but that is obviously not possible on FreeBSD. The packages used within CentOS are not directly usable for FreeBSD. I have asked OX to provide the various Community base and optional modules as .tar.gz files (raw) so that we can fetch them and install them on the proper location(s). As long as the .js/.jar files etc are all there and the scripts are modified to start, it will just work. I have not (yet) created a startup script for this yet. For the moment I will just start the VM and see whether there are updates and copy them over instead. Since I did not need to do additional changing on the main host, it is a very easy and straight forward process in this case.

Support

There is no support for OX on FreeBSD. Ofcourse I would like to see at least some support to promote my favorite OS more, but that is a financial situation. It might not cost a lot to deliver the .tar.gz files so that we can package them and spread the usage of OX on more installations (and thus perhaps add revenue for OX as commercial installation), but it will cost FTE’s to support more then that. If you see a commercial opportunity, please let them know so that this might be more and more realistic.

The documentation written above is just how I have setup the installation and I wanted to share it with you. I do not offer support on it, but ofcourse I am willing to answer questions you might have about the setup etc. I did not include the vhost configuration in it’s entirely, if that is a popular request, I will add it to this post.

Open Questions to OX:

So as mentioned I have questioned OX for some choices:

  • Please use a more portable path for the Bash shell (#!/usr/bin/env bash)
  • Please allow the use of a different localbase (/usr/local/open-xchange for example)
  • Please allow FreeBSD packagers to fetch a “clean” .tar.gz, so that we can package this for OX and distribute it for our end-users.
  • Unrelated to the post above: Please allow the usage of external caldav/carddav providers

Edit:

I have found another thing that I needed to change. I needed to use gsed (Gnu-sed) instead of FreeBSD-sed so that the listuser scripts work. Linux does that a bit differently but if you replace sed with gsed those scripts will work fine.

I have not yet got some feedback from OX.

Categories: FreeBSD, Howto Tags: Tags: , , , ,

The epic spam battle from SpamAssassin (10 + year user) to rspamd.

For many System Administrators that have public facing Mailservers, it is an ongoing battle.. SPAM. Since there is money to make, it will never ever go away, but we can try to mitigate this.

Introduction on my usage of anti-spam products:

For many moons I have used the SpamAssassin product in various forms, simply as a client to check every email on delivery, as daemon where multiple servers check one instance, as part of MailScanner where a single (replicated) database was responsible for storing all bits and pieces combined with local additional rules. This worked fine for years, but, our external MX servers are not the most powerful machines in the world. We need to be selective on what we load on them. And the ever increasing spam battle just makes sure that your memory and processing power is going faster then the system(s) could continuously deliver.More rules, more Anti-Virus, more regular expressions, more downloading, parsing and re2c’ing files that gets harder and harder for the systems every time the amount of rules etc increases.

I already mentioned that this worked fine for years. I switched to MailScanner for our MX’es not too long ago, and I am happy with that, except that it takes additional load on the machines, and will only judge about mails when they are already in. I contributed to MailScanner and specifically to the MailWatch project for reasons of LDAP authentication and more of those things, where I found space to improve. Even though I like the system very much, it is not how I want to prevent Spam from coming in. It might be a good fit for you though, it offers a quarantine where users can selectively release emails and mark them as spam and such and you can generate emails that send the amount of potentially missed emails and a link to them etc. Some of our users where happy with that as well, and so was I.

Limitations of our handling of email:

But, resources were becoming a problem. Yes I can upgrade my external MX’es ofcourse and load them with more memory and CPU power, but that costs money. Money that is hard earned in the hosting world, because there is plenty to choose from, even if we give the best prices around, it still takes multiple additional customers to warrant the higher bills (that is not taking into account that profit would be fun for additional investments in the company so that our users can get even better products).

So, given the saturated market, I was not going to spend additional money on our machines just yet. Another thing is that I wanted to prevent spam from coming into the machine in the first place, so reject them at the border where possible, so I do not have to cater them. (See it as border patrol, it’s easier to prevent things coming in, then to handle them once they are in). I noticed that several email servers where already doing that when we forward mail for our domains to lets say gmail or other companies that people are happy to use. Those servers, like gmail, either rate limit you or they just deny the emails before you are able to send them. Leaving you with the problems instead of the gmail user itself. Magnificent. But how does that work? for Postfix, which I use that means using a milter, specifically in this case rmilter, which binds into the product on the SMTP level, checks signatures stored, scans the content and verifies with bayes and a neural network whether this is OK or not, and then either rejects it before processing it, greylisting it when it seems spammy or adds an header to the message and forwards it to it’s final destination. If we are the final destination, then the header is taken into account and the message is automatically put in the Spam folder, or for gmail/hotmail users this is the ‘unwanted email’ folder or whatever it is called nowadays. I have put filters in place, that learn your behaviour, so if a message is put in the Spam folder and is not spam and you move it back to for example the INBOX, then the system learns that it should not mark it as spam and try to do better next time.

The product: rspamd

But what product delivers that ? After talking with a postmaster team member of FreeBSD, I found out about rspamd, and that the author is a fellow-FreeBSD-committer as well. I implemented it (it took some time to learn the curve, but essentially it is rather easy, try it!). It has less load then the various spam assassin products and additional applications that support it (like mailscanner and mailwatch), it does not need a webserver by itself etc. So it reduced my memory footprint with around 400mb’s continuously of less memory usage. That is a whole lot of you have mb’s to spare instead of handing them out.

How does it globally work?

I also configured rspamd to behave like the following;

  • Both our external MX’es have a local bayes-classifier and various other local databases. I used the suggested three database tier on the machine and I extended both machines to use stunnel to contact eachother over the stunnel to the remote database. I changed all configuration options to not only use “servers = “localhost”;“ but instead “servers = “localhost,localhost:26379”; and spreading that across every redis line I could find. I then restarted rspamd on both machines and noticed that there is a lot of things going on, it seems that everything is written and read on both machines. Using the webinterface, you’ll sometime get errors, not sure why that is, and history is not always consistent. but it’s for management purposes only so not very problematic in this case. Both MX’es are checking on their localhost, and “also_check” the remote machine over an internal private network that I have setup.
  • Our internal machines that handle the delivery of the email, use both MX’es as rspamd instance as configured in rmilter. They do not handle anything themselves, except for Virus Scanning (which is also done on the MX but as well on the local machine, but only for email not received from the MX’es, like outgoing email). That means less overhead for those machines and only using the two machines where we know they are working. I also extended these machines to use redis on the MX’es instead of locally and configured them both in the configuration, again using stunnel. rmilter uses the redis databases to store and save messages that we have send and get replies and such. In the future if rspamd is by itself capable of handling this, rmilter will be taken out and only rspamd will run like mentioned.

Learning spam/ham messages:

For now this seems to work very well, I have implemented a dovecot script that triggers when someone moves a message from spam to inbox (‘learn-ham.sh’) and from inbox or other mailboxes to the spambox (‘learn-spam.sh’).

The contents of the files look like the follwing, where learn_spam and learn_ham are in the appropriate places ofcourse.

#!/bin/sh

data=$(cat)

echo “$data” | /usr/local/bin/rspamc -h MX1 -P <secret password for MX1> learn_spam

echo “$data” | /usr/local/bin/rspamc -h MX2 -P <secret password for MX2> learn_spam

Ofcourse it takes additional understanding of how emails work, how your environment works and what is acceptable or not. On the course of just a few days we processed more then 10k of emails (yes there are many providers doing more emails, everyone has it’s own perks ;-)). and we have learned more then 60 emails in just a day after enabling users to do their own training.

One note:

A little note about the rejecting of spam, we only reject spam when the message is really spammy and cannot be easily something else. Most emails that I saw so far are forwarded with an additional header instead of being rejected and the emails that are rejected are really spam. Users will never ever see them, which is good enough for my environment but might be something different for your environment. Please dry-run it at first to see how it matches your environment.

References:

The script for learning spam under dovecot comes from: https://kaworu.ch/blog/2014/03/25/dovecot-antispam-with-rspamd/#comment-2436333602 user Alex.

The documentation I used for rspamd comes from http://www.rspamd.com itself.

The sieve filters that I use for dovecot are from Dovecot itself https://wiki2.dovecot.org/HowTo/AntispamWithSieve

Custom blacklisting of domains and such come from: https://gist.github.com/kvaps/25507a87dc287e6a620e1eec2d60ebc1

 

FreeBSD Dutch Documentation Project

So. It had been a while before I had proper time to look into the Dutch translation efforts again.

History

Due to various reasons not discussed here, I was not able to see to a proper translation. Rene did a lot of work (thank you for that Rene!).

The PO system

First of all, i am going to discuss a bit about the PO system, which is a gettext way of doing translations. It chops texts into msgstr’s (message strings) and then translates those strings using msgid’s. Same lines are translated the same, this might be a good option, unless the context changed between the lines and then you might get ‘google translate’ kind of ways.

Back to the story…

After getting time again to see this through I noticed that we started using the “PO” system, using gettext. Our handbook (for example) is now translated into one huge book.xml file which is then cut into msgstr’s that can be translated to msgid’s. For this I use the poedit application (the PRO version) so that I have counters and translation suggestions from the online Translation Memory(TM) that we all develop. I also contribute the FreeBSD translations back to the TM so that everyone can profit from it.

I am now first synchronising the Glossary because that didn’t change much with the current online translation and working my way back to what had been translated already and translating the missing bits and pieces in between. Mike (co worker at Snow) also did a tremendous job in getting this into better shape the last year which had not yet been merged back to the online variants because it was not yet complete. I can use that information though to generate a manual handbook variant of that version and use that to even further use the current translation effort into the gettext/po system.

Biting the bullet

As one of the first translation teams to use this, I expect to hit some rocks on the road. For example, there are lines that do not need translation, mailing list names are the same in every language, perhaps the description changes but not the ‘realnames’. Same goes for my entity (&a.remko) which does not change, nor my PGPkey. And if those things change, they require changing over all translation efforts as well as the original english version. We are looking into a way to ‘ignore’ them for the po system but include them when building. So that pgpkeys and such are always up to date.

I also had been discussing this with Vaclav the developer of poedit, and he mentioned that it does not matter much, because when a line changes and you update the po, those lines will be invalidated and need ‘retranslation’ for the entire string. So that all gets us in interesting situations that we did not encounter before. I am biting the bullet myself after we have discussed this a few years ago and I hope that the entire project can benefit from that.

Alternative options, pre-translate, merge current translations automatically?

And yes, a valid question would be, cannot you merge the current translated information into the po system automatically. If every word was on the exact same spot and line, yes this might be an option. Sadly because of grammer and different wording (longer/shorter) this changes rapidly from line 1 already and is thus not easily done. If you have suggestions however, we are always willing to listen. Please join us on ‘translators@FreeBSD.org’ so that we can discuss those things better :-).

Categories: Uncategorized Tags: Tags: