Snafu Blog

I’ve recently being trying to get my part in the mail hubs finished before heading off on paternity leave for a month. I had great fun today trying to get Mailscanner and Sophos syslogging properly. I’d installed all the software, stoked it up and found that though it was running nothing was getting logged. After checking sysylog.conf and running logger it seemed that the problem was with the perl Sys::Syslog module. After much fiddling around I upgraded it to 0.18 from 0.13 and Sophos started logging, which was great. Fired up MailScanner again but still no logs

I decided it was time to delve into the logging code for MailScanner and I noticed that if the OS was detected as Solaris it set the logging mode to udp. This won’t work on our Solaris 10 zones so I altered the code to use native mode. Fired it up again and it worked! I always seem to have to fiddle with MailScanner to get it to log properly, so I’m making a note of what I got up to here so it’s easier in a years time when I have to fiddle again

I’ve recently been playing with Sympa at work as it’s been decided the project needs to move on even though James is sunning himself in New Zealand

Sympa is a  mailing list system, in the same vein as Majordomo (which we currently use) and Mailman. A big advantage of Sympa over the others is the close integration with LDAP. Along with being able to auth against it (which is great as it means the users don’t need another username and password) it can create mailing lists based on LDAP searches. This means we could easily create a mailing list for just us Carters It also allows you to manually add people to these dynamic lists as well which is useful.

Creating the lists in this way means you get the benefits of all the access controls that Sympa provides, which is much more flexible than what is available with a generic LDAP dynamic list. They are also dead easy to setup as we had a test list working in a couple of minutes.

All in all it looks like Sympa will be a huge improvement over our current system, I can see the users particulary liking the web interface, no more clunky email commands (unless you really want to use them )

Well, I’ve finally got time to go over the things I’ve been working on since the start of the year. I’m going to try and make more of an effort to do these entries as it’s useful to jog my memory about exactly what I’ve been up to The days and weeks tend to be so hectic that I usually forget what I’ve done.

The first week and a bit of this year were very busy. I was involved in two major tasks that we wanted to get done before the bulk of the users came back, these were migration to a new printing accounting system and the move of files to a new filestore.

The printer accounting move was the first thing that needed doing. We’ve been using a good home grown system (written by a member of the Computer Science department) for many years and it was starting to get a little long in the tooth compared to some of the more modern products on the market. Features that we especially wanted were support for many different printer types, a web interface and easier management functions. After a thourough review process we finally settled on Papercut.
A lot of work was done before the start of term by a quite a lot of technical and user services to try and make the transition as seamless as possible. In the end it went pretty well with the actual import of users only taking a couple of hours compared to the days we thought it might do. We’ve had some interesting problems with some of the Unix queues scattered round the place and found quite a few printers we didn’t think existed Most problems have now been ironed out and the few issues we reported back to papercut are being worked
on.

Moving people from the old filestores of Croft and Compton (old P3 Xeon 500Mhz machines) to our new clustered and quotered system which has storage on the SAN went smoothly. The job wasn’t particularly technical but involved a fair amount of work as I needed to backup all the old data as the end of the day when it wasn’t in use and then restore it before 10am the next morning. Had to do some of the work from home which my other half wasn’t that impressed about
We’re now at the state where Compton has been switched off (it’ll make a good coffee table for someone) and Croft is just waiting on us moving the last tricky department. When they go it’ll free up a lot of space in the machine room which we can populate more efficiently.
The new clustered solution offers much more space than the old servers had (and can be grown on demand) and is much more resilient as the filestore is on a pair of clustered 1850s. This means the chances of an interruption to service are reduced and we can even maintain service whilst patching the machines. From a management point of view having quotas makes our lives much easier as rogue users can no longer fill up entire disk partitions any more!

Once these projects were out of the way I’ve spent my time logging more queries with Sun about Kentmail issues. I’ve mainly been concentrating on Outlook Connector issues and think I’ve finally got to the bottom of why users have been having odd problems writing to shared diaries. The long and short of it is that Connector seems to incorrectly order the ACEs meaning that a deny is applied before the allow in same cases, which is broken. I’ve had some fun convincing Sun of this and I suspect I may now more about how Calendar server works than some of their staff

I’ve also been doing my share of the large amount of Remedy queries we get after a vacation and I’ve been catching up on all the little jobs I’ve been putting off whilst working on the projects. These have included upgrading our monitoring software (Nagios) to the latest version and sorting out the queries that have been emailed to me directly. Sigh.
I’ve even tidied my desk and drawers and found the notes I made when I joined the University as a Helpdesk Operator 6 years ago. Sadly most of the notes are now irrelevant as most of the stuff they refer to has been retired. Oh well

I spent a fair amount of time sorting out how postmaster email is routed after we received a couple of reports that anyone trying to send email out to postmaster at other domains had their email snaffled and sent to our postmaster account

A look at the Exim config showed a rather confused routing for postmaster that involved rewrites and redirects to another account, back to postmaster and then finally back to the other account again for delivery. I removed all this gumph and replaced it with a single alias to route postmaster correctly. I also managed to cut our oldest mailhub, mercury, out of the loop meaning it’s days are numbered . As usual the actual change took very little time (once I’d figured what out what on earth it was doing) but testing to make sure I hand’t knackered the config took quite a while. It’s in service now and seems to be working, which is good.

A side effect of this is that spamchecking for the Postmaster account now works, a feature requested by the people that look after it. We’re slightly unsure whether this is wise given that postmaster is a likely recipient of Freedom of Information requests. However, they claim the spam folder is checked carefully daily (though you wonder if there is a point spam checking for the account in this case…).

As a quick project I was also asked if I could find a way of checking whether certain windows computers were turned on and run this at regular intervals (this is for lecture theatre pcs). Nice and easy if you have the IP/hostname but a bit more fiddly if you only have the NETBIOS name (yuck). A bit of searching found nbtscan, which while it doesn’t do quite we I wanted does do netbios lookups incredibly quickly and is likely to be a useful tool. In the end I used nmblookup to query the WINS server to look for registrations for the computers which seems to work. If I can get IPs for all the computers I can use nbtscan which might be a bit quicker but it’s difficult to find the IPs out as the hostnames have no correlation to the netbios name. Hohum

As I was off yesterday I spent a good portion of the day catching up on Remedy queries assigned to me and sorting my email out. I had a load of queries assigned about our email system, which is quite rare as the non-JES bit is usually fine (and was in this case as it turned out).
One of the queries was quite an impressive bit of spam. It’s engine had gone far enough as to look up the MX records for our domain and insert a couple of fake header lines into the email. If it had actually got the version number and ID string correct it would have been even better Not entirely sure why they bothered with this though, given how prevelant spam is.

Also had a meeting with one of our departments about taking their backups forward (and out of the 20th century ). Though we though it might have been a little difficult to part them from their system it looks like the promise of diskstaging has won them over. Now to do some testing with EBS 7.3 to ensure that it stages in a way that is convenient for them and also provides the level of data security we like.

As a final thing I spent some time firefighting the odd thing that went wrong here and there, pretty much like normal really

Had a meeting about the new Print Accounting system today (Papercut). Mainly to check progress (slightly behind) and to make sure we haven’t missed anything. Things seem to be progressing fairly smoothly at this stage, which is good.

Also spent some time helping Q&A with a query about mail scanning. Looks like the document I wrote nearly a year ago is finally going to be published so that users have an idea about exactly what the mail system does and does not allow.

Spent a good hour updating Kentmail Sun cases and closing the ones that patches have fixed. Deployed a test CE patch to the testmail system and did some basic testing to ensure it fixed the bugs it was meant to. Handed over to testers for more extensive testing before deployment on the production.

Spent an hour or so today reconfiguring the external mailhubs with the new simple config. No complaints so far so I guess it’s working That was my main task of the day, spent the rest of the time doing Remedy queries and catching up outstanding emails which is pretty common for a Monday. Left at 16:00 as it’s my cover night, which was nice and quiet as usual.

Spent the day doing mostly routine stuff. Big backup day as it’s Friday so I spent about 2 hours over the course of the day shuffling tapes, making sure the libraries were stocked for the weekend and putting the used tape in the firesafe. Need to run a recalibration on the Vishnu jukebox when the backups have finished as the picker alignment seems to have gone out of whack when picking tapes from the caddy. Have to see about logging a call if it does it again.

Badminton at lunch, good games as usual. We’re trying out the new scoring system which runs to 21. Doubles took a bit of getting used to but only having a single serve per side seems to make the game fairer and more dynamic. Won 2, lost 1 if I remember rightly. Used some feather shuttles again which I love, though as they get destroyed quickly it could become rather pricey

My new exim config seems to be working well, other than a slight hiccup with the spam routing for the test mailhub which is now fixed. It got missed out of the new config as the two configs that were running weren’t in step as they should have been. This is the major reason that I’ve moved to the one config model as it makes configuration easier and less error prone. Also simplified the creation of virtual domains so that we no longer need to edit the exim configs to add a new one. Now you simply add the domain name to one file and create another file to hold the aliases and the routing happens automatically. Switching this from text files to a DB or LDAP would seem like a good plan for the future though.

Started off today going through my email to answer all the queries I get emailed instead of going into the helpdesk system like they should. Queries I get this way tend to be forgotten for weeks as the helpdesk system prods me and my email doesn’t. Still, after an hour of answering and deleting my inbox is once again a fairly sane size.

We had a short powercut today as well (apparantly affected most of Canterbury). It was enough to get our PCs to reboot but thankfully the UPS saved any of the critical services going down though it did highlight a couple of machines that need putting on the UPS. Nothing mission critical thankfully. As usual we’re putting together an incident report, though one issue today highlighted is the need to control access to the machineroom during an incident. Everyone and their dog wants a piece of the action which make our job checking equipment and the environment more difficult.
The only major problem is that some of the cardlocks have packed up. They failed safe into secure mode but it’s still a pain in the arse and worrying they haven’t survived a fairly minor failure…

Once the excitement was over I spent the afternoon  tidying up the Exim config on our mailhubs. The University was a very early adopter of Exim (pre V1 ) and has been running a continually upgraded version of the config for years. This meant that it was beginning to get seriously unreadable and not very efficient as some of the things it was trying to do were pretty much obsolete.
I therefore spent the afternoon pruning, commenting and adding various things to the config to bring it into the 21st century. Another bonus is that I’ve managed to get MailScanner to user a single config file instead of the two it was using before and we now only spawn two master exim processes instead of three.
I’ve also added in a load of resource handling config so that the service will degrade gracefully under load and hopefully offload some onto it’s siblings.
After some testing offline I’ve now bought the config into service on a single hub for testing. All seems to be fine and relay testing pronounced the hub secure so I haven’t accidentally opened us up. Provided all goes well overnight I’ll deploy the config to the rest of the hubs tomorrow.

Now that this basic work is done I’m thinking more about the other changes I want to make. These include

• Better virtual domain support – Needs to be more dynamic than text files, maybe ldap integration?
• Block more extensions at rcpt time – If we do this earlier, rather than later as we do now, it will save a lot of processing time
• AV scan at rcpt time – Dump the viruses before we even accept them.
• Block on certain RBLS or spam score at rcpt time – Dump really obvious spam before we accept
• Graylisting – May help somewhat, has had success in the library

It’s nice to actually get a little bit of time to sort Exim out, I’d forgotten how flexible and powerful it is. Maybe it’s time for another Exim course this year

Had a meeting over at the print unit today to discuss the image library project. I built the original box some 18 months ago but the project stalled for some reason. The people behind it have now got it going again so a meeting was called to try and find out a bit about the technical aspects. Thankfully it all seems fairly straight forward and from the ops point of view we just need to do routine admin and patching, which is good.

The meeting lasted from 10:30 till just before lunch. Badminton at lunch and then a hospital appointment straight afterwards until 15:30. Finally got some proper work done in the afternoon, mainly catching up on Remedy queries and sorting backups out as it’s my week.