Weird Looking: Sorry about your emails

Here’s a horror story that I’ve thought about from time to time, which is probably a bit exaggerated but is based on my past experience with certain models of hard drives.

I’ve decided to tell it in ordered-list form, for no real good reason as far as I can tell.

1. Decide that current storage/backup solution is not good enough.
   
2. Choose a RAID mirror for storage redundancy.
   
3. Learn that all of the drives should be the same size and type.
   
4. Find a good deal on hard drives.
   
5. Purchase four identical hard drives and a RAID controller.
   
6. Set up RAID and install data.
   
7. Go on about your business.
   
8. Discover that the first drive fails.
   
9. Find a replacement, but continue running the system.
   
10. Order replacement, but this drive is hard to find these days and it cost more than it should have.
    
11. Wait for replacement, only to discover another drive is failing in the same manner as the first.
    
12. Decide that two failed drives might be a coincidence and order another replacement.
    
13. Scream in horror when a third drive fails.
    
14. Find out that the four drives you bought must are from a defective lot.
    
15. Frantically make a backup of your last copy of your data before the last drive fails…too late!
    

I’ve seen many hard drives of the same brand/model start going out at about the same point in their life.  I’ve also owned a motherboard with a RAID controller that managed to corrupt data when used with my particular video card.  That one doesn’t even seem possible?  Somebody screwed up big on that one.

In retrospect, I should have held out for more drives.  Somebody was too cheap to spring for anything fancy.  Like a case.

(I don’t know why I can’t spot my grammar errors until I read my own post in the syndication feed.)

If Frank could run a BBS on a computer on a chair without a case, you can run an email server like that.

I’m not sure if hard drives are designed to work upside down and diagonally, but I don’t make hard drives.  Supposedly, they are pretty resilient and reliable these days.

haha… no, they were mounted properly beforehand and all.  I just pulled them out today so I could test them out in another machine and left them dangling there in frustration.  I doubt the upside down part is a problem, but vibration would probably shorten their lifespan if they weren’t mounted long-term.

The really ghetto parts are the fan clipped to the support bar and the fact that the case has no faceplate, buttons, LEDs, slot covers, etc.  And that I was running a mail server with only one backup hard drive.

I’m trying to convince <i>The People who Sign the Checks</i> that I need a real server if they want to keep this going.

Hoisted by my own petard.

Was this a permanent repository for email like an IMAP or Exchange server?

I’m tempted to start forwarding all of my email to my gmail account, since they tend to do a better job at making regular backups than I do.

Plus, I can’t imagine ever filling my 2+ gigabytes.

It was web-based, just as bad.  I did write a pop3 server, but I don’t think many people used it.  I didn’t write an IMAP server because, well, have you ever looked at the IMAP protocol?

I forward all my stuff to gmail these days.  It’s nice to have available no matter what computer I’m on, and I’m never going to fill 2GB 2777 MB.  Heck, they’re adding storage to my account faster than I use it.  And their spam filters are pretty good.  Definitely better than when I wrote my own.

I’ve been trying to use the Google GMail Loader to transfer my mail from Thunderbird to Gmail, but I have had little luck.  I can’t get Google’s SMTP servers to work (they use a nonstandard port, I think), and my ISP’s SMTP servers cut me off after transferring about 10 messages.

I haven’t been able to use Muujware’s SMTP server in a while, but maybe when they upgraded our server (or, technically, sold out to a bigger company) the settings changed.

Interesting.  My ISP blocks all traffic on port 25 except to their own servers (I generally proxy through an SSH tunnel to get around it).  I imagine that would get in the way of using a program like this.

The mail server’s primary drive is being sent off to data recovery world.  For the price, they should give me a kiss too.

Port blocking…that seems to be the problem.  I am able to use Google’s SMTP server in Thunderbird because they support a non-standard port, but the Google Gmail Loader doesn’t seem to support changing the port.  I tried the server:port syntax and it doesn’t seem to work, and I’m not able to use any other SMTP servers through my ISP.

Luckily, I think I’ll be able to modify the Python source to connect on a different port.

Oh wait, I found a slight snag.  I had to set up Thunderbird to use TLS authentication.

I have a feeling that could be a bit harder than simply changing the port number.

Okay, somebody else had already fixed this problem, so I ran their version (with TLS support).  Apparently you get a protocol error for every message you send, but the message still transfers.

It shows up as a message from myself and sorts to the top, but that’s okay since I really just want to be able to search my old mail.  I rarely browse it.  Maybe I’ll just tag all of it as “Imported” so it’s easier to spot.

It’s a shame it’s not easier to move mail from one program to another.  Google should provide an mBox importer.

I give up!  After sending transferring several hundred messages, I apparently exceeded my daily quota.  Since there’s no good way to resume the transfer, I think I’m going to delete everything I sent and forget about it.

Innerestin'

I wonder how the gmail filesystem implementations move emails into the inbox, and if they’re bound by the same quota.

I think some of those libraries use the same web service calls that Gmail’s AJAX mail client uses.

I’ve seen tools for exporting Gmail to mbox, and supposedly Google is working on an export feature.

As for import, I don’t see anybody tackling it.  *sigh*

And now I won’t be able to send email until tomorrow.  Apparently sending mass email to yourself is spamming.

I came across the mailredirect extension for Thunderbird, which lets you pass a message on to somebody with headers more-or-less intact.  It even marks redirected messages with a red arrow so I know which ones I have already transferred.  I’m currently moving a month of messages at a time.

I also found out that my web host’s SMTP server can be used without SSL on the SSL-SMTP port (with a login, of course), bypassing any ISP filtering.

Let’s just hope that my Gmail doesn’t complain to my web host.

Here’s evidence that this was probably a bad idea:



I’m decided to transfer 10 a day using my ISP’s server.  This could take a while.

As annoying as blocked SMTP ports are, I’m pretty glad for them given the amount of spam that comes from regular broadband accounts.

I think I’m going to force any host that has “adsl” or the IP address with dashes in the reverse PTR to suffer greylisting.  Oh, and the entire pacific rim too.  I’ve never gotten a real email from asia.  They’re all spammers, all of them.

I’m more and more becoming and advocate of a new protocol.  It would need an SMTP gateway, but it could automatically greylist everything coming through the gateway.

Shutting down bot-nets would go a long way to help, too.  Maybe we need a managed operating system.

Let’s try that again: managed operating system.

I get the feeling most of the machines that make up botnets are running warezed copies of XP without updates.  Even a fantastic OS-level solution would take years and years to show results.

I’d advocate mandatory SPF for every domain.  Then we’ll have to get rid of registrars that are lax about the identities of domain owners.  Domains that allow any IP to send with SPF (gmail et al) will have to implement some sort of rate-limiting sender verification.

That won’t stop spammers, but the fewer options they have, the more effective blacklists are.

Hmmm…Windows Genuine Advantage is responsible for spam?

Delicious.

Sounds like a good article for Slashdot, provided you take the opportunity to evangelize Linux.

Then make a s/Linux/Mac/g version and submit it to Digg.

I got a shiny new hard drive containing about 30 GB of emails back from whoever.  I shut down my machine at work, installed it, hit the power button… and nothing happened.  The machine wouldn’t POST.  Wouldn’t even initialize the monitor.

I unplugged it, pulled the hard drive out, hit the power button… nope.  Disconnected all non-essential peripherals, tried with just one stick of RAM, then the other, made sure the PCI cards were seated properly… aaaand nothing.

So seriously, is it too late for me to become a plumber or maybe carpenter?

So, I procured a new machine and hard drive enclosure.  I’d like to keep XP instead of installing Linux this time so that I have IE at work.  Naturally the drive they sent was formatted with EXT3FS.

After a few hours of fooling around, I decided this was the best ext2/3 driver for Windows because it’s the only one that didn’t crash the computer.

So now I have access to all 400 brazillion femtobytes of data and I need to figure out how to upload that to the server.  Oh, except the ext2 driver generates read errors on softlinks.  Every ftp/sftp program I’ve tried gives up on the entire upload after one read error.  I have a LOT of softlinks.

I’m currently using xcopy with some magic switches to create an error-free ntfs copy of the data, whence I should be able to do something with it.

I seen where Gmail lets you import from IMAP accounts now.