Why the haitus? And what comes next?

You may have noticed there’s not been any real content on this blog in a hot minute. That’s because I made a career move: I stopped being an automation engineer/QA consultant, and began working on a new team in a more DevOps manner.

My team now is called Platform Ops. We’re a cross-functional team dedicated primarily to supporting and improving the lives of the developers at my company. The devs were reorganized into product-specific teams so they could be more agile, while we ended up with the cross-platform work:  implementing Docker  containers, supporting frameworks shared across teams, and responding to midnight emergencies when Operations has no idea what’s failing or who to call about it.

Now that it’s January and I’ve become more comfortable in my role, you can expect to see more articles on this blog. The content will be different; I’m going to talk about Docker, about keeping servers running, about testable infrastructure-as-code, about moving from a monolithic Waterfall environment to a DevOpsy microservice-friendly space, leapfrogging many of the in-between steps in our quest to please the unpleasable developers.

Watch this space!

Introducing: Terminus

So in my “Introducing: Agni” series, I explained the process I went through to get a used machine from Goodwill up and running to replace my previous living room server, Mercury, which exploded. I realize I left a loose end — I had to restore the mysql database from a several years old backup that I’d taken before I started backing up to the backup drive that died, and we just reconstructed the data after that point. Good thing a lot of the data was stored in flat files >.> I then set up regular backups to the same hard drive, intending to have them then copied to a backup drive once I had a replacement.

According to my records (meaning, my blog), Agni was purchased in February 2013. It’s kind of sad, then, that in September 2014, when I switched internet providers, I lost the ability to have a living room server at all. My apartment complex moved to a ethernet setup with a central router in the front office, and they do not allow port forwards past that gateway :(.

This is unacceptable, I told them, but in the meantime, my sites were all down and I wanted them up. So I finally caved and bought a VPS from DigitalOcean. (That link includes a referral code, by the way, if you want to go ahead and get $10 in credit. I used a code from a friend, and it ended up paying for 2 months of my new server setup, which was handy.) I went down my list of potential server names and settled on Terminus. Humming themes from Majora’s Mask, I set about transferring my files.

Having been working with SQL Server at work, it was a real treat to move back to MySQL, by the way. Yes, I know, enterprise, scaling, blah-de-blah, but really, the ease of use of the mysqldump command to take a full backup and restore it elsewhere from the command line streamlined the whole process.

The only snag I hit was when my ftp program lost about half the wordpress files because it couldn’t see them, which was annoying. I had to zip on agni, download via ftp,  upload via ftp, and unzip. But once I realized that, I was off and running.

I took this opportunity to correct a few unusual filesystem conventions I’d used on agni, and to tighten the security a little. I also took the opportunity to install linux-dash to make life easier on me when I go to check on the server, and as you can see, I bought a new domain name and uploaded a pet project I’d been working on a few weeks later.

I guess the moral is that this stuff is not hard in this day and age. It takes very little money, very little time, and very little effort to migrate to the cloud, and I’m glad I did. I went with the $5/month tier, which got me 512MB ram and 20GB disk space; the ram’s a little tight, but it’s a good chance to streamline and optimize. I run this blog, two other installs of wordpress, and a few custom php sites on the box, and after I installed some caching into the most image-heavy wordpress install (it’s a multi-site install), it’s been running like a treat. Granted, I have very low traffic, so we’ll see later if upgrading to a bigger box causes issues, but honestly, I can’t imagine it would. I also stuck with apache for ease of administering, since I’m used to apache by now, but I considered moving to nginx for a good hot minute.

I’ve never really been a hardware person; I prefer to work at the highest layer of abstraction that lets me get my work done. Why re-invent the kernel when I could be writing games, you know? I still have to find a good use for Agni, since I have the hardware lying around, but that shouldn’t be too hard. I’ve already got a few ideas floating around, but they’ll take time and money that I don’t plan to spend just yet. Maybe over the holidays when I take some time off work.

Agni lives!

So there’s this giant cloud hovering over Agni, and its name is Mysql.

Since I had the hard drive from mercury mounted instead of booting from it, I decided to just copy over the “live” database files instead of doing a dump and restore. I then went to bed. When I woke up, I saw increasing numbers of I/O errors followed by the end of the operation failing. Oops. I go to try again… and the drive can’t be read.

I reboot. The drive can’t be mounted.

I spent a few days messing with various linux tools to restore the drive, and nothing worked. So I grabbed the backup drive…and the blasted thing refuses to stay powered on. It’s a Maxtor OneTouch 4. Those things are friggan impossible to rip open. So that’s a project for a future date. I just hope there’s a semi-recent backup on it; I was terrible at remembering to take them.

But! There’s a silver lining! The wordpress installation went just fine, and it turns out my ISP doesn’t block port 80 (I thought they did, but it was my old router acting up, the new router lets the packets through just fine). So for now, Agni is up and running and ready to serve up my new sites! As soon as I recover my data…

Agni: Full brain transplant

My cable got here! YAY!


So I unpack everything, skim the instruction manual, and go to plug it in. The first thing I notice is that it’s got external power — yay! That means I could re-purpose this old small drive for the raspi, since it won’t need to draw power from the machine itself.


(Fun fact: the majority of the space in the box it shipped in was taken up by adapters for various international power supplies, lol. I recognize british and european)

At first glance, it didn’t look like the connector would fit into the slot:


But it seemed to plug in just fine when I tried it:


(insert innuendo here)

So I figured all was well and continued plugging in. When I put the USB into my laptop, it came up with found new hardware, installing, and then ready to go. Yay! It recognized the device as a new USB mass storage device. Double yay! But then… there’s no extra drive mounted to my computer. What?

The device manager sees a USB mass storage device as well, I just can’t seem to FIND it. The light for the drive on the connector is amberish, and the light for USB is red. I can hear the drive whirring softly, but it did that from the moment I plugged in the power, so…

The hard drive is a WD400BB, considered a legacy product, but the stats are still availible if you know where to look. It connects using 40-pin EIDE (also known as ATA-2). The manual for the cord says it handles both 40-pin and 44-pin IDE. There are differences between plain IDE and EIDE, but not ones a cable should care about, as far as I can tell.

That leaves… jumper settings? A quick google search suggests I jumper it like a slave drive instead of a master. It’s currently jumpered Dual (Master), so I unplug the USB, then power, then move the jumper, then re plug back in.

(as a side note, the drive gets hot when not in an enclosure at all >.>)

Aha! The drive light turned blue! Oh wait. Then amber again. Plug in USB aaaand… nope. Nothing.

Okay, so, what’s on this disk?


Dafuq am I looking at? Nothing jumps out as being of use. Lol, Windows 98.

Then it hit me. This drive? Totally formatted ext. Google suggests that Windows hides unformatted drives; maybe it just can’t read it? So I go get a program called Ext2Fsd, and sure enough:

ext2fsdIt even found the swap partition! Unfortunately, it won’t let me make a mountpoint via MountMgr (it just…. does nothing when I click “add”). I finally got it to work using “DefineDosDevice”, but I wonder what happened to MountMgr?

A little sftp magic, and I should be good to go! Except… Filezilla says “E:\ does not exist or cannot be accessed”. What? Win-R, cmd, “E:” gives me “The system cannot find the drive specified”. Awesome. And My Computer still sees no such drive letter.

Well let me try to add a mountpoint at H: with MountMgr and…. and now the list of mountpoints is empty. What?! Okay, this tool fucking sucks.

Let’s try Ext2IFS! Oh wait, that only supports Vista. Wait…. the top link on google says it’s the ONLY thing that works on win7. WTF? Let’s try running it in compatibility mode. Walk through the install, assign it to drive letter H:, go back to My Computer. Success! There’s an H:! Double-click:

formatFor the love of god, no!

Okay. So. They suggest running mountdiag.exe on it:


Stupid Windows… I wonder if it’d be worth it to make a batch file that launches an elevated command prompt and runs the command passed in?

Anyway, the real reason it didn’t mount is this:


I give up. Windows, you have defeated me. I plugged the stupid thing into Agni. Why didn’t I do that first, you ask? Because I’ve had issues mounting things from the command line in linux just recently (rasslefrassin raspi) and figured I’d be more comfortable using Windows 7 + sftp.

Telnet into Agni, lsusb shows the bridge just fine. lshw….. isn’t installed. Wut. Install it, it seems to be logical drive /dev/sdb. dmesg shows me it did, in fact, clean up from the dirty dismount. After much poking about, I discover it automounted just fine: to /media/4812427d-c3d2-4f5b-92bd-68cb31c75d08. Because THAT’S a logical place for it.

Anyway, copy the files to the right spot and voila! Agni up and running. Just gotta tweak the settings so they match the old ones and the old sites ought to be running smooth. Then it’s on to setting up WordPress and my new sites will be live too!

Agni: Up and Running

So I got food poisoning.

From marshmallows.

What is this I don’t even.


Anyway, Agni! I’ve ordered an IDE/ATA-to-USB connector cable, but it’ll be next week before it’s here, so in the meantime, let’s see what we have with Agni. Mercury, my previous box, used PS/2 connections for keyboard and mouse, but unfortunately, I’m only finding USB inputs.

Never fear, it came with a keyboard and mouse! ….oh. They’re ps/2 as well. Well. That’s. Helpful.

2013-03-02 11.07.52
This is a PS/2 connector btw.

So let’s see. I have a wireless usb mouse I’m using right now, and we have a usb keyboard attached to another computer, but we’d previously bought a kvm (Keyboard, Video, Mouse) switch that used ps/2 connections, so we’ll probably want a usb one of those in the long run. But I am determined to get this working. Determined, I tell you!

So we cannibalized a wireless USB mouse from a laptop and a USB keyboard from our desktop machine, figuring we’ll get it up and running and then control it via SSH until we can get the new kvm switch.

To install linux, I’ll need to make a bootable media so I can do the install. Except. We have a ton of DVDs, but it doesn’t read DVDs. It has a floppy drive, but it’s the only machine that does. I can’t seem to find any blank CDs, and I don’t want to reformat a USB stick since we’re using all of ours for data. Hmm. Decisions decisions.

Aha! My roommate to the rescue! He’s found a stack of blank CD-Rs and given me one. Now to download the AMD64 version of the debian network install (because I’ve got an AMD-brand 64-bit processor, as evidenced in post one).

Mercury ran xubuntu, keeping updated to each LTS version, but I think for Agni I’m going to just use ordinary debian + xfce. Since it’s a server, it can technically be run headless, but I have Chaos do tasks on the machine sometimes and he’s infinitely more comfortable with a GUI environment than a command line, so I like to put a lightweight GUI on the machine. XFCE is nice and small 🙂

Debian should also help avoid some of the weird issues I had with mercury installing apache into some non-default spot. I’m still not sure why that happened, but it seemed I had to modify every tutorial thanks to being on ubuntu instead of debian proper. Also he’ll match my new Rasberry Pi, which runs Rasbmc, which is Rasbian + xbmc, where rasbian is a debian fork for the raspberry. So they’ll match!

Wow that’s a lot of acronyms in two paragraphs. If you’re lost, never fear, have a marmoset:

Alright, back to work! So we need a usb keyboard…

but I don’t seem to have one? Goddamn, my tech stash is so inadequate. I’ve borrowed one from my husband’s laptop. Yet another thing to add to my list of stuff to purchase.

Anyway, with a borrowed wireless usb keyboard (James: “Won’t you need special drivers?” Me: “Betcha the install disk has them anyway.” Sure enough, It Just Works.), and a borrowed wireless USB mouse, we’re up and running! Agni now knows its name and speaks American English. I formatted two partitions out of the disk so I can keep the site data separate from the linux install in case I need to change distros or upgrade or something later on. It should also make backups easier.

Alright, debian installed, time to pop open a console and…

why is it black text on white background?! BIZZARE.

Anyway. Time to set up a static IP address. I just save off a backup copy of /etc/network/interfaces and change my settings… and it won’t come back up. Why did it fail? What is a “network” and what do I set it to? How about I strip the settings to the bare minimum? After ages of reading stuff off to one of my friends (since I’m having trouble reading this week, my vision’s been blurred due to some meds), we discover he didn’t hear a space when I read things off to him. Whoops.

Next problem: /sbin isn’t in our path? what? So I can’t use ifup… Fuck it. Reboot.

Nope, still not working. Okay. Restore from backup. Add interface eth0 to config file, but this time, use dchp instead of static. Test if it works now. Success! Ping google. Also success! Okay. So. That works for now. Screw it. I’m going to use my router to assign the IP address. ifdown, ifup, and we’re running.

Debian Wheezy and up install ssh by default, so now I can get into the box remotely. Huzzah. Except… connection refused. Joy. Maybe it didn’t install? apt-get install ssh… no, it’s there. Oh, maybe it’s not started? /etc/init.d/ssh start… nope. Could not load host key. Okay, that sounds like I need to regenerate the key somehow… more googling…

aha! ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key

So now SSH is up and running, time to install apache!


Agni: Unboxing

Alright, so in my last post I managed to figure out what, exactly, Agni is. Now the next most pressing question is: How the heck do I get inside it? I’ve got the tech specs now, so I’m fairly confident what I expect to see once I do get in, but a cursory inspection reveals no screws of any kind. Stupid hidden screw techniques.

My husband managed to figure it out though:

2013-02-26 08.42.27That little piece with the lock slides to one side, causing the entire side panel to pop off. Voila! And we’re inside!

2013-02-26 08.43.46It’s nice and compact, though I like my cases roomier so I can add to them. Definitely not going to be able to fit multiple drives in there. Of note, there does appear to be an optical drive that I’d missed in my earlier rundown (they don’t come with one by default I guess), and… snicker… there’s a 3.5″ floppy drive… lol.  Nice labeling on the cables though.

My first goal was to simply swap out the hard drive with the one I had on hand, already set up for web serving. Notice the tight space though: clearly you can’t slide the drive out of the bay like you would in a larger tower, not without removing the power supply. So how do you….

2013-02-26 08.45.05Oh! Those blue clips, when pressed together, cause the entire bay to undock, letting you lift the bay out and then swap the drive. Very clever, Dell. Very clever indeed.

(At work we have a system which actually has a bay open to the exterior of the machine, letting you simply flip some clips to disengaged and slide the drive entirely out without opening the case. I love that system. This is nice too though!)

2013-02-26 08.46.17Alright, now to slide the smaller but already configured drive in and…. uh-oh. Problem.

2013-02-26 08.47.32
Old drive on left, new on right

The old drive is… IDE, and the new is SATA. (The specs said something about slimline, but this doesn’t look slimeline, so I’m going to go with normal SATA) I clearly need an adapter  Unless….. I do have other machines in the house… and while transfer via ethernet takes for friggan EVER, it is feasible. Hmm…. I’ll have to ponder this further.

Introducing: Agni

Once upon a time, I mentioned to my father-in-law that I wished I had a sandbox server to play with my web development skills. Once upon a time, my father-in-law mentioned that he had an old desktop tower from a school that had recently upgraded that he didn’t have plans for. Eyes met, sparks flew, and my living room server, Mercury, was born.

Sadly, a few weeks ago, Merc’s motherboard shorted out. We pulled out the ram and HDD and tossed the case, but I’ve been looking for a good replacement ever since. Today, at the Goodwill, we found a likely candidate: a Dell small-form-factor PC, $70, in good condition.

2013-02-25 20.27.13

I’ve named this one Agni, following my trend of naming web servers after messenger gods (kind of). Here’s the sum total of what I know about Agni before beginning:

  • The sticker claims it has 1GB of ram
  • The sticker claims it has an 80GB hdd
  • The store clerk claims it works
  • The connectors look like this:


That’s it! Exciting, huh? (I can’t stop being dizzy even when I stop taking pain meds. I had to come home early from work and miss setting up the third box in my project-specific mini-network. I can’t go in to work tomorrow either if I’m this sick. So instead of feeling sorry for myself, I’m moving up projects I can work on).


So first I’ve decided to gather a little more info about the processing power. The sticker was in good shape, and it contains a model number: DCCY. A little googling indicates that that makes it an “OptiPlex 740”, which CNET claims runs the following specs:

  • AMD Athlon 64 3800+ / 2.4 GHz
  • 512.0 MB ram by default
  • 8GB ram max
  • Microsoft Windows XP Home Edition ,SP2, x32, with Media
  • NVIDIA Quadro NVS 210S graphics card

I don’t care about the graphics card whatsoever, given it’s going to be a server, but the ram capabilities are a lot nicer than we had before, and it was made in the XP era, so not that long ago (I’d not be surprised if the previous box was optimized for ME). Whatever it has on the HDD, I plan to blow away, since I have the HDD from the previous machine I can just boot from, so I don’t much care about the OS except to tell me when this was made. We know the ram is higher than the minimum, and hey, look at that, the processor is 64-bit, despite the original OS only being 32 (probably because this was meant to be a media center PC and likely existed before XP 64-bit came out, not that it ever had much adoption).

From what my more hardware-savvy friends tell me, AMD was better than Intel when 64-bit first started coming out (and yes, that processor dates to that period of time), so this is a good solid processor. This is way better than expected, to be honest. I expected something older and more limited. This is going to be great!