As a Systems Administrator, I have spent my share of time in various data centers. So, I was keen to read Ben Rockwood’s Personal Must-Haves. He wants a Leatherman, a particular mug, a particular water bottle, an iPhone and a particular bar code scanner. Honestly, it kind of felt like reading a marketing advertisement. I like the idea of a bar code scanner that can dump ASCII as if it were a keyboard device:
Your laptop will register it as a keyboard, so when you press the button to scan the contents of the barcode are “typed in” where ever you like, which means you can use it with Excel just as easily as my prefered auditing format, CSV’s created in vi.
I explained that I keep my liquids in the break area, so I don’t need any fancy mugs. And I don’t know if an iPhone supports making calls over wifi, which is important because mobile signal is often poor to none in a datacenter, nor have I any idea if the camera is all that handy for quick, low-light macro shots.
My own list would include:
In the Cage
A proper toolbox
An inventory of cables in a variety of sizes that match your color scheme.
Velcro cable ties
Label maker
A USB DVD-ROM/CD burner
Some blank DVDs and CDs
Some USB memory sticks
Spare server parts
In the Cage – Networking and Comunications
Wifi access point to a DMZ or sandbox.
A specific port on an Ethernet switch configured for the guest VLAN, and a long, loudly-colored cable reserved for connecting to it.
Dynamic DHCP (Seriously: sometimes your NetOps people don’t grok the convenience of DHCP . . .)
A terrestrial VoIP phone with a very long cable.
In the Cage – Human Sanity
Noise-cancelling headphones
Earplugs
A pocket camera, stashed in the toolbox, which can take good close pictures in low light with minimal shutter lag.
A power brick for your IT-issue laptop. (Especially if the cage is DC power!)
A sweater or jacket.
I like wearing the earplugs, then earbuds under noise-cancelling headphones, or over-the-ear earphones. You block out most of the noise and can enjoy some tunes while doing what is often non-thinking physical labor.
You never forget your first computer.
For Christmas of 1984, Grandpa gave us a
Commodore 64. A couple years later we
got a disk drive, and eventually we even
had a printer. Before the disk drive we
had to buy programs on cartridge, or
type them in to the basic interpreter
line by line. Mostly I just played
cartridge games.
Eventually we got a modem, and I could
talk to BBSes at 300 baud in 40 glorious
columns. (Most BBSes assumed
80-columns.) I was happier when I got a
1200 baud modem for my Amiga, which
could display 80 columns of text.
In my second year of college I
discovered the joy of C programming on
Unix workstations, which led to my
present career as a Unix SysAdmin. I
spend my days juggling multiple windows
of text, generally at least 80x24. /djh
I have been concerned that as Yahoo decays, that Flickr may at some point no longer remain a good place to host my photos. I do wish someone would create a competing service which supported the API. Some kid made Zooomr a few years ago, which was to sport a feature-complete Flickr API, but as best I can tell the kid moved to Japan and lost interest in Zoomr, which remains an abandoned stepchild.
Picasa? The desktop client is kind of neat but I don’t much like the web interface. It feels like another one of those one-offs Google bought but then had no idea what to do with it. Anyway, it’s just not my thing.
So, I took a look at SmugMug, who have been trying to lure Flickr refugees, but the consensus seems to be that if you like Flickr, SmugMug can not approximate Flickr. (The biggest concern for me is the loss of the “title” attribute. I’ve got 7,500 images online acquired over a decade . . .)
This is disappointing, because I like SmugMug’s promise of customization, and I have never been afraid to roll my arms up to hack on templates, HTML, and CSS to achieve my desires. Perhaps in the next few years SmugMug will become a little more flexible such that it can easily achieve what I want:
Individual pages for my photos
Support for a “title” attribute
An ability to browse title/descriptions (Flickr “detail” view)
Every so often I have this idea that the WordPress Gallery feature should take some steroids and create a friendly, Flickr-API-compatible hosting environment, which you could then customize just as much as you can customize a self-hosted WordPress blog . . . but that is very far beyond my code abilities and free time.
I’ll give Amazon.com credit for making their mp3 downloader available to Linux users:
I clicked on Ubuntu, and Chrome downloaded amazonmp3.deb and the Ubuntu Software Center fired up and told me:
Wrong architecture ‘i386’
That was frustrating. I sent a note to Amazon.com thanking them for their Linux support and asking them to please consider rolling some x86_64 packages. Then I asked Google for advice, and got this fine post:
(Your browser should hand the package off to an installer, else you’ll just do something like sudo dpkg -i Downloads/getlibs-all.deb.)
3) Run getlibs!
Looks something like this:
0-13:08 ~$ sudo getlibs /usr/bin/amazonmp3
libglademm-2.4.so.1: libglademm-2.4-1c2a
libgtkmm-2.4.so.1: libgtkmm-2.4-1c2a
libgiomm-2.4.so.1: libglibmm-2.4-1c2a
libgdkmm-2.4.so.1: libgtkmm-2.4-1c2a
libatkmm-1.6.so.1: libgtkmm-2.4-1c2a
libpangomm-1.4.so.1: libpangomm-1.4-1
libcairomm-1.0.so.1: libcairomm-1.0-1
libglibmm-2.4.so.1: libglibmm-2.4-1c2a
No match for libboost_filesystem-gcc42-1_34_1.so.1.34.1
No match for libboost_regex-gcc42-1_34_1.so.1.34.1
No match for libboost_date_time-gcc42-1_34_1.so.1.34.1
No match for libboost_signals-gcc42-1_34_1.so.1.34.1
No match for libboost_iostreams-gcc42-1_34_1.so.1.34.1
No match for libboost_thread-gcc42-mt-1_34_1.so.1.34.1
The following i386 packages will be installed:
libcairomm-1.0-1
libglademm-2.4-1c2a
libglibmm-2.4-1c2a
libgtkmm-2.4-1c2a
libpangomm-1.4-1
Continue [Y/n]?
Downloading ...
Installing libraries ...
3.1) If, like me, you got “no match for libboost” as above, or you get “amazonmp3: error while loading shared libraries: libboost_filesystem-gcc42-1_34_1.so.1.34.1: cannot open shared object file: No such file or directory” then do this bit:
So, I recently replaced my home router with an Actiontec GT724WGR. But I neglected to tell the TiVo, so it quietly started to lose programming data. Then I tried getting the TiVo to talk to the new router, but it wasn’t connecting. Long story short, it appears the TiVo is not supporting WPA2, so on the Actiontec, I went in to Wireless Setup > Advanced Settings and kicked it down to WPA and now the TiVo is updating its information successfully.
But it took some doing to figure that out. It would be helpful if the TiVo N02 error page touched upon the limitations of TiVo’s support for wireless security.
That “Sea of Glass” building from a different perspective.
While clicking around in a WordPress install last night I discovered that under Appearance > Media I can change the size of images posted through WordPress.
The G2 is fast as heck. It has all the cool new Android apps, and T-Mobile let’s you do tethering out of the box. We moved our apartment last month and setting up a wireless access point on my phone was braindead easy and plenty fast while we waited for the DSL installation. Everything works faster, and the battery life is better to boot.
The keyboards has a generally nice feel to it. But . . .
The biggest drawback is the lack of a number row on the keyboard. Really irritating to have to press ALT to type numbers. Entering “special” characters is a bitch-and-a-half. For example, to type a < you have to type ALT-ALT-long-press-j. WTF? Also, I miss the scroll wheel. There is a button on the phone that sometimes-but-not-always works as a directional pad to surf through a text field but I have learned to stab my thumb at the screen until I manage to land the cursor where I want it. (What I really miss is the Sidekick 2 direction pad.)
It is a very very nice phone with a short list of dumb shortcomings.
Automation saves time both by getting tasks done more quickly and by ensuring consistency, thus reducing support calls.
Start with a script that outputs the commands that would do the task. The SA can review the commands for correctness, edit them for special cases, and then paste them to the command line. Writing such scripts is usually easier than automating the entire process and can be a stepping stone to further automation of the process.
A simple script that assists with the common case may be more valuable than a large system that automates every possible aspect of a task. Automate the 80 percent that is easy and save the special cases for the next version. Document which cases require manual handling, and what needs to be done.
There have been times in my career when I have felt that people look at automation as a one-off task. “Write a script to automate this task.” Other times I have been asked how I go about automating things, and my answer is that automation isn’t a task so much as an iterative process:
I try to do the task at least once, maybe a few times.
Along the way I document what I had to do to get the job done.
From there, I follow the documentation, and refine edge cases as I go.
After that I’ll write a script, and get it working. (do)
I revise the documentation to explain how to use the script. (document)
And then, I use the script to complete requests, fixing the script when it fails. (refine)
Often enough I have been called upon to help another group automate something. That is a little trickier because I may never get the chance to do the task. Hopefully the other group has written some documentation, otherwise I’ll have to tease it out of them. The whole refinement process is the most obviously collaborative. I’ll document “use the script . . . it should do this . . . if it does something else, send me details.”
There is also the question of what-is-worth-automating. I believe it is the “Practice of System and Network Administration” which breaks tasks in to four buckets: frequent-easy, frequent-difficult, infrequent-easy, infrequent-difficult. You get the most payoff by focusing your automation on the frequent tasks. Easier tasks are generally easier to automate, so go ahead and start there, then turn your focus on the frequent-yet-difficult tasks. If you regard automation as an iterative process, then infrequent tasks are that much harder to automate. This is doubly true when the task is sufficiently infrequent that the systems have a chance to evolve between task execution. Infrequent tasks tend to be adequately served by well-maintained documentation in lieu of an automated process.
A last note for infrequent tasks. Part of the difficulty for these can be a combination of remembering to do them, and finding the correct documentation. One approach to “automating” an infrequent task would be to write a script that files a request to complete the task. This request should of course include a pointer to the documentation. For example, I have a cron job which sends me an e-mail to complete a monthly off-site backup for my personal web site. The e-mail contains the list of commands I need to run. (And yes, the daily local database backups are executed automatically.)
Several leading institutions fail, leading consumers to a rush on the remaining institutions, causing them to fail. A cascade of failures brings the whole system crashing down until the central authority undertakes a massive, unprecedented intervention to bring the system back to normal. At first, the degree of central intervention required is underestimated, but in time sufficient resources are brought to bear and the complex system recovers.
Interestingly, Skype’s network is actually a peer-to-peer network. It is a complex system which normally proves highly resilient, with in-built safety mechanisms to contain failure and ensure reliability. But under the right circumstances, failure can cascade. I couldn’t help but read that as a metaphor for free-market economics, which can usually take care of itself, but will enter a fugue state often enough to require a strong authority to intervene and put it right.
As a SysAdmin, the Skype network sounds like a very interesting beast. I figure that an action item against a future failure might be to provide a “central reserve bank” that monitors the health of so-called supernodes and automatically fires up large numbers of the dedicated mega-supernodes in the event of a widespread failure. (And such a strategy could well exacerbate some other unanticipated failure mode.)
Good news! Southwest Airlines offers wifi on my flight! Only $5 introductory price! I have to try this out!
The service is “designed by Yahoo!”
It is kind of really really slow to make connections.
Wait . . . WTF is this?!!
Yup. Southwest Airlines wifi does HTTP session hijacking to inject content in to your web pages.
This is a perfect illustration of the need for net neutrality: your Internet Service Provider should not interfere with your ability to surf web pages. This would be comparable to your phone company interrupting your telephone calls with commercials. Outrageous! Wrong! Bad!!
(On Mei’s computer there are actual ads in the blue bar on top, but my AdBlock plugin filters those.)
It gets worse from there. On the “designed by Yahoo!” experience you can surf on over to Yahoo! just fine. But I’m a Google man. Here’s what Google looks like:
Work-around #1: On sites that support them, use HTTPS URLs. Those are encrypted, so they can’t be hijacked. So, where http://www.google.com/ fails, https://www.google.com/ gets through!
But my little WordPress blog lacks fancy-pants HTTPS. And the session hijacking breaks my ability to post.
Work-around #2: If you have a remote shell account, a simple ssh -D 8080 will set up a SOCKS proxy, and you can tell your web browser to use SOCKS proxy localhost:8080 . . . now you are routing through an encrypted connection: no hijacking!
Update: they charge is $5/segment, so $10 if your plane stops in Las Vegas, and you get to type your credit card number a second time. Though, on the second segment, Google loads okay, but I still had to route through the proxy because the magic header was blocking WordPress’ media interface.
The Gogo Wireless on Virgin America always worked way better than this, and Google covers the cost over the holidays. And as far as I know: no session hijacking!
I saw this float across my Google Reader yesterday, thanks to Tom Limoncelli. If you are a sysadmin in an environment fixing to do more virtualization, it is well worth a skim:
NOTE: this isn’t an anti-virtualization rant, more of a “things to watch out for” briefing.
Some of my take-aways:
Sure we’ll have fewer physical servers, but the number of deployed systems will grow more quickly. As that grows so too will our systems management burden.
As the system count grows faster, we may hit capacity chokepoints on internal infrastructure like monitoring, trending, log analysis, DHCP or DNS faster than previously assumed.
Troubleshooting becomes more complex: is your slow disk access an application, OS, or hardware issue becomes also a potential host OS, networking/SAN or filer issue as well.
Regarding troubleshooting: we may add another team to the mix (to manage virtualization) while trouble-shooting has an increased probability of requiring cooperation across multiple teams to pin down. Increased importance on our ability to cooperate across teams.
Change management impacts: small changes against a larger number of systems sharing architecture snowball even more. One can add something to the base image that increases disk use by 1% for any one system, but multiply that across all your systems and you have a big new load on your filer. (1,000 butterflies flapping their wings.)
Reduced fault isolation: as we have greater ability to inadvertently magnify increased load and swamp network and storage infrastructure, we have a greater ability to impact the performance of unrelated systems which share that infrastructure.
The article also cautions against relying on vendor-provided GUIs because they don’t scale as well as a good management and automation framework.
Ah, the other thing noteworthy there is the ACM Queue magazine is now including articles on systems administration. (I subscribed to the system administration feed.)
A wiki feature I would like to see: when a node has not been edited for a certain period of time, it sends a message to the author and contributors, to review the node for relevance, and update or delete the node, if appropriate. The period of time should be configurable, and the feature should be able to CC the message in to the local request management system.
The check should be repeated on multiples of the period of time. For example, if you want to review your nodes once per year, the first year comes, the author reviews the node, sees no need to make any changes, then another year passes, you get a reminder to take another looksee . . .
This would be useful especially for an operations environment, to ensure that the shared knowledge hasn’t gone too stale, keeping the wiki resource relevant. I sincerely hope this feature already exists in a few systems!
Also useful for tracking down dependencies: I was reminded that for FreeBSD, what I would wish to call pkg_which is pkg_info -W, which will tell you what package a file came from.
Potentially nifty: a text-to-speech utility that writes its output to an audio device, which you could set as your system’s sound input. This way you could have “conversations” in your headphones via VoIP or Skype without having to make disruptive noises or emitting sensitive information in a shared environment like an open office or a library.
Hrmmm.
sudo apt-get install epos
sudo /etc/init.d/epos start
say "this does not work"
sudo apt-get remove epos
sudo apt-get install espeak
espeak "hello there"
But it appears there is no good way on Linux, anyway, to tell a command to dump its audio output to the microphone. Bah.
Wednesday, November 3
I got my sutures out the other day. I’ve still got the band-aid on my chin, cleaning and re-dressing twice a day, until the skin is no longer broken. Doc said I’d be shaving regular next week.
Thursday, November 4
Nice: I got the WordPress for Android App working thanks to Dan at Automattic pointing out I had an SVN-corrupted xmlrpc.php file.
Annoying: No idea how I can possibly type < or > with this nice-but-crippled G2 keyboard.
I made it up to San Bruno today, aboard Caltrain. Today is the first time I took my bike on Caltrain and it really wasn’t as bad as a colleague made it sound. The yellow tags are impossible to find though, so I made a couple labels to stick on my bike indicating what stations I ride between. In San Bruno, I stopped at the curb where I face-planted last week. Nothing special about it. I got to the office way faster than I have by walking. Probably 20 minutes compressed to 5 or not more than 10. The only unpleasant part is crossing El Camino. The crossing I chose had a crosswalk on only one side, so I had to cross against traffic to get rolling across the street.
Friday, November 5
Well, I finally have a new watch band . . . my old one broke such that it would no longer attach to my wrist, but I could dangle it off my belt. Now I have to get used to looking at my wrist again.
Monday, November 8
Some days I just feel blue. Like a dark cloud is over my heart. Not unlike overcast weather. Fortunately I often have the self-awareness to understand that, whether or not I know its origin, this is likely just a passing cloud, and the best way to weather it is to just take life on as a normal day, perhaps with an added dash of industriousness to stave off the natural slothitude that a funk brings on.
I was just reading in The New Yorker about different ways that salaries are determined. I identified with having accepted the “authority ranking” or feudal model earlier in my career, when what I valued most was the opportunity to work, to serve and build and learn. Back when I still kind of hoped that wages were fair, that bosses valued the contribution of their employees. Back in 1999 this even seemed true: the startup I was working for noticed that I was kicking butt, and ramped my salary up by 20% after my first six months on the job, and again after a year . . . everything was noble and virtuous.
But nobility and virtue don’t often last. Recessions hit, money dries up, eager young employees burn themselves out and haven’t a clue what to do about it. Layoffs come . . . a decade later I have left for greener pastures as many times as I have been layed off. Welcome to the “market pricing” model of economic interaction.
I find some difficulty feeling passionate about being a market priced, employment-at-will agent. Just as sex feels better with someone you love than when you’re getting paid for it, so too does work. Fortunately, the current gig offers competetive compensation, likely stability, and the chance to relate to coworkers over several years. I guess that is as close to “married” one can get in our industry . . . and yet of course I keep my eye out for new opportunities.
And I frequently worry about getting too comfortable, getting a little flabby in the skill set because there is plenty of work I would have to do at a startup that is off-my-plate at a large company. Better attain depth where I needn’t worry over breadth, eh?
Thursday, November 11
I had a meeting today that I haven’t been looking forward to. Basically, I have spent most of my career at smaller companies, where I tend to have a lot more say over how we do things, and where the simple, obvious, light-weight solution will tend to carry the day. But at a large company, there are enough competing interests that the way we do things is often not up to me, and is far more complex and open to error (in my view) than how I would go about doing it.
Note an editorial bias, right? Of course I have a high opinion of my own way of solving a problem. Doesn’t mean my approach is the right one . . .
So, at the end of this meeting, I got McCoy in my head. “I’m just a simple country Doctor.” Well, I’m just a small-shop sysadmin, serving on a corporate flagship. I don’t really understand or approve of everything that is going on, but that isn’t my problem. I focus on getting my own job done and I am happy to give Captain Kirk a piece of my mind, but at the end of the day the Captain gets what the Captain asked for.
Wednesday, November 17
From today’s work log:
Scout around a bit as to the advantages of managing system configuration
files in git. Git’s strength appears to be strong branch and merge
capabilities, working offline from the central repository, and the
capacity for fine-grained commits. Disadvantage is a steeper learning
curve. Anyway, we could potentially allow staff to grab a local branch
make several changes, review changes and reject those that proved
infeasible, then submit changes back to the central repository. Later,
a change management team could review differences between the central
repository and the stage / production repositories, then selectively
merge changes to the more stable environments in an appropriate manner.
I had lunch with Mei today at a Chinese place on Castro St in Mountain View. At the next table I overheard some guys talking about the size of the Oort Cloud if Earth were the size of a grain of sand . . . hard-core nerdy lunch conversation. I recounted that when I got off the light rail the other day I heard one guy explaining to another guy the theory behind anti-matter reactions that power the warp propulsion system in Star Trek. When you live here you live in the pulsing underbelly heart of nerd-dom. I kind of like it.
Thursday, November 18
My bicycle lights came in from Amazon.com and I tried them out yesterday on the ride to the light rail. (Between the weather and a recent injury I’m usually reluctant to ride all the way to work.) The front light was somewhat occluded by the basket so last night I moved it to a helmet mount, which required some careful trimming of a cross-member atop the helmet so the thing would fit, but nothing likely to compromise structural integrity.
Yesterday I also received my replacement G2. I got it up and running and it went and updated itself. It made a big todo about “wifi calling” which . . . uses minutes? Really? REALLY?! So, I’ll provide the bandwidth and you’ll charge me . . . but it also quietly enabled Tethering, via USB and WiFi. I’m using it now. I heard a rumor that T-Mobile was fixing to charge an extra fee for tethering. Hopefully though at the moment they’re content to charge customers to make telephone calls via their home wifi networks.
(Oh yeah, and I’m on the light rail at the moment, though updating a blog is hardly much of a test of tethering.)
I wish I wish that carriers would simply figure out a monetary equivalence between bandwidth and minutes, then just let me pay a transparent rate for what I use. Eventually I guess someone will drop the shenanigans and attract customers like me. As it is I’m miffed that I pay more per month for a calling plan I rarely use than I do for the data surcharge, which as far as I am concerned is the primary point of carrying around a location-aware pocket computer.
Telephone calls? Not my thing.
Later, I was looking at Google’s announcement regarding improved navigation UI. The improved transit overview is nice, but then I happened to request the bike route between work and home, and now that Google has caught on that the Bay Trail North of Moffett is open, it suggests that at the first choice, despite it taking ten minutes longer than more direct options. Anyway, it is nice to know now that my bicycle commute is 11.6 miles in 56 minutes. (I think it takes me a bit longer as I usually take a little break along the way.)
Saturday, November 20
Learned some basic git, and used it for updating the web site.
Installation to more-current-version of git:
sudo yum install gettext-devel expat-devel curl-devel zlib-devel openssl-devel
git clone git://git.kernel.org/pub/scm/git/git.git
cd git
make
make install
The big thing being it is trivial to create branches and switch among them in your working directory. So, you can start working on some feature, put it on the shelf, work on a different feature, and basically submit only the changes you feel are baked back to the main line.
I honestly believe that Google really does intend to not-be-evil. And though I know they screw up and have to deal with some grey areas, I put a lot of trust in Google with my personal data. Trust I wouldn’t put in Microsoft or Facebook.
Anyway, my faith in Google was recently re-affirmed when I fired up Google Chrome on a new box and was presented with this dialog:
If you’re not being evil, you make it trivial for users to switch to a different search engine. If you’re making an effort to really do right by the user, you ask them which search engine they prefer, rather than just defaulting them to your own.
