Big thanks to Mei for baking a cake and hosting a party and putting up with me most days. Gratitude as well to friends who came by and made it a nice party last night.
The Birthday Cake
I had to open the patio door to help clear the smoke from the room. Next year I’ll ask for a fire extinguisher for Christmas.
Last month I “cut and copied” the following letter printed in the Palo Alto Daily News. (Or I think its called the Daily Post now.) Now I shall paste, transcribe and share:
The text reads:
Bike changed a life
Dear Editor: A recent letter on “bikes vs. cars” stated that the over-50 crowd was “not about to go out and buy a bicycle” to replace their cars. Read on. Three years ago, I got in my car to go to an appointment and discovered that I had a dead battery. Frustrated (my wife had our other car) I slammed the car door only to notice right above me was my son’s old mountain bike hanging from the garage rafters.
I got it down — both tires were flat — pumped them up and rode off to my appointment.
Until that moment, I had not been on a bike in 40 years. After three or four blocks I wondered why it had taken me so long to get back on a bike. It was fantastic!
Several days later, I purchased my own bike on Craigslist and was soon riding to and from work — 15 miles round trip — taking the bus on days it was too cold or to dark. I’ve lost weight and never felt better.
After two months, my wife and I realized we could get by with one car, so we sold my car and used the money to put solar panels on our house. I now pay nothing for electricity. We’ve lowered our carbon footprint significantly. I’m 57 years old.
I was up ’til 1am working on some diagrams to help illustrate a book. At 5:30am the cat started jumping on us for breakfast. That’s not right, so I expressed my disapproval and locked her out of the bedroom.
But I had to hop out of bed and engage her a few more times with the spray bottle before she stopped trying to dig through the door.
Then she started crying. And I worried that maybe the neighbors might complain.
I have that uneasy feeling that I am forgetting something. I guess it may have something to do with the fact that after having resigned herself to my loyalty to my beat up old round-the-world college backpack, Mei had me empty it so she could take it in for repairs as a birthday present. Subtract that missing element from the new apartment I’m still unpacking in to after the holidays . . .
Or its that yesterday I spent some time at the hospital visiting a friend from older days, hanging out with his folks and keeping them company while their son, my age, drifted in and out of sleeping off the stroke he had on Friday. I remember the time spent in Colorado when it was Dad’s turn to shake off his own stroke.
And then there’s the Congressman shot clear through her left lobe. I listen attentively when they explain that, as with my friend and with my father, the left is where language is. One question is whether there is motor control in the right hand, since the hand is controlled next door from language.
I worry about my friend, but I know he will be okay, one way or another. One way he won’t be able to work, and may even need some personal assistance. Another way is that between his youth, spirit, and clean living, he will rehabilitate so well that years from now he will have difficulty convincing people who hadn’t seen it that he had once had a stroke.
Only time will tell. For now his folks are taking turns sleeping in the reclining chair next to his bed in the critical care. The son is there to rest and cooperate with the Doctors. The parents are there because there really is nowhere else in the world they can be right now. They attend to the details of managing their son’s life and care while he is down. I worry more about them, because I have some idea of where they are, and their needs can be better understood without a medical degree.
That may be it. I feel like I am missing something because instead of the hospital I am headed to the office. I would rather wait around at the hospital. Fortunately my friend and his family are inundated by visitors, and dropping by for a while in the evening after work, I won’t be in the way.
The man spends a few hours in the morning posting to his blog, and the rest of the day wandering the streets of New York City, his senses keen for prey, which he captures with the bravery of asking a stranger if he can take their photo.
How does he pay the rent? The video doesn’t get in to that. For me it is enough to see a guy has found his particular thing, that he’s in his element, and that he is this human archetype, the lone hunter wandering his territory in search of a prize.
Automation saves time both by getting tasks done more quickly and by ensuring consistency, thus reducing support calls.
Start with a script that outputs the commands that would do the task. The SA can review the commands for correctness, edit them for special cases, and then paste them to the command line. Writing such scripts is usually easier than automating the entire process and can be a stepping stone to further automation of the process.
A simple script that assists with the common case may be more valuable than a large system that automates every possible aspect of a task. Automate the 80 percent that is easy and save the special cases for the next version. Document which cases require manual handling, and what needs to be done.
There have been times in my career when I have felt that people look at automation as a one-off task. “Write a script to automate this task.” Other times I have been asked how I go about automating things, and my answer is that automation isn’t a task so much as an iterative process:
I try to do the task at least once, maybe a few times.
Along the way I document what I had to do to get the job done.
From there, I follow the documentation, and refine edge cases as I go.
After that I’ll write a script, and get it working. (do)
I revise the documentation to explain how to use the script. (document)
And then, I use the script to complete requests, fixing the script when it fails. (refine)
Often enough I have been called upon to help another group automate something. That is a little trickier because I may never get the chance to do the task. Hopefully the other group has written some documentation, otherwise I’ll have to tease it out of them. The whole refinement process is the most obviously collaborative. I’ll document “use the script . . . it should do this . . . if it does something else, send me details.”
There is also the question of what-is-worth-automating. I believe it is the “Practice of System and Network Administration” which breaks tasks in to four buckets: frequent-easy, frequent-difficult, infrequent-easy, infrequent-difficult. You get the most payoff by focusing your automation on the frequent tasks. Easier tasks are generally easier to automate, so go ahead and start there, then turn your focus on the frequent-yet-difficult tasks. If you regard automation as an iterative process, then infrequent tasks are that much harder to automate. This is doubly true when the task is sufficiently infrequent that the systems have a chance to evolve between task execution. Infrequent tasks tend to be adequately served by well-maintained documentation in lieu of an automated process.
A last note for infrequent tasks. Part of the difficulty for these can be a combination of remembering to do them, and finding the correct documentation. One approach to “automating” an infrequent task would be to write a script that files a request to complete the task. This request should of course include a pointer to the documentation. For example, I have a cron job which sends me an e-mail to complete a monthly off-site backup for my personal web site. The e-mail contains the list of commands I need to run. (And yes, the daily local database backups are executed automatically.)
Heck, let us jump upon the social media bandwagon. If you don’t “get” Twitter then I’d say that Twitter is pretty much what you make of it. And for me, that’s a distraction where I can pop in and see if anyone I follow has come up with anything entertaining to say, and I can share a thoughtlet of what is on my mind, and then as quick as it came, Twitter is gone and I’m back to the rest of my day.
The following are entertaining bits I have seen fit to “re-tweet” and share with others during 2010, and now I’ll share them with you.
I took Mei to Europe. We visited London, Paris, Lyon, Rome, and Venice. Then the volcano erupted in Iceland, so we visited Florence, and camped out at Lido, near Rome’s airport.
We also made it out to visited Dad and Gwen in Colorado, and Mom and Grandma visited us in Brooklyn.
Poland lost much of its executive branch in a plane crash, and BP began spilling oil into the gulf of Mexico.
May
Mei learned to ride a bicycle. I got to tour the New York’s abandoned “City Hall” subway station. We began fostering two older “rescue” kittens, Maxwell and Maggie, in an attempt to “socialize” them to living with people. Mei’s folks visited to attend her graduation from residency, and a week later I took her to Coney Island.
On May 19, a young man, Ronald Glover, was murdered around the corner from our apartment. BP continued spilling oil in to the Gulf of Mexico.
One weekend after brunching at Two Boots in Park Slope, Mei and I were walking through Prospect Park. I asked her to stop, got down on a knee and asked if she would marry me. With tears in her eyes, she accepted my proposal, and we kissed.
BP continued spilling oil in to the Gulf of Mexico, while we watched world football games on television.
Mei and I trekked to Hoboken, New Jersey, to watch the fireworks.
In Oakland, BART police officer Johannes Mehserle received a light sentence of manslaughter for his shooting death of Oscar Grant. Oakland, to its credit, failed to riot. Mid-way through the month, BP stopped spilling oil in to the Gulf of Mexico.
As Mei was finished with her residency, and I was still employed by a San Jose-based company, we prepared for our move back to Northern California.
ROAD TRIP! We drove all of our belongings in a Penske rent-a-truck from New York City to Mountain View, CA, stopping in Chicago and Pueblo, CO along the way.
So, how does working from home compare with working from the office? Working from home allows greater productivity, because you skip the commute and can just grind away for several hours with few interruptions. It can also get a bit lonely at times. At the office, I’m not as productive as I was at the home office, but I get more opportunities to collaborate with colleagues: sharing skills and refining ideas. I’d say that for technology, a 40-80% telecommute could be ideal, but I haven’t had the chance to experiment, as our first Mountain View apartment was a one-bedroom.
San Francisco won the World Series. Fans torched the city. I wish we would stop spending public money to subsidize professional sports.
I had a chance to attend the “LISA 2010” sysadmin conference . . . in San Jose. Met a lot of nice sysadmins.
For Thanksgiving, we visited Mei’s family in Hawaii. This was my first visit. Nice place! There was much feasting, and we selected a venue for the wedding, and set a date in 2011.
When we got home, we took receipt of a notice from the landlord giving us three days to pay or quit. The deadline had passed. I sent a letter requesting an explanation.
The landlord never answered my letter, but instead filed a civil suit of unlawful detainer against us. I talked to a bunch of people in Virginia to establish that they had made a billing error and undercharged our November rent, and they wanted me to pay the difference, plus a late fee, plus re-pay the December rent, plus their legal fees. I talked to some lawyers who indicated that we had a good case, so I compiled an answer, and am looking forward to the hearing.
However, the stress of worrying over an eviction proceeding over the holidays was a bit much, so we took the opportunity to seek out and move to a bigger apartment in a nicer complex. Since nobody wants to move the week before Christmas, they gave us the first month’s rent free.
Mei was notified that she had passed the medical Board Exam for which she had been studying since finishing her residency. To say that she was elated would be an understatement.
Congress repealed “Don’t Ask Don’t Tell” and there was much rejoicing.
We made it home to Chicago for the holidays. There was much eating and visiting family and friends.
Several leading institutions fail, leading consumers to a rush on the remaining institutions, causing them to fail. A cascade of failures brings the whole system crashing down until the central authority undertakes a massive, unprecedented intervention to bring the system back to normal. At first, the degree of central intervention required is underestimated, but in time sufficient resources are brought to bear and the complex system recovers.
Interestingly, Skype’s network is actually a peer-to-peer network. It is a complex system which normally proves highly resilient, with in-built safety mechanisms to contain failure and ensure reliability. But under the right circumstances, failure can cascade. I couldn’t help but read that as a metaphor for free-market economics, which can usually take care of itself, but will enter a fugue state often enough to require a strong authority to intervene and put it right.
As a SysAdmin, the Skype network sounds like a very interesting beast. I figure that an action item against a future failure might be to provide a “central reserve bank” that monitors the health of so-called supernodes and automatically fires up large numbers of the dedicated mega-supernodes in the event of a widespread failure. (And such a strategy could well exacerbate some other unanticipated failure mode.)
Good news! Southwest Airlines offers wifi on my flight! Only $5 introductory price! I have to try this out!
The service is “designed by Yahoo!”
It is kind of really really slow to make connections.
Wait . . . WTF is this?!!
Yup. Southwest Airlines wifi does HTTP session hijacking to inject content in to your web pages.
This is a perfect illustration of the need for net neutrality: your Internet Service Provider should not interfere with your ability to surf web pages. This would be comparable to your phone company interrupting your telephone calls with commercials. Outrageous! Wrong! Bad!!
(On Mei’s computer there are actual ads in the blue bar on top, but my AdBlock plugin filters those.)
It gets worse from there. On the “designed by Yahoo!” experience you can surf on over to Yahoo! just fine. But I’m a Google man. Here’s what Google looks like:
Work-around #1: On sites that support them, use HTTPS URLs. Those are encrypted, so they can’t be hijacked. So, where http://www.google.com/ fails, https://www.google.com/ gets through!
But my little WordPress blog lacks fancy-pants HTTPS. And the session hijacking breaks my ability to post.
Work-around #2: If you have a remote shell account, a simple ssh -D 8080 will set up a SOCKS proxy, and you can tell your web browser to use SOCKS proxy localhost:8080 . . . now you are routing through an encrypted connection: no hijacking!
Update: they charge is $5/segment, so $10 if your plane stops in Las Vegas, and you get to type your credit card number a second time. Though, on the second segment, Google loads okay, but I still had to route through the proxy because the magic header was blocking WordPress’ media interface.
The Gogo Wireless on Virgin America always worked way better than this, and Google covers the cost over the holidays. And as far as I know: no session hijacking!
I saw this float across my Google Reader yesterday, thanks to Tom Limoncelli. If you are a sysadmin in an environment fixing to do more virtualization, it is well worth a skim:
NOTE: this isn’t an anti-virtualization rant, more of a “things to watch out for” briefing.
Some of my take-aways:
Sure we’ll have fewer physical servers, but the number of deployed systems will grow more quickly. As that grows so too will our systems management burden.
As the system count grows faster, we may hit capacity chokepoints on internal infrastructure like monitoring, trending, log analysis, DHCP or DNS faster than previously assumed.
Troubleshooting becomes more complex: is your slow disk access an application, OS, or hardware issue becomes also a potential host OS, networking/SAN or filer issue as well.
Regarding troubleshooting: we may add another team to the mix (to manage virtualization) while trouble-shooting has an increased probability of requiring cooperation across multiple teams to pin down. Increased importance on our ability to cooperate across teams.
Change management impacts: small changes against a larger number of systems sharing architecture snowball even more. One can add something to the base image that increases disk use by 1% for any one system, but multiply that across all your systems and you have a big new load on your filer. (1,000 butterflies flapping their wings.)
Reduced fault isolation: as we have greater ability to inadvertently magnify increased load and swamp network and storage infrastructure, we have a greater ability to impact the performance of unrelated systems which share that infrastructure.
The article also cautions against relying on vendor-provided GUIs because they don’t scale as well as a good management and automation framework.
Ah, the other thing noteworthy there is the ACM Queue magazine is now including articles on systems administration. (I subscribed to the system administration feed.)
A wiki feature I would like to see: when a node has not been edited for a certain period of time, it sends a message to the author and contributors, to review the node for relevance, and update or delete the node, if appropriate. The period of time should be configurable, and the feature should be able to CC the message in to the local request management system.
The check should be repeated on multiples of the period of time. For example, if you want to review your nodes once per year, the first year comes, the author reviews the node, sees no need to make any changes, then another year passes, you get a reminder to take another looksee . . .
This would be useful especially for an operations environment, to ensure that the shared knowledge hasn’t gone too stale, keeping the wiki resource relevant. I sincerely hope this feature already exists in a few systems!
Also useful for tracking down dependencies: I was reminded that for FreeBSD, what I would wish to call pkg_which is pkg_info -W, which will tell you what package a file came from.
Potentially nifty: a text-to-speech utility that writes its output to an audio device, which you could set as your system’s sound input. This way you could have “conversations” in your headphones via VoIP or Skype without having to make disruptive noises or emitting sensitive information in a shared environment like an open office or a library.
Hrmmm.
sudo apt-get install epos
sudo /etc/init.d/epos start
say "this does not work"
sudo apt-get remove epos
sudo apt-get install espeak
espeak "hello there"
But it appears there is no good way on Linux, anyway, to tell a command to dump its audio output to the microphone. Bah.
Wednesday, November 3
I got my sutures out the other day. I’ve still got the band-aid on my chin, cleaning and re-dressing twice a day, until the skin is no longer broken. Doc said I’d be shaving regular next week.
Thursday, November 4
Nice: I got the WordPress for Android App working thanks to Dan at Automattic pointing out I had an SVN-corrupted xmlrpc.php file.
Annoying: No idea how I can possibly type < or > with this nice-but-crippled G2 keyboard.
I made it up to San Bruno today, aboard Caltrain. Today is the first time I took my bike on Caltrain and it really wasn’t as bad as a colleague made it sound. The yellow tags are impossible to find though, so I made a couple labels to stick on my bike indicating what stations I ride between. In San Bruno, I stopped at the curb where I face-planted last week. Nothing special about it. I got to the office way faster than I have by walking. Probably 20 minutes compressed to 5 or not more than 10. The only unpleasant part is crossing El Camino. The crossing I chose had a crosswalk on only one side, so I had to cross against traffic to get rolling across the street.
Friday, November 5
Well, I finally have a new watch band . . . my old one broke such that it would no longer attach to my wrist, but I could dangle it off my belt. Now I have to get used to looking at my wrist again.
Monday, November 8
Some days I just feel blue. Like a dark cloud is over my heart. Not unlike overcast weather. Fortunately I often have the self-awareness to understand that, whether or not I know its origin, this is likely just a passing cloud, and the best way to weather it is to just take life on as a normal day, perhaps with an added dash of industriousness to stave off the natural slothitude that a funk brings on.
I was just reading in The New Yorker about different ways that salaries are determined. I identified with having accepted the “authority ranking” or feudal model earlier in my career, when what I valued most was the opportunity to work, to serve and build and learn. Back when I still kind of hoped that wages were fair, that bosses valued the contribution of their employees. Back in 1999 this even seemed true: the startup I was working for noticed that I was kicking butt, and ramped my salary up by 20% after my first six months on the job, and again after a year . . . everything was noble and virtuous.
But nobility and virtue don’t often last. Recessions hit, money dries up, eager young employees burn themselves out and haven’t a clue what to do about it. Layoffs come . . . a decade later I have left for greener pastures as many times as I have been layed off. Welcome to the “market pricing” model of economic interaction.
I find some difficulty feeling passionate about being a market priced, employment-at-will agent. Just as sex feels better with someone you love than when you’re getting paid for it, so too does work. Fortunately, the current gig offers competetive compensation, likely stability, and the chance to relate to coworkers over several years. I guess that is as close to “married” one can get in our industry . . . and yet of course I keep my eye out for new opportunities.
And I frequently worry about getting too comfortable, getting a little flabby in the skill set because there is plenty of work I would have to do at a startup that is off-my-plate at a large company. Better attain depth where I needn’t worry over breadth, eh?
Thursday, November 11
I had a meeting today that I haven’t been looking forward to. Basically, I have spent most of my career at smaller companies, where I tend to have a lot more say over how we do things, and where the simple, obvious, light-weight solution will tend to carry the day. But at a large company, there are enough competing interests that the way we do things is often not up to me, and is far more complex and open to error (in my view) than how I would go about doing it.
Note an editorial bias, right? Of course I have a high opinion of my own way of solving a problem. Doesn’t mean my approach is the right one . . .
So, at the end of this meeting, I got McCoy in my head. “I’m just a simple country Doctor.” Well, I’m just a small-shop sysadmin, serving on a corporate flagship. I don’t really understand or approve of everything that is going on, but that isn’t my problem. I focus on getting my own job done and I am happy to give Captain Kirk a piece of my mind, but at the end of the day the Captain gets what the Captain asked for.
Wednesday, November 17
From today’s work log:
Scout around a bit as to the advantages of managing system configuration
files in git. Git’s strength appears to be strong branch and merge
capabilities, working offline from the central repository, and the
capacity for fine-grained commits. Disadvantage is a steeper learning
curve. Anyway, we could potentially allow staff to grab a local branch
make several changes, review changes and reject those that proved
infeasible, then submit changes back to the central repository. Later,
a change management team could review differences between the central
repository and the stage / production repositories, then selectively
merge changes to the more stable environments in an appropriate manner.
I had lunch with Mei today at a Chinese place on Castro St in Mountain View. At the next table I overheard some guys talking about the size of the Oort Cloud if Earth were the size of a grain of sand . . . hard-core nerdy lunch conversation. I recounted that when I got off the light rail the other day I heard one guy explaining to another guy the theory behind anti-matter reactions that power the warp propulsion system in Star Trek. When you live here you live in the pulsing underbelly heart of nerd-dom. I kind of like it.
Thursday, November 18
My bicycle lights came in from Amazon.com and I tried them out yesterday on the ride to the light rail. (Between the weather and a recent injury I’m usually reluctant to ride all the way to work.) The front light was somewhat occluded by the basket so last night I moved it to a helmet mount, which required some careful trimming of a cross-member atop the helmet so the thing would fit, but nothing likely to compromise structural integrity.
Yesterday I also received my replacement G2. I got it up and running and it went and updated itself. It made a big todo about “wifi calling” which . . . uses minutes? Really? REALLY?! So, I’ll provide the bandwidth and you’ll charge me . . . but it also quietly enabled Tethering, via USB and WiFi. I’m using it now. I heard a rumor that T-Mobile was fixing to charge an extra fee for tethering. Hopefully though at the moment they’re content to charge customers to make telephone calls via their home wifi networks.
(Oh yeah, and I’m on the light rail at the moment, though updating a blog is hardly much of a test of tethering.)
I wish I wish that carriers would simply figure out a monetary equivalence between bandwidth and minutes, then just let me pay a transparent rate for what I use. Eventually I guess someone will drop the shenanigans and attract customers like me. As it is I’m miffed that I pay more per month for a calling plan I rarely use than I do for the data surcharge, which as far as I am concerned is the primary point of carrying around a location-aware pocket computer.
Telephone calls? Not my thing.
Later, I was looking at Google’s announcement regarding improved navigation UI. The improved transit overview is nice, but then I happened to request the bike route between work and home, and now that Google has caught on that the Bay Trail North of Moffett is open, it suggests that at the first choice, despite it taking ten minutes longer than more direct options. Anyway, it is nice to know now that my bicycle commute is 11.6 miles in 56 minutes. (I think it takes me a bit longer as I usually take a little break along the way.)
Saturday, November 20
Learned some basic git, and used it for updating the web site.
Installation to more-current-version of git:
sudo yum install gettext-devel expat-devel curl-devel zlib-devel openssl-devel
git clone git://git.kernel.org/pub/scm/git/git.git
cd git
make
make install
The big thing being it is trivial to create branches and switch among them in your working directory. So, you can start working on some feature, put it on the shelf, work on a different feature, and basically submit only the changes you feel are baked back to the main line.
I honestly believe that Google really does intend to not-be-evil. And though I know they screw up and have to deal with some grey areas, I put a lot of trust in Google with my personal data. Trust I wouldn’t put in Microsoft or Facebook.
Anyway, my faith in Google was recently re-affirmed when I fired up Google Chrome on a new box and was presented with this dialog:
If you’re not being evil, you make it trivial for users to switch to a different search engine. If you’re making an effort to really do right by the user, you ask them which search engine they prefer, rather than just defaulting them to your own.