There seems to be some backlash going on against the religion of “Agile Software Development” and it is best summarized by PragDave, reminding us that the “Agile Manifesto” first places “Individuals and interactions over processes and tools” — there are now a lot of Agile Processes and Tools which you can buy in to . . .
He then summarizes how to work agilely:
What to do:
Find out where you are
Take a small step towards your goal
Adjust your understanding based on what you learned
Repeat
How to do it:
When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier.
Sounds like sensible advice. I think I’ll print that out and tape it on my display to help me keep focused.
Tommy can not get enough of this song, so I picked this out on the banjo. My first attempt at building my own tab:
-----0-0-----0-0--|-2-------2-------|---0-0-----0-0---|------------------
---0-------0------|-----3-------3---|-----------------|-----0--------0---
-0-------0--------|-----------------|-----------------|-2-2---0--2-2---0-
------------------|-----------------|-----------------|------------------
------------------|---0-------0-----|-0-------0-------|------------------
G B D D G B D D E G D E G D G D D G D D A A B G A A B G
As a total total total newbie, the wife helped me pick the notes out on the piano. I then shifted them left 3: C –> G, E –> B, because that is what was sounding right on the banjo. And I had picked out my own notes chart using the “Fine Chromatic Tuner” app:
0 1 2 3 4 5 6 (fingers on frets..)
D D# E F F# G
B C C# D D#
G G# A A# B C C#
D D# E F F# G G#
G -----------------
The revelation was when I “discovered” (I’m sure I have been told but it hadn’t been relevant at the time) that the fifth “G” string matches first string, fifth fret. Before that I was picking 2, then 5 on the D string.
With any luck I can pick around on this and mush it into some roll somehow. It would be nice if the second bit mapped to a chord . . .
------0-------0----|-2-------2-------|------0-------0----|-----------------|
----0---3---0---3--|-----3-------3---|----0-------0------|---------0-------|
--0-------0--------|-----------------|-------------------|-2---2-------0---|
-------------------|-----------------|-------------------|-----------------|
-------------------|---0-------0-----|--0-------0--------|-----------------|
G B D D G B D D E G D E G D G B D G B D A A B G
I have been thinking about whether and where there are “drone notes” to work a roll around. I started with cleaning up the timing. (My knowledge of this song consists mainly of YouTube videos from India.) The bars go:
father finger x2
where are you? x2
here i am! x2
how do you do? x1
Each note here corresponds to a syllable, with quarter-note spacing on the last phrase. At least at the speed I play at this time, there is no roll to be added.
I did make two changes:
1) The fourth and eighth Ds in the first phrase moved from fifth string to fourth. Now the middle finger needs to be ready on that third fret a bit earlier, but we get out of plucking the fifth string twice in a row.
2) To avoid double-plucking in the third phrase, I went from G D D to G B D. They sound equally good to me, and to get G D D you can just keep your finger on the third fret.
I have a Nagios check to monitor the power being drawn through our PDUs. While setting all this up, I had to figure out alert threshholds. The convention for disk capacity is warn at 80%, critical at 90%. I also had this gem to work with:
National Electric Code requires that the continuous current drawn from a branch circuit not exceed 80% of the circuit’s maximum rating. “Continuous current” is any load sustained continuously for at least 3 hours.
Lately I have been getting a lot of warning notifications about circuits exceeding 80%. Ah, but the NEC says that is only a problem if they are at 80% for more than three hours. So, I dig through Nagios documentation and split my check out into two services:
define service{ # PDU load at 90% of circuit rating
use generic-service
hostgroup_name pdus
service_description Power Load Critical
notification_options c,u,r
check_command check_sentry
contact_groups admins
}
define service{ # PDU sustained load at 80% of circuit rating for 3 hours
use generic-service
hostgroup_name pdus
service_description Power Load High
notification_options w,r
first_notification_delay 180
check_command check_sentry
contact_groups admins
}
The first part limits regular notifications to critical alerts. In the second case, the first_notification_delay should cover the “don’t bug me unless it has been happening for three hours” caveat and I set that service to only notify on warnings and recovery.
I have some racks in a data center that were designed to use 3-phase PDUs. This allows for greater density, since a 3-phase circuit delivers more watts. The way three-phase works is that the PDU breaks the circuit into three branches, and each branch is split across two of the legs. Something like:
Branch A: leg X->Y
Branch B: leg Y->Z
Branch C: leg Z->X
The load needs to be balanced as evenly as possible across the three branches. When the PDU tells me the power draw on Branch A is say, 3A, I am really sure if that is because of equipment on Branch A or … Branch C? This only becomes a problem when one branch reports chronic overloading and I want to balance the loads.
I chatted with a friend who has a PhD in this stuff. He said that if my servers don’t all have similar characteristics, then the math says I want to randomly distribute the load. That is hard to do on an existing rack.
I came up with an alternative approach. Most of my servers fall into clusters of similar hardware and what I would call a “load profile.” I had a rack powered by two unbalanced 3-phase circuits. I counted through the rack’s server inventory and classified each host into a series of cohorts based on their hardware and load profile. Two three-phase circuits gives me six branches to work with, so I divided the total by six to come up with a “branch quota.” Something like:
<= 2 hadoop node, hardware type A
<= 2 hadoop node, hardware type B
1 spark node, hardware type A
1 spark node, hardware type C
1 misc node, hardware type C
I then sat down with a pad and paper, one circuit on either side of the sheet, and wrote down what servers I had, and where the circuit was relative to quota. So, one circuit might have started off as:
I then moved servers around, updating each branch circuit with pencil and eraser as I went, like a diligent Dungeon Master. (And also, very important: the PDU receptacle configuration.)
The end result:
I started moving servers around Thu 15:00, and was done about 16:30, which is also when the hadoop cluster went idle. It kicked back up again at 17:00, and started spinning down around midnight.
What is important is to keep sustained load on the branches under the straight green line, which represents 80% of circuit capacity. You can see that on the left, especially the second circuit had two branches running “hot” and after the re-balancing the branch loads fly closer together, and top off at the green line.
I have been working with AWS to automate disaster recovery. Sync data up to S3 buckets (or, sometimes, EBS) and then write Ansible scripts to deploy a bunch of EC2 instances, restore the data, configure the systems.
Restoring data from Glacier is kind of a pain to automate. You have to iterate over the items in a bucket and issue restore requests for each item. But it gets more exciting than that on the billing end: Glacier restores can be crazy expensive!
2) Amazon Glacier will also charge you money if you delete data that hasn’t been in there for at least three months. If you Glacier something, you will pay to store it for at least three months. So, Glacier your archive data, but for something like a rolling backup, no Glacier.
3) When you get a $,$$$ bill one month because you were naive, file a support request and they can get you some money refunded.
I inherited a bunch of ProxMox. It is a rather nice, freemium (nagware) front-end to virtualization in Linux. One of my frustrations is that the local NAS is pretty weak, so we mostly run VMs on local disk. That compounds with another frustration where ProxMox doesn’t let you build local RAID on the VM hosts. That is especially sad because it is based on Debian and at least with Ubuntu, building software RAID at boot is really easy. If only I could easily manage my VMs on Ubuntu . . .
And then, on your local Ubuntu workstation: (you are a SysAdmin, right?)
sudo apt-get install virt-manager
Then, upon running virt-manager, you can connect to the remote host(s) via SSH, and, whee! Full console access! So far the only kink I have had to iron is that for guest PXE boot you need to switch Source device to vtap. The system also supports live migration but that looks like it depends on a shared network filesystem. More to explore.
ServerA> virsh -c qemu:///system list
Id Name State
----------------------------------------------------
5 testvm0 running
ServerA> virsh -c qemu+ssh://ServerB list # Test connection to ServerB
Id Name State
----------------------------------------------------
The real trick is to work around a bug whereby the target disk needs to be created. So, on ServerB, I created an empty 8G disk image:
This will take some time. I used virt-manager to attach to testvm0’s console and ran a ping test. At the end of the procedure, virt-manager lost console. I reconnected via ServerB and found that the VM hadn’t missed a ping.
And now:
ServerA> virsh -c qemu+ssh://ServerB/system list
Id Name State
----------------------------------------------------
4 testvm0 running
Admittedly, the need to manually create the target disk is a little more janky than I’d like, but this is definitely a nice proof of concept and another nail in the coffin of NAS being a hard dependency for VM infrastructure. Any day you can kiss expensive, proprietary SPOF dependencies goodbye is a good day.
Early in my career, I didn’t interact much with management. For the past decade, the companies I have worked at had regular one-on-one meetings with my immediate manager. At the end of my tenure at Cisco, thanks to a growing rapport and adjacent cubicles, I communicated with my manager several times a day, on all manner of topics.
One of the nagging questions I’ve never really asked myself is: what is the point of a one-on-one? I never really looked at it beyond being a thing managers are told to do, a minor tax on my time. At Cisco, I found value in harvesting bits of gossip as to what was going in the levels of management between me and the CEO.
Ben Horowitz has a good piece on his blog. In his view, the one-on-one is an important end point of an effective communication architecture within the company. The employee should drive the agenda, perhaps to the point of providing a written agenda ahead of time. “This is what is on my mind,” giving management an opportunity to listen, refine strategies, clarify expectations, un-block, and provide insight up the management chain. He suggests some questions to help get introverted employees talking.
I am not a manager, but as an employee, the take-away is the need to conjure an agenda: what is working? What is not working? How can we make not merely the technology, but the way we work as a team and a company, more effective?
It was about twenty years ago, I was in college, up late in the computer lab writing an email to President Clinton asking him not to sign the “Defense of Marriage Act” into law. Today, I am proud of my country, and the speed with which we have “evolved” to better recognize more of the civil rights of our people.
Thank you, Justice Kennedy, and to the countless advocates who have helped us all open our eyes.
Life has been busy lately. I have failed at carving out time for the little things like keeping up with email and reading and writing. One theme that is just below the surface these days, is an understanding of the Individual’s impermanence, that one will inevitably be swept away down the river. But, the good news is, it is the river that is the thing. You stick your foot in the river, and you feel the tug of the currents: this one fast and warm, that one slow and cool. In life, we are these currents, flowing together, mingling, becoming something identifiable and satisfying while also becoming the river itself.
Death has been on my mind lately. Dad passed about a year back, and the Reaper has expressed an interest in the health of another loved one. I am not opposed to Death. We’re all going to get there. Life, the abused cliché reminds us, is the journey and not the destination. I’ll be forty in January. One can read that as the half way mark. I want to pull over and look around. Close at hand, I see my toddler Son, his eyes wide with the possibilities and joys of life, his future for him to know and hopefully to share with his old man. And, not far off, I see my Father, whom my Son will ever know through stories, mainly told by me. Stories I mainly lack. And, yonder still, my own Grandfather, whom I know mainly through the most exaggerated of stories.
We all come from somewhere, and we are all headed somewhere. This bend in the River knows only a short ways upstream, toward the various and contradictory legends of the Wellsprings, and only a short ways ahead, toward the various legends of the Delta, where we believe the River as we have ever understood it will cease as it merges with the Great Ocean.
The August issue of The Sun Magazine brought with it an interview with Stephen Jenkinson, whom some call “The Death Whisperer” … he packs a lot of great ideas that resonate with me into eight pages. Not bad. What follows is a bit of perspective on the idea of one’s influences.
Hoffner: Who would you say are your influences?
Jenkinson: Anyone who claims to know his or her influences probably doesn’t. I think our influences are a lot subtler than we think. For example, I was born nine years after the closing of Auschwitz and the bombing of Hiroshima half a world away. When those soldiers came home from World War II suffering from post-traumatic stress disorder before we had a name for it, North America created the suburbs for them. I grew up in the unacknowledged presence of those wartime horrors: Auschwitz on the one side, Hiroshima on the other, and the suburbs in the middle. That’s an influence on me.
As a child I was read to every night. Before I even understood the words, I was carried along by the momentum of the human voice. The pageant of the story has its way with you, even if its not in a language you can comprehend. Story is a sublime practice that makes us recognizable to ourselves.
These days I admire the songwriter Leonard Cohen, my countryman and a polestar in the firmament for anyone who has faith in human eloquence. Eloquence is a conjuring; it’s magic, and Cohen is a servant as well as a practitioner and a repository of that magic. He’s a patron saint of the Orphan Wisdom School, unawares. I don’t know what kind of life he lives, but it’s inconceivable to me that those songs might come from a duplicitous nature. In a country that appreciated its artists, he would be a national treasure and wouldn’t have to work five minutes in his life unless he was so inclined. As it is he’s been on the road for years trying to make back all the money his manager stole from him.
I met another of my influences at Harvard. As a young man I was on fire with learning about the historical Jesus. I didn’t come from a religious background, but I applied to Harvard Divinity School and got in. I was determined to be a preacher of some sort. I don’t know what else you could do with that kind of education. At the divinity school I met a fellow who was the living incarnation of a stereotypical televangelist: power-blue suit that didn’t fit so good: too-tight white shirt that was popping its buttons. He was in charge that year of vocational counsel. I told him I planned to get a master of divinity and become a pastor or a minister. He asked me the name of my sponsoring congregation, and I said I hadn’t worked that out yet. Then he asked my denominational affiliation. I told him I didn’t have one. “Son, where do you go to church?” he asked. I said that I didn’t, and he asked, “Well, where did you go to church, then?” No answer. So he said, “Let me understand this: you propose to go into the ministry, and you’ve never been to church?” “Yes sir,” I replied. “Well, I nev-uh,” he said, just like that. I was three questions into my vocational interview, and I was done.
My career as a preacher came to an end at that moment. I was counseled to register for a master of theological studies — a layperson’s degree — instead. That same week I met a preaching instructor whose name was Hugh Morgan Hill, but everyone knew him as Brother Blue. He was a vibrant speaker in the African American tradition. He said I should be in his class. I told him I’d already been counseled out of the master of divinity program. “Nobody needs to know,” he said. On my way to his first class I picked up a harmonica. The class had already begun when I got there, and Professor Hill was in full flight. He was a great storyteller and performer. For some reason I started to play my harmonica along with what he was doing, just improvising.
The next week his wife phoned and asked if I’d come with him to a church service — and bring my harmonica.
I performed with Hill on and off for seven years. It was an unofficial apprenticeship. We traveled all over the U.S. and Canada. This was the era of school integration, remember, and there were race riots in some cities, but since we were together, we were OK. He was a holy man from the ghettos of the American heartland. Virtually everything he did in the world was self-initiated. He never seemed to have a job description. He carved it out every time he stood up to speak. I learned from him the importance of proceeding without the green light, the red carpet, the Get Out of Jail Free card. I was emboldened by his example when I was working in palliative care, because I realized that if I was going to serve these dying people well, then I couldn’t wait for anyone to ask me to do it. And if I’m going to serve the era I’ve been born into well, then I can’t wait for approval and recognition. I’m going to have to proceed without it. If it comes, it comes; if it doesn’t, it doesn’t. That, and a lot more, is what I got from him.
Brother Stephen articulates a lot of ideas about Life and Death. Ideas I am still digesting. But I suppose I can share some notes.
To be Alive is to be In Debt to Death. Everything we have, everything … the food we eat day after day, the clothes we wear, the fuel we use to get around … animals and plants, and if you think about it, the Solar energy that fuels All Of It comes from the slow annihilation of the Sun, as its atoms fuse into ever heavier elements. “What will your death feed?”
The Debt is non-negotiable and it will be re-paid.
“Grief is not sadness. There’s sadness in grief, but grief is not exhausted when the sadness goes away. … Sadness has a shelf life, but grief endures …” I picture a small pot, held over the flame of death. The sadness bubbles and splatters and evaporates. The pot is withdrawn from the flame. There is a residue left over. That residue is grief. It does not boil away. You paint with it. You leave a mark somewhere so that when one needs a reminder that “this too shall pass” they may thus be reminded. This paragraph is painted with grief. I hope it feeds your wisdom.
In the interview, Jenkins has a riff about the need for an element missing in our culture: the Rite of Passage in which childhood ends and adulthood begins. A consumer culture derives better profits from a population that is not asked to Grow Up: You Deserve More and More!! His critique resonates but I disagree with the idea that a Rite must “kill off childhood” … on the very next page he explains that it is misguided to shelter children from the idea of death … I think that Childhood is maybe what lends Death its greatest contrast. Young and full of possibility and very self-involved … adults should not be so self-involved, the grown ups must labor to pay the interest on our life debt … but we still need to grow and learn. While we would prefer for grownups to not be self-involved narcissists we need also ask them to be sufficiently self-involved, and other involved, to cultivate their self awareness.
Other involved is what we ask of a child, and what we give a child, when we share with them the fact of Death. Children can appreciate Wisdoms, just as Adults can appreciate Wonders.
A last thought, while I sat in my favorite coffee shop, taking and making these notes, and watching an Old Man drink in the Joyful Clown Antics of some toddlers across the way, was that nursing homes really ought to co-locate with nurseries. Children bring Life into the Room. Elders bring Life as well. The children are starting from scratch, painting with incoherent vitality. The elders have taken their lives, chipped away at it, and produced works of art, called Lives. Day by day, they reveal these works to the children: some are beautiful, some are perplexing, some are sad, and some are horrible. The children react, embrace, reject, imitate, and iterate. Culture ensues, and the river flows enriched.
I have been loath to embrace containers, especially since I attended a conference that was supposed to be about DevOps but was 90% about all the various projects around Docker and the like. I worked enough with Jails in the past two decades to feel exasperation at the fervent religious belief of the advantages of reinventing an old wheel.
I attended a presentation about Kubernetes yesterday. Kubernetes is an orchestration tool for containers that sounds like a skin condition, but I try to keep an open mind. “Watch how fast I can re-allocate and scale my compute resources!” Well, I can do that more slowly but conveniently enough with my VM and config management tools . . .
There was an undercurrent there that Kubernetes is the Great New Religion that Will Unify All the Things. I used to embrace ideas like that, then I got really turned off by thinking like that, and now I know enough to see through the True Beliefs. I could deploy Kubernetes as an offering of my IT “Service Catalog” as a complimentary option versus the bare metal, hadoopclusters, VM, and otherservices I have to offer. It is not a Winner Take All play, but an option that could improve productivity for some of our application deployment needs.
At the end of the day, as an IT Guy, I need to be a good aggregator, offering my users a range of solutions and helping them adopt more useful tools for their needs. My metrics for success are whether or not my solutions work for my users, whether they further the mission of my enterprise, and whether they are cost-effective, in terms of time and money.
$ for s in a b c d; do echo ; echo sd${s} ; sudo dd if=/dev/sd${s} of=/dev/null; done
sda
3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB) copied, 17940.3 s, 112 MB/s
sdb
3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB) copied, 15457.9 s, 129 MB/s
sdc
3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB) copied, 15119.4 s, 132 MB/s
sdd
3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB) copied, 16689.7 s, 120 MB/s
The back story is that I had a system with two bad disks, which seems a little weird. I replaced the disks and I am trying to kick some tires before I put the system back into service. The loop above says “read each disk in turn in its entirety.” Prior to replacing the disks, a loop like the above would cause the bad disks, sdb, and sdc, to abort read before completing the process.
The disks in this system are 2TB 7200RPM SATA drives. sda and sdd are Western Digital, while sdb and sdc are HGST. This is in no way intended as a benchmark, but what I appreciate is the consistent pattern across the disks: throughput starts high and then gradually drops over time. What is going on? Well, these are platters of magnetic discs spinning at a constant speed. When you read a track on the outer part of the platter, you get more data than when you read from closer to the center.
I appreciate the clean visual illustration of this principle. On the compute cluster, I have noticed that we are more likely to hit performance issues when storage capacity gets tight. I had some old knowledge from FreeBSD that at a certain threshold, the OS optimizes disk writes for storage versus speed. I don’t know if Linux / ext4 operates that way. It is reassuring to understand that, due to the physical properties, traditional hard drives slow down as they fill up.
On Christmas Eve Santa Claus wanted to upgrade my wife from a Nexus 5 to a Nexus 5X, only to find the SIM card was too big! Google brought him to a helpful web site, and that jolly old elf took scissors and a file and trimmed that card down freehand like some kind of old school Unix SysAdmin.
