JIRA, Religion, Technical, Technology

HOWTO: Develop Software Agilely

Link: https://dannyman.toldme.com/2015/03/06/howto-program-agile/

There seems to be some backlash going on against the religion of “Agile Software Development” and it is best summarized by PragDave, reminding us that the “Agile Manifesto” first places “Individuals and interactions over processes and tools” — there are now a lot of Agile Processes and Tools which you can buy in to . . .

He then summarizes how to work agilely:

What to do:

Find out where you are
Take a small step towards your goal
Adjust your understanding based on what you learned

How to do it:

When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier.

Sounds like sensible advice. I think I’ll print that out and tape it on my display to help me keep focused.

Feedback Welcome


“Father Finger” Banjo Tab

Link: https://dannyman.toldme.com/2015/03/11/father-finger-banjo-tab/

Tommy can not get enough of this song, so I picked this out on the banjo. My first attempt at building my own tab:

 G B D D G B D D    E G D   E G D     G D D   G D D     A A B G  A A B G

As a total total total newbie, the wife helped me pick the notes out on the piano. I then shifted them left 3: C –> G, E –> B, because that is what was sounding right on the banjo. And I had picked out my own notes chart using the “Fine Chromatic Tuner” app:

0  1  2  3  4  5  6   (fingers on frets..)
D  D# E  F  F# G
B  C  C# D  D#
G  G# A  A# B  C  C#
D  D# E  F  F# G  G#
G  -----------------

The revelation was when I “discovered” (I’m sure I have been told but it hadn’t been relevant at the time) that the fifth “G” string matches first string, fifth fret. Before that I was picking 2, then 5 on the D string.

With any luck I can pick around on this and mush it into some roll somehow. It would be nice if the second bit mapped to a chord . . .

Good times!

Feedback Welcome


“Father Finger” Banjo Tab v2

Link: https://dannyman.toldme.com/2015/03/23/father-finger-banjo-tab-v2/

  G B D D G B D D    E G D   E G D      G B D   G B D      A   A   B   G

I have been thinking about whether and where there are “drone notes” to work a roll around. I started with cleaning up the timing. (My knowledge of this song consists mainly of YouTube videos from India.) The bars go:

father finger x2
where are you? x2
here i am! x2
how do you do? x1

Each note here corresponds to a syllable, with quarter-note spacing on the last phrase. At least at the speed I play at this time, there is no roll to be added.

I did make two changes:
1) The fourth and eighth Ds in the first phrase moved from fifth string to fourth. Now the middle finger needs to be ready on that third fret a bit earlier, but we get out of plucking the fifth string twice in a row.
2) To avoid double-plucking in the third phrase, I went from G D D to G B D. They sound equally good to me, and to get G D D you can just keep your finger on the third fret.

Feedback Welcome


Nagios: Delay WARNING Notifications

Link: https://dannyman.toldme.com/2015/03/23/nagios-delay-warning-notifications/

I have a Nagios check to monitor the power being drawn through our PDUs. While setting all this up, I had to figure out alert threshholds. The convention for disk capacity is warn at 80%, critical at 90%. I also had this gem to work with:

National Electric Code requires that the continuous current drawn from a branch circuit not exceed 80% of the circuit’s maximum rating. “Continuous current” is any load sustained continuously for at least 3 hours.

(Thanks to mike Pennington, via http://serverfault.com/a/413307/72839)

So, I went with 80% warning, 90% critical.

Lately I have been getting a lot of warning notifications about circuits exceeding 80%. Ah, but the NEC says that is only a problem if they are at 80% for more than three hours. So, I dig through Nagios documentation and split my check out into two services:

define service{ # PDU load at 90% of circuit rating
    use                     generic-service
    hostgroup_name          pdus
    service_description     Power Load Critical
    notification_options    c,u,r
    check_command           check_sentry
    contact_groups          admins

define service{ # PDU sustained load at 80% of circuit rating for 3 hours
    use                       generic-service
    hostgroup_name            pdus
    service_description       Power Load High
    notification_options      w,r
    first_notification_delay  180
    check_command             check_sentry
    contact_groups            admins

The first part limits regular notifications to critical alerts. In the second case, the first_notification_delay should cover the “don’t bug me unless it has been happening for three hours” caveat and I set that service to only notify on warnings and recovery.

Feedback Welcome


Balancing 3-Phase Power

Link: https://dannyman.toldme.com/2015/03/27/balancing-3-phase-power/

I have some racks in a data center that were designed to use 3-phase PDUs. This allows for greater density, since a 3-phase circuit delivers more watts. The way three-phase works is that the PDU breaks the circuit into three branches, and each branch is split across two of the legs. Something like:

Branch A: leg X->Y
Branch B: leg Y->Z
Branch C: leg Z->X

The load needs to be balanced as evenly as possible across the three branches. When the PDU tells me the power draw on Branch A is say, 3A, I am really sure if that is because of equipment on Branch A or … Branch C? This only becomes a problem when one branch reports chronic overloading and I want to balance the loads.

I chatted with a friend who has a PhD in this stuff. He said that if my servers don’t all have similar characteristics, then the math says I want to randomly distribute the load. That is hard to do on an existing rack.

I came up with an alternative approach. Most of my servers fall into clusters of similar hardware and what I would call a “load profile.” I had a rack powered by two unbalanced 3-phase circuits. I counted through the rack’s server inventory and classified each host into a series of cohorts based on their hardware and load profile. Two three-phase circuits gives me six branches to work with, so I divided the total by six to come up with a “branch quota.” Something like:

<= 2 hadoop node, hardware type A
<= 2 hadoop node, hardware type B
   1 spark node, hardware type A
   1 spark node, hardware type C
   1 misc node, hardware type C

I then sat down with a pad and paper, one circuit on either side of the sheet, and wrote down what servers I had, and where the circuit was relative to quota. So, one circuit might have started off as:

Circuit A, Branch XY:

hadoopA: RM 1
hadoopB: ADD 2
sparkA:  --
sparkC:  ADD 1
misc:    --

I then moved servers around, updating each branch circuit with pencil and eraser as I went, like a diligent Dungeon Master. (And also, very important: the PDU receptacle configuration.)

The end result:



I started moving servers around Thu 15:00, and was done about 16:30, which is also when the hadoop cluster went idle. It kicked back up again at 17:00, and started spinning down around midnight.

What is important is to keep sustained load on the branches under the straight green line, which represents 80% of circuit capacity. You can see that on the left, especially the second circuit had two branches running “hot” and after the re-balancing the branch loads fly closer together, and top off at the green line.

Feedback Welcome

Technical, Technology

AWS: Glacier Restores Can Get Pricey

Link: https://dannyman.toldme.com/2015/04/08/aws-melt-glaciers-slowly/

I have been working with AWS to automate disaster recovery. Sync data up to S3 buckets (or, sometimes, EBS) and then write Ansible scripts to deploy a bunch of EC2 instances, restore the data, configure the systems.

Restoring data from Glacier is kind of a pain to automate. You have to iterate over the items in a bucket and issue restore requests for each item. But it gets more exciting than that on the billing end: Glacier restores can be crazy expensive!

A few things I learned this week:

1) Amazon Glacier restore fees are based on how quickly you want to restore your data. You can restore up to 5% of your total S3 storage on a given day for free. If you restore more than that they start to charge you and at the end of the month you’re confused by the $,$$$ bill.

2) Amazon Glacier will also charge you money if you delete data that hasn’t been in there for at least three months. If you Glacier something, you will pay to store it for at least three months. So, Glacier your archive data, but for something like a rolling backup, no Glacier.

3) When you get a $,$$$ bill one month because you were naive, file a support request and they can get you some money refunded.

Feedback Welcome

Linux, Technical

Easy VM Management on Ubuntu!

Link: https://dannyman.toldme.com/2015/04/22/virt-manager-hell-yeah/

I may have just had a revelation.

I inherited a bunch of ProxMox. It is a rather nice, freemium (nagware) front-end to virtualization in Linux. One of my frustrations is that the local NAS is pretty weak, so we mostly run VMs on local disk. That compounds with another frustration where ProxMox doesn’t let you build local RAID on the VM hosts. That is especially sad because it is based on Debian and at least with Ubuntu, building software RAID at boot is really easy. If only I could easily manage my VMs on Ubuntu . . .

Well, turns out, we just shake the box:

On one or more VM hosts, check if your kernel is ready:

sudo apt-get install cpu-checker

Then, install KVM and libvirt: (and give your user access..)

sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils
sudo adduser `id -un` libvirtd

Log back in, verify installation:

virsh -c qemu:///system list

And then, on your local Ubuntu workstation: (you are a SysAdmin, right?)

sudo apt-get install virt-manager

Then, upon running virt-manager, you can connect to the remote host(s) via SSH, and, whee! Full console access! So far the only kink I have had to iron is that for guest PXE boot you need to switch Source device to vtap. The system also supports live migration but that looks like it depends on a shared network filesystem. More to explore.

UPDATE: You CAN live migrate between local filesystems!

It will get more interesting to look at how hard it is to migrate VMs from ProxMox to KVM+libvirt.

Feedback Welcome

Linux, Technical

Live Migrate VM with Local Storage: virsh migrate

Link: https://dannyman.toldme.com/2015/04/23/virsh-migrate-local-storage-live/

In my previous post I lamented that it seemed I could not migrate a VM between virtual hosts without shared storage. This is incorrect. I have just completed a test run based on an old mailing list message:

ServerA> virsh -c qemu:///system list
 Id    Name                           State
 5     testvm0                       running

ServerA> virsh -c qemu+ssh://ServerB list # Test connection to ServerB
 Id    Name                           State

The real trick is to work around a bug whereby the target disk needs to be created. So, on ServerB, I created an empty 8G disk image:

ServerB> sudo dd if=/dev/zero of=/var/lib/libvirt/images/testvm0.img bs=1 count=0 seek=8G

Or, if you’re into backticks and remote pipes:

ServerB> sudo dd if=/dev/zero of=/var/lib/libvirt/images/testvm0.img bs=1 count=0 seek=`ssh ServerA ls -l /var/lib/libvirt/images/testvm0.img | awk '{print $5}'`

Now, we can cook with propane:

ServerA> virsh -c qemu:///system migrate --live --copy-storage-all --persistent --undefinesource testvm0 qemu+ssh://ServerB/system

This will take some time. I used virt-manager to attach to testvm0’s console and ran a ping test. At the end of the procedure, virt-manager lost console. I reconnected via ServerB and found that the VM hadn’t missed a ping.

And now:

ServerA> virsh -c qemu+ssh://ServerB/system list
 Id    Name                           State
 4     testvm0                        running

Admittedly, the need to manually create the target disk is a little more janky than I’d like, but this is definitely a nice proof of concept and another nail in the coffin of NAS being a hard dependency for VM infrastructure. Any day you can kiss expensive, proprietary SPOF dependencies goodbye is a good day.

Feedback Welcome

About Me, Good Reads, Technical

What’s the Deal With One on Ones?

Link: https://dannyman.toldme.com/2015/06/24/one-on-ones/

Early in my career, I didn’t interact much with management. For the past decade, the companies I have worked at had regular one-on-one meetings with my immediate manager. At the end of my tenure at Cisco, thanks to a growing rapport and adjacent cubicles, I communicated with my manager several times a day, on all manner of topics.

One of the nagging questions I’ve never really asked myself is: what is the point of a one-on-one? I never really looked at it beyond being a thing managers are told to do, a minor tax on my time. At Cisco, I found value in harvesting bits of gossip as to what was going in the levels of management between me and the CEO.

Ben Horowitz has a good piece on his blog. In his view, the one-on-one is an important end point of an effective communication architecture within the company. The employee should drive the agenda, perhaps to the point of providing a written agenda ahead of time. “This is what is on my mind,” giving management an opportunity to listen, refine strategies, clarify expectations, un-block, and provide insight up the management chain. He suggests some questions to help get introverted employees talking.

I am not a manager, but as an employee, the take-away is the need to conjure an agenda: what is working? What is not working? How can we make not merely the technology, but the way we work as a team and a company, more effective?

Feedback Welcome

About Me, News and Reaction, Testimonials

Love is Love

Link: https://dannyman.toldme.com/2015/06/26/love-is-love/

“Love is Love”

–President Barack Obama, June 26, 2015

It was about twenty years ago, I was in college, up late in the computer lab writing an email to President Clinton asking him not to sign the “Defense of Marriage Act” into law. Today, I am proud of my country, and the speed with which we have “evolved” to better recognize more of the civil rights of our people.

Thank you, Justice Kennedy, and to the countless advocates who have helped us all open our eyes.

Feedback Welcome

About Me, Excerpts, Free Style

Around the Bend

Link: https://dannyman.toldme.com/2015/09/22/around-the-bend/

Life has been busy lately. I have failed at carving out time for the little things like keeping up with email and reading and writing. One theme that is just below the surface these days, is an understanding of the Individual’s impermanence, that one will inevitably be swept away down the river. But, the good news is, it is the river that is the thing. You stick your foot in the river, and you feel the tug of the currents: this one fast and warm, that one slow and cool. In life, we are these currents, flowing together, mingling, becoming something identifiable and satisfying while also becoming the river itself.

Death has been on my mind lately. Dad passed about a year back, and the Reaper has expressed an interest in the health of another loved one. I am not opposed to Death. We’re all going to get there. Life, the abused cliché reminds us, is the journey and not the destination. I’ll be forty in January. One can read that as the half way mark. I want to pull over and look around. Close at hand, I see my toddler Son, his eyes wide with the possibilities and joys of life, his future for him to know and hopefully to share with his old man. And, not far off, I see my Father, whom my Son will ever know through stories, mainly told by me. Stories I mainly lack. And, yonder still, my own Grandfather, whom I know mainly through the most exaggerated of stories.

We all come from somewhere, and we are all headed somewhere. This bend in the River knows only a short ways upstream, toward the various and contradictory legends of the Wellsprings, and only a short ways ahead, toward the various legends of the Delta, where we believe the River as we have ever understood it will cease as it merges with the Great Ocean.

The August issue of The Sun Magazine brought with it an interview with Stephen Jenkinson, whom some call “The Death Whisperer” … he packs a lot of great ideas that resonate with me into eight pages. Not bad. What follows is a bit of perspective on the idea of one’s influences.

Hoffner: Who would you say are your influences?

Jenkinson: Anyone who claims to know his or her influences probably doesn’t. I think our influences are a lot subtler than we think. For example, I was born nine years after the closing of Auschwitz and the bombing of Hiroshima half a world away. When those soldiers came home from World War II suffering from post-traumatic stress disorder before we had a name for it, North America created the suburbs for them. I grew up in the unacknowledged presence of those wartime horrors: Auschwitz on the one side, Hiroshima on the other, and the suburbs in the middle. That’s an influence on me.

As a child I was read to every night. Before I even understood the words, I was carried along by the momentum of the human voice. The pageant of the story has its way with you, even if its not in a language you can comprehend. Story is a sublime practice that makes us recognizable to ourselves.

These days I admire the songwriter Leonard Cohen, my countryman and a polestar in the firmament for anyone who has faith in human eloquence. Eloquence is a conjuring; it’s magic, and Cohen is a servant as well as a practitioner and a repository of that magic. He’s a patron saint of the Orphan Wisdom School, unawares. I don’t know what kind of life he lives, but it’s inconceivable to me that those songs might come from a duplicitous nature. In a country that appreciated its artists, he would be a national treasure and wouldn’t have to work five minutes in his life unless he was so inclined. As it is he’s been on the road for years trying to make back all the money his manager stole from him.

I met another of my influences at Harvard. As a young man I was on fire with learning about the historical Jesus. I didn’t come from a religious background, but I applied to Harvard Divinity School and got in. I was determined to be a preacher of some sort. I don’t know what else you could do with that kind of education. At the divinity school I met a fellow who was the living incarnation of a stereotypical televangelist: power-blue suit that didn’t fit so good: too-tight white shirt that was popping its buttons. He was in charge that year of vocational counsel. I told him I planned to get a master of divinity and become a pastor or a minister. He asked me the name of my sponsoring congregation, and I said I hadn’t worked that out yet. Then he asked my denominational affiliation. I told him I didn’t have one. “Son, where do you go to church?” he asked. I said that I didn’t, and he asked, “Well, where did you go to church, then?” No answer. So he said, “Let me understand this: you propose to go into the ministry, and you’ve never been to church?” “Yes sir,” I replied. “Well, I nev-uh,” he said, just like that. I was three questions into my vocational interview, and I was done.

My career as a preacher came to an end at that moment. I was counseled to register for a master of theological studies — a layperson’s degree — instead. That same week I met a preaching instructor whose name was Hugh Morgan Hill, but everyone knew him as Brother Blue. He was a vibrant speaker in the African American tradition. He said I should be in his class. I told him I’d already been counseled out of the master of divinity program. “Nobody needs to know,” he said. On my way to his first class I picked up a harmonica. The class had already begun when I got there, and Professor Hill was in full flight. He was a great storyteller and performer. For some reason I started to play my harmonica along with what he was doing, just improvising.

The next week his wife phoned and asked if I’d come with him to a church service — and bring my harmonica.

I performed with Hill on and off for seven years. It was an unofficial apprenticeship. We traveled all over the U.S. and Canada. This was the era of school integration, remember, and there were race riots in some cities, but since we were together, we were OK. He was a holy man from the ghettos of the American heartland. Virtually everything he did in the world was self-initiated. He never seemed to have a job description. He carved it out every time he stood up to speak. I learned from him the importance of proceeding without the green light, the red carpet, the Get Out of Jail Free card. I was emboldened by his example when I was working in palliative care, because I realized that if I was going to serve these dying people well, then I couldn’t wait for anyone to ask me to do it. And if I’m going to serve the era I’ve been born into well, then I can’t wait for approval and recognition. I’m going to have to proceed without it. If it comes, it comes; if it doesn’t, it doesn’t. That, and a lot more, is what I got from him.

Brother Stephen articulates a lot of ideas about Life and Death. Ideas I am still digesting. But I suppose I can share some notes.

To be Alive is to be In Debt to Death. Everything we have, everything … the food we eat day after day, the clothes we wear, the fuel we use to get around … animals and plants, and if you think about it, the Solar energy that fuels All Of It comes from the slow annihilation of the Sun, as its atoms fuse into ever heavier elements. “What will your death feed?”

The Debt is non-negotiable and it will be re-paid.

“Grief is not sadness. There’s sadness in grief, but grief is not exhausted when the sadness goes away. … Sadness has a shelf life, but grief endures …” I picture a small pot, held over the flame of death. The sadness bubbles and splatters and evaporates. The pot is withdrawn from the flame. There is a residue left over. That residue is grief. It does not boil away. You paint with it. You leave a mark somewhere so that when one needs a reminder that “this too shall pass” they may thus be reminded. This paragraph is painted with grief. I hope it feeds your wisdom.

In the interview, Jenkins has a riff about the need for an element missing in our culture: the Rite of Passage in which childhood ends and adulthood begins. A consumer culture derives better profits from a population that is not asked to Grow Up: You Deserve More and More!! His critique resonates but I disagree with the idea that a Rite must “kill off childhood” … on the very next page he explains that it is misguided to shelter children from the idea of death … I think that Childhood is maybe what lends Death its greatest contrast. Young and full of possibility and very self-involved … adults should not be so self-involved, the grown ups must labor to pay the interest on our life debt … but we still need to grow and learn. While we would prefer for grownups to not be self-involved narcissists we need also ask them to be sufficiently self-involved, and other involved, to cultivate their self awareness.

Other involved is what we ask of a child, and what we give a child, when we share with them the fact of Death. Children can appreciate Wisdoms, just as Adults can appreciate Wonders.

A last thought, while I sat in my favorite coffee shop, taking and making these notes, and watching an Old Man drink in the Joyful Clown Antics of some toddlers across the way, was that nursing homes really ought to co-locate with nurseries. Children bring Life into the Room. Elders bring Life as well. The children are starting from scratch, painting with incoherent vitality. The elders have taken their lives, chipped away at it, and produced works of art, called Lives. Day by day, they reveal these works to the children: some are beautiful, some are perplexing, some are sad, and some are horrible. The children react, embrace, reject, imitate, and iterate. Culture ensues, and the river flows enriched.

Feedback Welcome

About Me, Technical, Technology

IT Strategy: Lose the Religion

Link: https://dannyman.toldme.com/2015/10/29/it-strategy-lose-the-religion/

High Scalability asks “What Ideas in IT Must Die?” My own response . . .

I have been loath to embrace containers, especially since I attended a conference that was supposed to be about DevOps but was 90% about all the various projects around Docker and the like. I worked enough with Jails in the past two decades to feel exasperation at the fervent religious belief of the advantages of reinventing an old wheel.

I attended a presentation about Kubernetes yesterday. Kubernetes is an orchestration tool for containers that sounds like a skin condition, but I try to keep an open mind. “Watch how fast I can re-allocate and scale my compute resources!” Well, I can do that more slowly but conveniently enough with my VM and config management tools . . .

. . . but I do see potential utility in that containers could offer a simpler deployment process for my devs.

There was an undercurrent there that Kubernetes is the Great New Religion that Will Unify All the Things. I used to embrace ideas like that, then I got really turned off by thinking like that, and now I know enough to see through the True Beliefs. I could deploy Kubernetes as an offering of my IT “Service Catalog” as a complimentary option versus the bare metal, hadoop clusters, VM, and other services I have to offer. It is not a Winner Take All play, but an option that could improve productivity for some of our application deployment needs.

At the end of the day, as an IT Guy, I need to be a good aggregator, offering my users a range of solutions and helping them adopt more useful tools for their needs. My metrics for success are whether or not my solutions work for my users, whether they further the mission of my enterprise, and whether they are cost-effective, in terms of time and money.

Feedback Welcome

FreeBSD, Linux, Technical

Variability of Hard Drive Speed

Link: https://dannyman.toldme.com/2015/11/11/variability-of-hard-drive-speed/

Munin gives me this beautiful graph:

Read speed starts around 140MB/s then trails off linearly toward 100MB/s over the four hours it takes to read the entire disk.

This is the result of:

$ for s in a b c d; do echo ; echo sd${s} ; sudo dd if=/dev/sd${s} of=/dev/null; done                                                

3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB) copied, 17940.3 s, 112 MB/s

3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB) copied, 15457.9 s, 129 MB/s

3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB) copied, 15119.4 s, 132 MB/s

3907029168+0 records in
3907029168+0 records out
2000398934016 bytes (2.0 TB) copied, 16689.7 s, 120 MB/s

The back story is that I had a system with two bad disks, which seems a little weird. I replaced the disks and I am trying to kick some tires before I put the system back into service. The loop above says “read each disk in turn in its entirety.” Prior to replacing the disks, a loop like the above would cause the bad disks, sdb, and sdc, to abort read before completing the process.

The disks in this system are 2TB 7200RPM SATA drives. sda and sdd are Western Digital, while sdb and sdc are HGST. This is in no way intended as a benchmark, but what I appreciate is the consistent pattern across the disks: throughput starts high and then gradually drops over time. What is going on? Well, these are platters of magnetic discs spinning at a constant speed. When you read a track on the outer part of the platter, you get more data than when you read from closer to the center.

I appreciate the clean visual illustration of this principle. On the compute cluster, I have noticed that we are more likely to hit performance issues when storage capacity gets tight. I had some old knowledge from FreeBSD that at a certain threshold, the OS optimizes disk writes for storage versus speed. I don’t know if Linux / ext4 operates that way. It is reassuring to understand that, due to the physical properties, traditional hard drives slow down as they fill up.

Feedback Welcome

Free Style, Letters to The Man, News and Reaction, Politics, Religion


Link: https://dannyman.toldme.com/2015/12/08/trump/


Sometimes an image is worth far more than a mere thousand words.

Even the milquetoast is calling him out:

Donald Trump is unhinged. His "policy" proposals are not serious.

— Jeb Bush (@JebBush) December 7, 2015

Feedback Welcome


Christmas Adventure: Micro SIM to Nano SIM

Link: https://dannyman.toldme.com/2015/12/26/micro-sim-to-nano-sim/

On Christmas Eve Santa Claus wanted to upgrade my wife from a Nexus 5 to a Nexus 5X, only to find the SIM card was too big! Google brought him to a helpful web site, and that jolly old elf took scissors and a file and trimmed that card down freehand like some kind of old school Unix SysAdmin.

Feedback Welcome

Arrr! . . . Avast!
Site Archive