Shell HOWTO: Remove Duplicate Elements from a Variable

If you are a seasoned Unix admin, you’ve been doing stuff like this for years:

cat $file | sort | uniq

Which is a handy way to eliminate duplicate lines in a file, or a collection of files. The uniq -c will even tell you how many duplicated lines there are, and you might even do:

cat $file | sort | uniq -c | sort -n

For example, you could run a command like this to see who is receiving the most mail on your system:

awk ’{print $7}’ < /var/log/maillog | grep ^to= | sort | uniq -c | sort -rn

Anyway, even if you are a seasoned Unix admin, you probably aren’t a big expert on shell scripting. It is something I like to do not only because automation makes my life better, but to prove that I’m not a Perl weenie. Today I have a script that copies a bunch of files into a directory hierarchy to set things up for a chroot()ed environment:

manifest="/usr/bin/scp /usr/libexec/sftp-server /usr/local/libexec/rssh_chroot_helper /bin/sh"

# Copy manifest
for file in $manifest; do
    mkdir -p $1/`dirname $file`
    cp -p $file $1/$file
    libs="$libs `ldd $file | awk '{print $3}'`"

Before you copy your file to $1, you have to ensure that the target directory exists, hence the dirname. But, in order for the target executable to run, you will also need your shared libraries, which I sniff out with ldd and awk. I could then copy the $libs in much the same way that I am copying $manifest, above. But there are surely duplicates, ya? So, the question is, how do I “uniq” a shell variable?

Twenty seconds of thinking later, and I have an idea, and it works. Very briefly, it is just a question of slicing the spaces into newlines so you can use your everyday Unix admin tools to finish the work:

0-10:00 djh@mito ~> sh
$ libs="a b c d"
$ libs="$libs b c d f g"    
$ echo $libs
a b c d b c d f g
$ libs=`echo $libs | tr ’ ’ ’\n’ | sort | uniq`
$ echo $libs
a b c d f g

So, my script continues:

# Copy libs
libs=`echo $libs | tr ’ ’ ’\n’ | sort | uniq`
libs="$libs /libexec/ld-elf.so.1"
for lib in $libs; do
        mkdir -p $1/`dirname $lib`
        cp -p $lib $1/$lib

And of course, I drop some science here, just in case someone gets stuck in a similar situation.

Read More

Categories: Technical