The goal here is to remove duplicate entries from the PATH variable.
\nBut before I begin, let’s be clear: there’s no compelling reason to
\nto do this. The shell will, in essence, ignore duplicates PATH entries;
\nonly the first occurrence of any one path is important.
\nTwo motivations drive this exercise.
\nThe first is to look at an awk one-liner that initially
\ndoesn’t really appear to do much at all.
\nThe second is to feed the needs of those who are annoyed by
\nsuch things as having duplicate PATH entries.<\/p>\n
I first had the urge to do this when working with Cygwin<\/a>. Your first thought about how to this might be to break up the path The original idea for this was not mine. I found the basic PATH=$(echo $PATH | awk -v RS=: -v ORS=: ‘!($0 in a) ‘)<\/p>\n And it’s close. It almost works, but before looking at the output, $ cat showpath.sh<\/b> $ bash showpath.sh<\/b> So, back to the original code. !($0 in a) { The condition here is !($0 in a). If the current path is not in the array, $ cat nodupes.sh<\/b> $ bash nodupes.sh<\/b> As I said, it almost works. $ cat nodupes2.sh<\/b> $ bash nodupes2.sh<\/b> And you’re almost there, except for the trailing colon, which is not actually $ cat nodupes3.sh<\/b> $ bash nodupes3.sh<\/b> You may be a bit confused by this at first glance. As I said at the outset, there’s no reason you have to remove
\nOn Windows, which puts almost every executable in a different
\ndirectory, your PATH variable quickly can become overwhelming,
\nso removing duplicates makes it slightly less confusing
\nwhen you’re trying to decipher what’s actually in your PATH variable.<\/p>\n
\ninto the individual elements with sed and
\nthen pass that through sort and uniq to get rid of duplicates.
\nBut you’d quickly realize that that doesn’t work, since you’ve
\nnow reordered the paths, and you don’t want that. You want to keep
\nthe paths in their original order, just with duplicates removed.<\/p>\n
\ncode for it on the internet. I don’t remember exactly where, but
\nI believe it was on Stack Exchange<\/a>.
\nThe original bash\/awk code was something like this:<\/p>\n
\nlet’s look at why\/how it works.
\nTo do that, first notice the -v options. Those set the input
\nand output Record Separator variables that awk uses to separate
\nthe input data into individual records of data
\nand how to reassemble them on output.
\nThe default is to separate them by newlines\u2014that is, each
\nline of input is a separate record.
\nInstead of newlines, let’s use colons as the separators,
\nwhich gives each of the individual paths in the PATH variable
\nas a separate record.
\nYou can see how this works in the following where you change only
\nthe input separator and leave the output separator as the newline,
\nand come up with a simple awk one-liner to print each of the elements
\nof the path on a separate line:<\/p>\n
\nexport PATH=\/usr\/bin:\/bin:\/usr\/local\/bin:\/usr\/bin:\/bin
\nawk -v RS=: ” <<<$PATH<\/p>\n
\n\/usr\/bin
\n\/bin
\n\/usr\/local\/bin
\n\/usr\/bin
\n\/bin<\/p>\n
\nTo help understand it, let’s make it look at bit more awkish by reformatting
\nit so that it has the more normal pattern { action }
\nor condition { action } look to it:<\/p>\n
\na[$0];
\nprint
\n}<\/p>\n
\nIn this, $0 is the current input record, and a is an awk variable
\n(the use of the in operator, tells you that a is an array).
\nRemember, each input record is an individual path from the PATH variable.
\nThe part inside the parentheses, $0 in a tests to see if the path
\nis in the array a.
\nThe exclamation and the parentheses are to negate the condition.
\nSo, if the current path is not in a, the action executes.
\nIf the current path is in a, the action doesn’t execute,
\nand since that’s all there is to the script, nothing happens in that case.<\/p>\n
\nthe code in the action uses the path as a key to
\nreference into the array.
\nIn awk, arrays are associative arrays, and referencing a
\nnon-existent element in an associate array automatically creates
\nthe element.
\nBy creating the element in the array, you’ve now set the array so
\nthat the next time you see the same path element, your condtiion !($0 in a)
\nwill fail and the acton will not execute.
\nIn other words the action will execute only the first time that you see a path.
\nAnd finally, after referencing the array, you print the current path,
\nand awk automatically adds the output separtor.
\nNote that an empty print is equivalent to print $0.
\nLet’s see it in action:<\/p>\n
\nexport PATH=\/usr\/bin:\/bin:\/usr\/local\/bin:\/usr\/bin:\/bin
\necho $PATH | awk -v RS=: -v ORS=: ‘!($0 in a) ‘<\/p>\n
\n\/usr\/bin:\/bin:\/usr\/local\/bin:\/bin
\n:<\/p>\n
\nThe only problem is there’s an extra newline and an extra colon on
\nthe following line.
\nThe extra newline comes from the fact that echo is adding a newline
\nonto the end of the path, and since awk is not treating newlines as
\nseparators, it gets added to the end of the last path,
\nwhich, in this case, causes it to look like awk failed to remove a duplicate.
\nBut awk doesn’t see them as duplicates, it sees
\n\/bin and \/binn.
\nYou can eliminate the trailing newline by using the -n option to echo:<\/p>\n
\nexport PATH=\/usr\/bin:\/bin:\/usr\/local\/bin:\/usr\/bin:\/bin
\necho -n $PATH | awk -v RS=: -v ORS=: ‘!($0 in a) ‘<\/p>\n
\n\/usr\/bin:\/bin:\/usr\/local\/bin:<\/p>\n
\na problem. Empty PATH elements will be ignored, but since you’ve come this
\nfar on this somewhat pointless journey, you might as well go the distance.
\nTo fix the problem, use awk’s printf command rather than print.
\nUnlike print, printf does not automatically include output record separators,
\nso you have to output them yourself:<\/p>\n
\nexport PATH=\/usr\/bin:\/bin:\/usr\/local\/bin:\/usr\/bin:\/bin
\necho -n $PATH | awk -v RS=: ‘!($0 in a) ‘<\/p>\n
\n\/usr\/bin:\/bin:\/usr\/local\/bin<\/p>\n
\nRather than eliminating the trailing separtor,
\nyou’ve reversed the logic, and you’re outputting the separator first,
\nthen the PATH element, so instead of needing to eliminate the
\ntrailing separator, you need to suppress a leading separator.
\nThe record separator is output by the first %s format specifier
\nand comes from the length(a) > 1 ? “:” : “”,
\nso it is only printed when there’s more than one element in the array
\n(that is, the second and subsequent times).<\/p>\n
\nduplicate path entries; they cause no harm.
\nHowever, for some, the simple fact that they are there is
\nreason enough to eliminate them.<\/p>\n