{"id":1996,"date":"2018-10-31T00:44:59","date_gmt":"2018-10-31T00:44:59","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw92\/?p=1996"},"modified":"2018-10-31T03:44:40","modified_gmt":"2018-10-31T03:44:40","slug":"normalizing-filenames-and-data-with-bash","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2018\/10\/31\/normalizing-filenames-and-data-with-bash\/","title":{"rendered":"Normalizing Filenames and Data with Bash"},"content":{"rendered":"

URLify: convert letter sequences into safe URLs with hex
\nequivalents.<\/em><\/p>\n

This is my 155th column. That means I’ve been writing for Linux
\nJournal<\/em> for:<\/p>\n

$ echo “155\/12” | bc
\n12<\/p>\n

No, wait, that’s not right. Let’s try that again:<\/p>\n

$ echo “scale=2;155\/12” | bc
\n12.91<\/p>\n

Yeah, that many years. Almost 13 years of writing about shell scripts and
\nlightweight programming within the Linux environment. I’ve covered a lot
\nof ground, but I want to go back to something that’s fairly basic and
\ntalk about filenames and the web.<\/p>\n

It used to be that if you had filenames that had spaces in them, bad things would
\nhappen: “my mom’s cookies.html” was a recipe for disaster, not
\ngood cookies\u2014um, and not those sorts of web cookies either!<\/p>\n

As the web evolved, however, encoding of special characters became the norm,
\nand every Web browser had to be able to manage it, for better or worse. So
\nspaces became either “+” or %20 sequences, and everything else that
\nwasn’t a regular alphanumeric character was replaced by its hex ASCII
\nequivalent.<\/p>\n

In other words, “my mom’s cookies.html” turned into
\n“my+mom%27s+cookies.html” or “my%20mom%27s%20cookies.html”.
\nMany symbols took on a second life too, so “&” and “=” and
\n“?” all got their own meanings, which meant that they needed to be
\nprotected if they were part of an original filename too. And what about if
\nyou had a “%” in your original filename? Ah yes, the recursive nature
\nof encoding things….<\/p>\n

So purely as an exercise in scripting, let’s write a script that
\nconverts any string you hand it into a “web-safe” sequence. Before
\nstarting, however, pull out a piece of paper and jot down how you’d solve
\nit.<\/p>\n

Normalizing Filenames for the Web<\/h3>\n

My strategy is going to be easy: pull the string apart into individual
\ncharacters, analyze each character to identify if it’s an alphanumeric,
\nand if it’s not, convert it into its hexadecimal ASCII equivalent,
\nprefacing it with a “%” as needed.<\/p>\n

There are a number of ways to break a string into its individual letters,
\nbut let’s use Bash string variable manipulations, recalling that
\n${#var}
\nreturns the number of characters in variable $var, and that
\n$ will
\nreturn just the letter in $var at position x. Quick now, does indexing start
\nat zero or one?<\/p>\n

Here’s my initial loop to break $original into its component letters:<\/p>\n

input=”$*”<\/p>\n

echo $input<\/p>\n

for (( counter=0 ; counter < ${#input} ; counter++ ))
\ndo
\necho “counter = $counter — $”
\ndone<\/p>\n

Recall that $* is a shortcut for everything from the invoking command line
\nother than the command name itself\u2014a lazy way to let users quote the
\nargument or not. It doesn’t address special characters, but that’s
\nwhat quotes are for, right?<\/p>\n

Let’s give this fragmentary script a whirl with some input from the
\ncommand line:<\/p>\n

$ sh normalize.sh “li nux?”
\nli nux?
\ncounter = 0 — l
\ncounter = 1 — i
\ncounter = 2 —
\ncounter = 3 — n
\ncounter = 4 — u
\ncounter = 5 — x
\ncounter = 6 — ?<\/p>\n

There’s obviously some debugging code in the script, but it’s
\ngenerally a good idea to leave that in until you’re sure it’s working
\nas expected.<\/p>\n

Now it’s time to differentiate between characters that are acceptable
\nwithin a URL and those that are not. Turning a character into a hex sequence
\nis a bit tricky, so I’m using a sequence of fairly obscure
\ncommands. Let’s start with just the command line:<\/p>\n

$ echo ‘~’ | xxd -ps -c1 | head -1
\n7e<\/p>\n

Now, the question is whether “~” is actually the hex ASCII sequence
\n7e or not. A quick glance at http:\/\/www.asciitable.com<\/a> confirms that, yes, 7e is
\nindeed the ASCII for the tilde. Preface that with a percentage sign, and
\nthe tough job of conversion is managed.<\/p>\n

But, how do you know what characters can be used as they are? Because of the weird
\nway the ASCII table is organized, that’s going to be three ranges:
\n0\u20139 is in one area of the table, then A\u2013Z in a second area and
\na\u2013z in a
\nthird. There’s no way around it, that’s three range tests.<\/p>\n

There’s a really cool way to do that in Bash too:<\/p>\n

if [[ “$char” =~ [a-z] ]]<\/p>\n

What’s happening here is that this is actually a regular expression (the
\n=~) and a range [a-z] as the test. Since the action
\nI want to take after
\neach test is identical, it’s easy now to implement all three tests:<\/p>\n

if [[ “$char” =~ [a-z] ]]; then
\noutput=”$output$char”
\nelif [[ “$char” =~ [A-Z] ]]; then
\noutput=”$output$char”
\nelif [[ “$char” =~ [0-9] ]]; then
\noutput=”$output$char”
\nelse<\/p>\n

As is obvious, the $output string variable will be built up to have the
\ndesired value.<\/p>\n

What’s left? The hex output for anything that’s not an otherwise
\nacceptable character. And you’ve already seen how that can be implemented:<\/p>\n

hexchar=”$(echo “$char” | xxd -ps -c1 | head -1)”
\noutput=”$output%$hexchar”<\/p>\n

A quick run through:<\/p>\n

$ sh normalize.sh “li nux?”
\nli nux? translates to li%20nux%3F<\/p>\n

See the problem? Without converting the hex into uppercase, it’s a bit
\nweird looking. What’s “nux”? That’s just another step in the subshell
\ninvocation:<\/p>\n

hexchar=”$(echo “$char” | xxd -ps -c1 | head -1 |
\ntr ‘[a-z]’ ‘[A-Z]’)”<\/p>\n

And now, with that tweak, the output looks good:<\/p>\n

$ sh normalize.sh “li nux?”
\nli nux? translates to li%20nux%3F<\/p>\n

What about a non-Latin-1 character like an umlaut or an n-tilde? Let’s
\nsee what happens:<\/p>\n

$ sh normalize.sh “Se\u00f1or G\u00fcnter”
\nSe\u00f1or G\u00fcnter translates to Se%C3B1or%200AG%C3BCnter<\/p>\n

Ah, there’s a bug in the script when it comes to these two-byte character
\nsequences, because each special letter should have two hex byte sequences. In
\nother words, it should be converted to se%C3%B1or g%C3%BCnter (I restored the
\nspace to make it a bit easier to see what I’m talking about).<\/p>\n

In other words, this gets the right sequences, but it’s missing
\na percentage sign\u2014%C3B should be %C3%B, and
\n%C3BC should be %C3%BC.<\/p>\n

Undoubtedly, the problem is in the hexchar assignment subshell statement:<\/p>\n

hexchar=”$(echo “$char” | xxd -ps -c1 | head -1 |
\ntr ‘[a-z]’ ‘[A-Z]’)”<\/p>\n

Is it the -c1 argument to xxd? Maybe. I’m going to leave identifying and
\nfixing the problem as an exercise for you, dear reader. And while you’re
\nfixing up the script to support two-byte characters, why not replace
\n“%20” with “+” too?<\/p>\n

Finally, to make this maximally useful, don’t forget that there are a
\nnumber of symbols that are valid and don’t need to be converted within
\nURLs too, notably the set of “-_.\/!@#=&?”, so you’ll want to
\nensure that they don’t get hexified (is that a word?).<\/p>\n

Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

URLify: convert letter sequences into safe URLs with hex equivalents. This is my 155th column. That means I’ve been writing for Linux Journal for: $ echo “155\/12” | bc 12 No, wait, that’s not right. Let’s try that again: $ echo “scale=2;155\/12” | bc 12.91 Yeah, that many years. Almost 13 years of writing about … <\/p>\n

Continue reading “Normalizing Filenames and Data with Bash”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1996","post","type-post","status-publish","format-standard","hentry","category-linux"],"_links":{"self":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/1996","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/comments?post=1996"}],"version-history":[{"count":1,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/1996\/revisions"}],"predecessor-version":[{"id":2052,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/posts\/1996\/revisions\/2052"}],"wp:attachment":[{"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/media?parent=1996"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/categories?post=1996"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.appservgrid.com\/paw92\/index.php\/wp-json\/wp\/v2\/tags?post=1996"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}