{"id":3366,"date":"2018-11-14T00:49:50","date_gmt":"2018-11-14T00:49:50","guid":{"rendered":"https:\/\/www.appservgrid.com\/paw92\/?p=3366"},"modified":"2018-11-17T14:29:05","modified_gmt":"2018-11-17T14:29:05","slug":"automate-sysadmin-tasks-with-pythons-os-walk-function","status":"publish","type":"post","link":"https:\/\/www.appservgrid.com\/paw92\/index.php\/2018\/11\/14\/automate-sysadmin-tasks-with-pythons-os-walk-function\/","title":{"rendered":"Automate Sysadmin Tasks with Python’s os.walk Function"},"content":{"rendered":"

Using Python’s os.walk function to walk through a tree of files and
\ndirectories.<\/em><\/p>\n

I’m a web guy; I put together my first site in early 1993. And
\nso, when I started to do Python training, I assumed that most of my
\nstudents also were going to be web developers or aspiring web
\ndevelopers. Nothing could be further from the truth. Although some of my
\nstudents certainly are interested in web applications, the majority of them
\nare software engineers, testers, data scientists and system
\nadministrators.<\/p>\n

This last group, the system administrators, usually comes into my
\ncourse with the same story. The company they work for has been writing Bash
\nscripts for several years, but they want to move to a higher-level
\nlanguage with greater expressiveness and a large number of third-party
\nadd-ons. (No offense to Bash users is intended; you can do amazing
\nthings with Bash, but I hope you’ll agree that the scripts can become
\nunwieldy and hard to maintain.)<\/p>\n

It turns out that with a few simple tools and ideas, these system
\nadministrators can use Python to do more with less code, as well as create
\nreports and maintain servers. So in this article, I describe
\none particularly useful tool that’s often overlooked: os.walk, a
\nfunction that lets you walk through a tree of files and
\ndirectories.<\/p>\n

os.walk Basics<\/h3>\n

Linux users are used to the ls command to get a list of files in a
\ndirectory. Python comes with two different functions that can return
\nthe list of files. One is os.listdir, which means the “listdir”
\nfunction in the “os” package. If you want, you can pass the name of a
\ndirectory to os.listdir. If you don’t do that, you’ll get the names
\nof files in the current directory. So, you can say:<\/p>\n

In [10]: import os<\/p>\n

When I do that on my computer, in the current directory, I get the following:<\/p>\n

In [11]: os.listdir(‘.’)
\nOut[11]:
\n[‘.git’,
\n‘.gitignore’,
\n‘.ipynb_checkpoints’,
\n‘.mypy_cache’,
\n‘Archive’,
\n‘Files’]<\/p>\n

As you can see, os.listdir returns a list of strings, with each
\nstring being a filename. Of course, in UNIX-type systems, directories
\nare files too\u2014so along with files, you’ll also see subdirectories
\nwithout any obvious indication of which is which.<\/p>\n

I gave up on os.listdir long ago, in favor of
\nglob.glob, which means
\nthe “glob” function in the “glob” module. Command-line users are used
\nto using “globbing”, although they often don’t know its name. Globbing
\nmeans using the * and ? characters, among others, for more flexible
\nmatching of filenames. Although os.listdir can return the list of
\nfiles in a directory, it cannot filter them. You can though with
\nglob.glob:<\/p>\n

In [13]: import glob<\/p>\n

In [14]: glob.glob(‘Files\/*.zip’)
\nOut[14]:
\n[‘Files\/advanced-exercise-files.zip’,
\n‘Files\/exercise-files.zip’,
\n‘Files\/names.zip’,
\n‘Files\/words.zip’]<\/p>\n

In either case, you get the names of the files (and subdirectories) as
\nstrings. You then can use a for loop or a list comprehension to iterate
\nover them and perform an action. Also note that in contrast with
\nos.listdir, which returns the list of filenames without any path,
\nglob.glob returns the full pathname of each file, something I’ve
\noften found to be useful.<\/p>\n

But what if you want to go through each file, including every file in
\nevery subdirectory? Then you have a bit more of a problem. Sure, you could
\nuse a for loop to iterate over each filename and then use
\nos.path.isdir to figure out whether it’s a subdirectory\u2014and if so,
\nthen you could get the list of files in that subdirectory and add them
\nto the list over which you’re iterating.<\/p>\n

Or, you can use the os.walk function, which does all of this and
\nmore. Although os.walk looks and acts like a function, it’s actually a
\n“generator function”\u2014a function that, when executed, returns a
\n“generator” object that implements the iteration protocol. If you’re
\nnot used to working with generators, running the function can be
\na bit surprising:<\/p>\n

In [15]: os.walk(‘.’)
\nOut[15]: <generator object walk at 0x1035be5e8><\/p>\n

The idea is that you’ll put the output from os.walk in a
\nfor
\nloop. Let’s do that:<\/p>\n

In [17]: for item in os.walk(‘.’):
\n…: print(item)<\/p>\n

The result, at least on my computer, is a huge amount of output,
\nscrolling by so fast that I can’t read it easily. Whether that
\nhappens to you depends on where you run this for loop on your
\nsystem and how many files (and subdirectories) exist.<\/p>\n

In each iteration, os.walk returns a tuple containing three
\nelements:<\/p>\n