Want to Learn Python – Starter Pack
I am not going to cover actual Python coding here (well, may be, a little at the end) but the good and bad places to start and things to avoid.
First used at the Hollywood, Florida and Rome, Italy IBM Technical University conferences – we call them TechU
Alternatives
- You could just search Google, YouTube, and many other places and find 10 billion hits
- You will quickly get totally swamped with options
- This is Nigel’s starter pack for a quick start.
- This is what I found very useful – You, of course, may be different !!!
What is Python good for?
- Data Scientist job & serious mega-bucks – You can double your already large salary!
- New new technology areas line PowerAI, Artificial Intelligence , Machine Learning, Deep Learning, etc.
- Data manipulation fixing a file format and restructuring the data
- Web 2.0 web pages + REST API
How to develop code & run Python
- Edit file and run file
- IDE (integrated development environment)
- Initially IDE is a pain in the backside
- As you have to learn both the IDE and Language together
- This sets you back 1 month!
- But good for a full time developer
- I recommend: edit and run but also you can run the python in console mode to try things out.
- Having Programmed in Python for about a year I think I am ready to try a IDE for slicker editing and debugging.
- Probably the PyCharm Community Edition IDE for a start.
Environments
- Windows = yuck!
- Tablet – you can run PyCharms IDE but get yourself a Keyboard for typing.
- OSX = if you really have too! Sorry never really got on with the Mac
- Linux = this is the natural home of Python.
- I am using a 160 CPU, 256 GB RAM, POWER8 S922LC – rather overkill but it is fast 🙂
- I also use a Raspberry Pi – that is pretty quick too if the data files are not about a few 1/2 GB. The Raspberry Pi memory is limited.
- AIX
- it is in the AIX Open Source toolbox for downloading
- take care with exotic modules as might have to use git & compile them yourself
How does Python actually run?
- Compiled – No like say C
- Interpreted – Yes but highly optimised, cached and parallelised. I have had some code that finishes so fast I assumed it crashed but it actually work.
Which Python version 2.7 or 3.x ?
- 3.<latest> – at the rime of writing 3.5 to 3.7 depending on how current your OS is!
- No one is writing 2.7 any more
- But there is lots of it in use today but declining over time
- Not a massive difference but best to learn Python 3
Quick Assumption: You have in the past done at least some of these?
- C, Korn or bash shell script writing – excellent
- C programming – brilliant
- JavaScript programs – very good
- Python Programming – why are you reading this???
Then you are already done the heavy lifting
Everyone can write a simple program!
A=42 print "The number is " $A if [[ $A == 42 ]] then print "eureka" fi
Plus For loop & Functions
What is this? Well is work on my Korn Shell OK on AIX.
Mega Tip 1: If you know any of the languages above then Python is going to be very simple
- Updated Read the magic Python by Example or watch the video by the originator Derek Banas
- A big thanks for GjB in the comments for finding the link. I had completely lost it.
- YouTube link (which is includes a link to the code exam
ples ): htt ps:/ /www .you tube .com /wat ch?v =N4m EzFD jqtA &t=3 s - Approaching 5 million YouTube views – amazing for a techie video
- The video is 43 minutes and fast paced. A whole language with examples in under an hour.
- The web page dire
ctly : htt p:// www. newt hink tank .com /201 4/11 /pyt hon- prog ramm ing / - Full marks to Derek Banas the content.
- Briefly the contents
- Data types:
- string,
- integers & float,
- tuples,
- lists,
- dictionary
- Converting between them
- Conditionals: if, then, else
- Loops: for, while
- Functions
- User input
- String manipulation
- File I/O: read and write
- Classes and objects
- Inheritance <– IMHO very advanced and for class module developers
- Polymorphism <– IMHO very advanced and for class module developers
Mega Tip 2: Socratica videos on YouTube
We looks at many training course, Online content and YouTube series’ and these are by far the best and absolutely free.
- Python Programming Tutorials (Computer Science)
- Concise with dry humour and some computer jokes – see recursion
- Mostly with worked example
- Excellent style
- Caltech grads
- 33 videos (Don’t watch the two or three for Python2)
- Most ~8 minutes
- Total 3.5 hours
- 15 million views
- YouT
ube Socr atic a Pl ayli st V ideo s - A Geek person told me Socraticia is the female for of Socrates – I think the creators are female. They also cover maths.
- I have watched all of these twice – about 6 months apart
- They are short but to consolidate what your learn try to have a quick go yourself on each topic
Mega Tip 3: python.org = This is the Python Mother Ship!!
- https://python.org
- Then Downloads then Alternative Platforms
- For all the official documentation http
s:// docs .pyt hon. org/ 3/tu tori al / - Tiny example
- Also if you are stuck for the syntax of a statement or the details of some module or function then use then Google: python3 <your questions spelt out in full>
- Often you get http://Python3.org but http
://s tack over flow .co m answer with worked examples is very good but scan down the answer a bit (the first might not be the best answer or exactly what you want)
Mega Tip 4: Get yourself a project to force you to code and work though problems and new features
- Something simple
- Something you are interested in
- Specially web focused
- Python strong at
- Website interaction
- REST API to an online service
- Data mani
pula tion /tra nsfo rmat io n - File conversion / filtering
Mega Idiot: My first project was the REST API to a HMC to extract Temp, Watts + performance stats for SSP, server & VM
- It was a BIG mistake
- The bad news was the API was so badly documented it was actually impossible to use!
- With totally unnecessarily complicated XML – using features that are very rarely used by anyone.
- I had to interview the developers in the end to workout the hidden details of the REST API
- In simple terms it was the “REST API from Hell!”
- But I learnt a lot
- In the end wrote a Python class module to hid the horrible REST API from Python programmers – its 1100 line of code.
- It returns simple to use Python data structures
- So in simply ~40 lines of Python to extract, manipulate & save in:
- CSV file,
- .html with GoogleChart graphs
- Insert into an influxDB database
Mega Tip 5: JSON format files are exactly the same as the Python native data type called Dictionaries
- So when learning Python concentrate on Dictionaries
- These are (very simple) { “some label”: data, more here }
- and the data can be
- “Strings” in double or single quote
- Integers like 12345 or -42
- Floating point numbers 123.456 (note the decimal point)
- Often we have a list of dictionaries – lists look like [ item, item, item, . . . ]
JSON file example of stats called “mydata.json”:
[ # list of samples { # 1st sample = Python dictionary "datetime": "2018-04 -16T 00:0 6:32 ", "cpus_active": 32, "mhz": 3521, "cpus_description": "PowerPC_POWER9., "cpu_util": { "user": 50.4, .sys": 9.0, "idle": 40.4, "wait": 0.2 } }, .# end of 1st sample { . . . } # 2nd sample = Python dictionary ]
Python Program to load the data file above – NEW Fixed a few Typos here, due to Cut’n’paste issues i.e. double quotes became full stops.
# Read the file as plain text f = open("my data .jso n"," r") text = f.read() f.close() # convert to Dictionary import json #module to handle JSON format jdata = json.loads(text)
- That json.loads() function converts a string (text) to the dictionary called jdata at 10’s of MBs of JSON a second.
- Now lets extract certain fields using a natural Python syntax
# get the Mhz from the first record (numbers zero) print("M Hz=% d"%( jdat a[0] ["mh z"]) ) # Loop through all the records pulling out the MHz numbers and the CPU utilisation user mode percent (its in sub dictionary called cpu_util) for sample in jdata: prin t("M Hz=% d"%( samp le[" mhz" ])) print("User perc ent= %d"% (sam ple[ "cpu _uti l"][ "use r"]) )
Latest project using Python is njmon for AIX and Linux – the new turbo nmon.
-
The J is for JSON and we use Python to make data handling very easy
- For AIX uses libperfstat C library – if you want details see: man libperfstat on AIX or vi /usr
/inc lude /lib perf stat .h - Or find the worked example C code in KnowledgeCenter
- Status quirky but usable for expert C programmer
- Vast quantity of perf stats running in to 1000 stats for AIX and VIOS (if you have many disks, nets or ask for processes stats then that grows rapidly)
- And for a bonus libperfstat gives us the current CPU MHz
- Similar for Linux
- njmon written in C to use C function into the UNIX kernel generates JSON data. Then we use Python to accept the data and inject it live in to a Time Series Data fro graphing in real-time
Stand by for something strange
- Well known programming problem = swamping the values of two variables a and b. Classic solution is using a temporary variable.
temp = a a = b b = temp
- But can you do that without the temp variable?
- No in C – I have known this to 40 years!!
- Python answer
a,b = b,a
- It is using a native data structure called a tuple. As its a common programming task they built it into the language.
- Warning weirdness next:
- How about this?
-
a = a + b b = a - b a = a - b
- Wow! I thought it was impossible!
Next a tiny Web grabbing Python example
- Lots of websites and web services keep stats that you can download with your browser.
- I have used sourcfogre.net (used below) and youtube.com for examples.
- They are most often in JSON and Python has a requests module that makes “talking” to website very simple
- As an example bung this in your browser ( NOT Internet Explorer )
- http
s:// sour cefo rge. net/ proj ects /nmo n/fi les/ stat s/js on?s tart _dat e=20 00-1 0-29 &end _dat e=20 20-1 2-31 &os_ by_c ount ry=f als e - And you should get a load of JSON date back that Firefox and Chrome will organise and make pretty.
- Using Python, requests module and one of my own for graphing we can draw t downloads from the nmon project on SourceForge over time
- We also need to change the date format from which shows of some of Pythons simple data manipulation
- ,[2018-09-17 00:00:00],2
- to
- ,[‘D
ate( 2018 ,9,1 7,00 ,00, 00)’ , 2] - Below is the source code – with many extra print lines and comments so if you run it would will see the data structures.
- NEW Changes the code here to NOT relying on my nchart Python Module
- Green bits a debug but useful it you run it to see the data
- Red bits are the webpage preable and postamble to setup the Googlecahert library graph.
#!/usr/bin/python3 #------- ---- ---- ---- ---- ---- ---- -- Get the data using REST API from sourceforge.net import requests URL= 'htt ps:/ /sou rcef orge .net /pro ject s/nm on/f iles /sta ts/j son? star t_da te=2 000- 10-2 9&en d_da te=2 020- 12-0 4&os _by_ coun try= fals e' ret = requests.get(URL) prin t(re t.st atus _cod e) #print("return code was %d"% (ret .sta tus_ code )) #print("characters returned %d"% (len (ret .tex t)) ) #--- ---- ---- ---- ---- ---- ---- ---- --- Create dictionay import json jdata = json .loa ds(r et.t ext) #print(jdata) months=0 count=0 for row in jdata['downloads']: # print(row) months=months+1 count=count+row[1] prin t("m onth s=%d "%(m onth s)) print("count =%d"%(count)) #--- ---- ---- ---- ---- ---- ---- ---- --- Create web page+graph using Googlechart library file = open ("do wnlo ads. html ","w ") file .wri te(' <htm l>\n ' ' <head>\n' ' <script type ="te xt/j avas crip t" src= "htt ps:/ /www .gst atic .com /cha rts/ load er.j s">< /scr ipt> \n' ' <script type ="te xt/j avas crip t">\ n' ' goog le.c hart s.lo ad(" curr ent" , {"pa ckag es": ["co rech art" ]}); \n' ' goog le.c hart s.se tOnL oadC allb ack( draw Char t);\ n' ' function drawChart() {\n' ' var data = goog le.v isua liza tion .arr ayTo Data Tabl e([\ n' '[{type: "datetime", label: "Date"},"Files"]\n' ) for row in jdata['downloads']: str=row[0] str = str. repl ace( "-", ",") str = str.replace(" ",",") str = str. repl ace( ":", ",") file .wri te(" ,['D ate( %s)' ,%d] \n"% (str ,row [1]) ) file.write(' ]);\n' ' var options = {title: "nmon Downloads", vAxis: {minValue: 0}};\n' ' var chart = new goog le.v isua liza tion .Are aCha rt(d ocum ent. getE leme ntBy Id(" char t_di v")) ;\n' ' chart.draw(data, options);\n' ' }\n' ' </script>\n' ' </head>\n' ' <body>\n' ' <div id="chart_div" style="width: 100%; height: 500px;"></div>\n' ' </body>\n' '</html>\n') file.close()
- The output – skipping the dump of the JSON and the 105 rows of monthly stats looks like this
['2018-05-01 00:00:00', 14153] ['2018-06-01 00:00:00', 12794] ['2018-07-01 00:00:00', 12422] ['2018-08-01 00:00:00', 13127] ['2018-09-01 00:00:00', 11872] ['2018-10-01 00:00:00', 13628] ['2018-11-01 00:00:00', 12805] ['2018-12-01 00:00:00', 15611] months=114 count =686634
- So that was captured in Jan 2019 and so far 686,634 downloads of nmon and its tools and the monthly download generated graph looks like this:
- NEW The generated downloads.html file has the following contents – note I removed a few 100 lines of data in the middle. Colours are from the vim editor – see later comments.
- So that was captured in Jan 2019 and so far 686,634 downloads of nmon and its tools and the monthly download generated graph looks like this:
- NEW Simpler graph
C Programmers be aware:
I keep making the same mistakes in writing Python.
- On Linux with the right export TERM=linux setting and using vi (actually vim) then you have syntax highlighting which reduces errors a lot – go for a white background or comments in dark blue are unreadable. See the picture below – I have not done that colouring – it is all vim.
- vim also helps with auto indentation.
- If, for and while statements have a “:” at the end of the line.
- In Python it is print and in C it is printf – I had to teach my fingers to miss out the final “f”
- Those maddening 4 stop indentations have to be exactly right!
- Anything I missed?
– – – The End – – –