Python make pretty pictures

Well as those who know me, know that I’ve been on sort of a Python kick lately. A few months ago I started learning Python. So now that a few months have past I’ll share the latest reason I like Python… Graphics….

Early in my time with HRD I started working with graphical products. Initially I was playing with a product called IDL. I thought the product was great, but we had a mathematician in join our group who was way more advanced with IDL than I and I let him take control of the product and was very happy to move on to other projects. This said I’ve always been interested in computer graphics as far back as I can remember… in grade school it started with Logo, and then Graphics on an Apple IIe…. Back to the some what present… even in grad school I took a course on OpenGL, even made a nifty 3d maze game. Anyways again I digress (pretty typical of all my blog posts…)

So graphics has always been something I’ve wanted to get into, plus with the always evident idea in my group of making our product portable and deployable, I also new one day that I’d have to find a non-IDL/commercial solution. I state non-commercial as well since I could also have looked at MatLab or such. Anyways the first two products that leapt out at me a while back were GDL a GNU clone of IDL and Octave, another GNU project compatible with MatLab. The only problem was that GDL at the time did not support many of the IDL tools we were now using, and Octave… well I still don’t know MatLab so that wasn’t going to work.

Anyways…. so I place the idea of a free and distributable version of our product tools on the very back burner… you know the one that was never even attached to the stove…

So we move foward in time to the real present… Like I said in the beginning of the post, a few months ago I started learning Python. Like learning any new programming language, it is mostly syntax and package/module discovery. The syntax part was straight forwards, I’m a C/Fortran/Java/Unix scripter…. so yes, Python syntax, not anything new. As for packages… well they are abuyndant, but easily managable. Unlike Perl which I don’t even know where to start on how large it has gotten (and yes in my early days I was even a Perl programmer), trying to find the ‘best’ package can take as long as trying to write your own… So as for Python, I quickly learned to read and write ASCII files and perform some simple mathematical calculations on data and a few other simpler tasks.

Then a co-worker of mine asked me about writing a program for a project he was looking into. The project seemed straight forward enough… read in a couple years worth of data (all of which were simple 2d matrixs), perform some simple stats on them (mean, std, median, min, max), spit out a simple ASCII summary, and generate some pretty pictures. All of the first steps seemed straight forward, the pretty pictures… well that was a different story. Anyways still being on my Pytohn kick, I quickly started out on this new project in Python. Took a couple of days to get the files read and stats processed, was even easier when I learned about NumPy and SciPy. NumPy is great, it provides great array handling for large matrixes and such. Plus now the reason for this post… an additional package called MatPlotLib is also part of the NumPy/SciPy family. MatPlotLib made plotting the matrixes an almost breeze. Basically once I figured out my dimensions and the geospatial area for my map to plot on, all I had to do was give it my NumPy array and say draw, and poof…. pretty picture.

Don’t ask me what the picture is exactly of, but ain’t it pretty? I do know that it is some sort of thermal data from the GOES satellite (or atleast I think it is from the GOES satellite). But ain’t that cool… all I did was create Basemap and then tell it to plot using the pcolor(…) command. It was that simple. Another reason I think Python and I are going to have a long happy relationship. A few more tweaks and I should have my program fully ready to go… only issue is… currently it is kind of slow.

When White spaces aren’t White

As most know, most of my programming work relates to data processing. One of my responsibilities with the Hurricane Research Division and the H*Wind team to be exact has been to process data and insert it in our database. So over the years (almost 12) I’ve written a number of programs to help along my way. A few years ago (before I discovered Python), I wrote a rather significant program for my own use. It is a generic parser. I wrote it as part of a project to re-analyze Hurricane Katrina. We had data coming in from all over the place and almost none of it in the same ‘format’, format in quotes since I refer to column order, field delimiters, units, etc. but almost all the data was ASCII. Anyways… I’m digressing from my point… so this parser was basically designed so that the user (basically me) would simply list the column order and such, and out would pop a file ready for the database. Well to the point of the blog post title, today (and yesterday) I was trying to figure out why my program wasn’t working. The default delimiter in my program is WHITESPACE, basically any not alpha-numeric-symbol character, aka space, tab, return…. well incomes this file I processing… it was generated in Fortran, so my first assumption is that it should be space delimited, but it was not, it was TAB delimited. Okay no problem, my program can do that, but it didn’t. For some reason, TAB is not being recognized as a whitespace…. argh, however when I distinctly tell my program with command-line arguments to accept TAB as a delimiter, it works…. argh….. Oh well, at least it works… thankfully. I sure the issue has something to do with being prepared on a HPUX machine, transfered to a Linux box, emailed to a Mac box and then processed…. you know somewhere in there something had to get lost in whitespace translation…. anyways… again the program works, but it is always fun when you write a program to do something and it doesn’t work the way it should… thank goodness for Unix and the many workarounds that are available.

Of course this whole experience is another reason I’m really starting to like Python programming… maybe I should take the time to re-write this c program into Python… then again it works 99% of the time, so if it is not broke, why fix it?

Learning Python

I’m a long time C developer. Basically I figure that anything that needed to be coded could be coded in C. I still hold that belief, but 7-8 years ago I came to the realization that while that may be true, it is not always the best answer. For project reasons at work I learned Fortran and began to recognize its value when it comes to math. For other project reasons I learn how powerful simple Unix scripts can be. So basically for the last 7-8 years I’ve seen Unix scripting as a steering tool, C for text processing, Fortran for math, and Java for object-oriented and gui tasks. But recently based on a number of article from Computing in Science and Engineering (CiSE) I’ve come to learn about Python.

Python is a scripting language at its core, but overall it is much more. Python has a variety of modules and add-ons. Some basic include modules include NumPy/SciPy for math and science, CSV for delimited text, and XML for xml parsing and generation. My first Python program took me only a day or so to develop and test, it was a CSV translator. Basically it converted one CSV file into another. My next program took me another 2-3 days and it converted a gridded file into a CSV formatted file. And today, I wrote my first XML parser. Using ‘minidom’ from the xml package I was able to put together a simple parser for converting an XML file into a csv file.

All this I’ve basically accomplished over a few weeks of learning using only a handful of simple resources both online and in print. Yes I have many years of experience with a variety of programming languages, but I definitely have found Python the easiest to learn yet. Here is a quick list of resources I’ve found helpful:

• Tutorial at Python.org
• Python.org Library Reference
• Dive into Python by Mark Pilgrim
• Python/XML Howto by A.M. Kuchling
• Python XML Frequently Asked Questions by Dave Kuhlman
• Book: XML Processing with Perl, Python, and PHP by Martin C. Brown
• Magazine: CiSE issue: Python: Batteries Included Volume 9, Number 3
• Python and XML: An Introduction
• Creating XML with Python

I’ve bookmarked a number of other sites for future use but have not gotten to them yet. My next task will be to add a simple XML version of the gridded data file mentioned above, plus add NetCDF writing support and GIS writting support, namely ESRI Shapefiles. I also plan on learning howto wrap my C packages with Python binding. To put it plainly, things that I’ve spent years developing and testing in C for parsing weather related data, will all slowly be phased out and replaced with python code and modules and all opensource.

update 2008-03-19: added 2 more links