More tools, interfaces and refinements

I've been writing some minor tools, and going GUI crazy the last couple of weeks.

Removing Print Commands
After the discovery of how major an effect using the print command has, I decided to go through all my major scripts and removing print commands from them. Unless I'm debugging I don't need them. I also added a "Done" window that tells me how long a script takes to my longer scripts like the Animation Save/Load utility, and the spur to joint utility.

In the process, I created a new standalone script called printWin. It consists of three processes: printWin initializes the script, you invoke "addPrintWin " to add the contents of a variable to a line in the text scroll list, and showPrintWin to show the window. I didn't combine the commands because I don't want the window to show up until all the data is added, and I don't want to add showWindow to addPrintWin because it can be used in highly iterative loops, and I don't want to call showWindow over and over again hundreds of times.

I also added a log utility to the printWin window, so I could highlight some or all lines and then save them out to a log file for copying, pasting and testing if there's a malfunction. I can also open it in a text file to search for problem lines in thousands of commands to figure out what the problem is. The name and path of the log file can be changed.

This is how I located a problem with my venerable "spur to joint" script, a major script that takes all the vertices in a character, analyzes their weights, removes them from spurs and adds them to the parent bone's weight assignment. Even if it doesn't have any weights assigned to it (which is pretty typical).

In working on the problem, I decided to rewrite a bit of the code to make it more efficient. A lot of repetitive loops could be combined, and instead of parsing the name of the joint to get rid of the suffix, I could use the substitute command. In the months since I wrote the script originally, I've discovered I don't have to put `gmatch` statements into a variable, but can use them directly in "if" statements without an equal. I didn't know that before.

I also made it so I don't have to select the vertices. I can select the objects or the vertices, and the script will figure it out.

So I got the spur to joint script down to between 2 and 2 1/2 minutes running time from 5 minutes or more.

Cleanup and Windows
I wrote a cleanup utility to conform all 17 current characters to a unified naming convention. This utility forced me to learn how to manipulate GUIs in a much more coherent fashion than I'd been doing before. I needed to throw up separate analytical windows, and I didn't want them all over the place.

Window # 1 has the basic controls with 5 buttons. I used the rowColumnLayout command to make the buttons span evenly across the window, two rows of two and a close button at the bottom. There's a textField to enter a character name, and all the changes executed by the script are logged in a textfile, with new entries appearing below existing ones.

Window # 2 appears directly below it and the left edge lines up with the first. This window lists all the joints that are different from the standard model, so I can examine them and see if there's a misnaming, a difference in the hierarchy that shouldn't be there, or additional bones like tails, ears, ponytails, trunks, etc. I found a lot of names of objects that were out of standard that way.

While Window # 2 originally was designed to be used with one character at a time, I had an occassion to check multiple characters. So I rewrote it yesterday to give me the parent transform node of each character, so I'd know which character had what problem. I'm finding I have to do this often enough that I'm going to have to write a standalone script to find the character names in the scene so I don't keep writing the same code over and over again. I can see many uses for this when I do the scene building scripts.

I align the 2nd window with the first by querying the first. For example:
window -tlc (`window -q -te "window_1"` + `window -q -h "window_1`)
`window -q -le "window_1"` -h 400 "window_2";

And I'd do something like that for all the windows. I also have it so all the windows are minimized if the first one is minimized, and the close window closes all the diagnostic windows associated with the tool.

Window_2 has text on the top, followed by a scrollLayout, followed by text, a scrollLayout and then the buttons. This is pretty handy. But it forced me to learn how to put all of this together.

At the end of the script for window_2, it does a query to find out how wide the text lines are, and scales the window and scroll lists to that size. Since all of the windows next to it use a call like `window -q -w "window_2"` as part of the formula to determine their position when opened, this means that all the windows fit together like pieces of a puzzle, and self assemble even if something is moved.

This all means that I am going back to my old scripts to neaten them up. If I have a row of buttons, rather than explicitly placing them with formLayout -e, I'm using automatic calculation to determine how to get the buttons exactly spaced for the width of the window or frameLayout.

The way I'm accomplishing this is with a line like this:
int $windowWidth = `window -q -w "window_2"`;
int $offset = 3;
int $columnSpace = 3;
rowColumnLayout -e
-cw 1 (($windowWidth/5)-$offset)
-cw 2 (($windowWidth/5)-$offset)
-cw 3 (($windowWidth/5)-$offset)
-cw 4 (($windowWidth/5)-$offset)
-cw 5 (($windowWidth/5)-$offset)
-cs 1 $columnSpace
-cs 2 $columnSpace
-cs 3 $columnSpace
-cs 4 $columnSpace
-cs 5 $columnSpace
"buttonRCL";

$offset is the amount subtracted to compensate for the scroll bar and/or frameLayout margins
I don't subtract that from the window value directly so I can alter the widths of the columns to make the buttons as wide as they need to be to fit the text in each button.

Using these techniques, if a window is resized because it's contents are wider than the initial window size, that's the value that's used to resize all the components that weren't responsible for resizing the window. It's a profound discovery for me.

What Comes First, The Code or the UI?
For a simple tool, it's pretty easy to just write the code. But when I need something that does a lot of complicated things, I find myself dummying up the UI first, which then tells me what I have to write into the script. What do I want the tool to do? How do I want it to work. Writing the UI first is sort of an advanced combination of an outline and flow chart. It's actually saved me a lot of time except when I have to learn new UI tricks, but I only have to learn them once. And I write them very quickly now. It also gives me a good idea of what needs to be broken into separate global procs and what can be combined into one.

But I'm getting to the point where I'm starting to think about writing a UI to build UIs. There are some out there, but as usual, I like the control and know what I want.

Converting Motion Data
What I'm currently working on is a method for converting data on a character done before the character was all zeroed out, and when the character's primary pose was done in an "A" pose instead of a "T" pose. I've chosen to go with a "brute force" quasi-mechanical methodology over actually trying to do matrix conversions. This involves posing the old skeleton in the new skeleton's pose, creating a huge number of transforms, synchronizing one with the old skeleton, synchronizing the other with the new skeleton, parenting new to old and then synchronizing the whole thing frame by frame with the old animation. The data is then extracted from the new skeleton transforms as world space information, and transplanted either onto the skeleton itself or controls that drive the IK. A lot of flexibility can be achieved with various parenting schemes.

That's the theory anyway.

Scripting is like cooking...Part 1

I've posted out of sequence this week, since this is a two parter, and I thought Part 1 should be posted on top for more natural reading. Part 2 is below.

Before I start a big THANK YOU to Robert "Rob The Bloke" Bateman. He posted a reply on my CG Talk Thread that gave me the insight to solve a major, show stopping problem.

I don't think an analogy to my rapid, seat of the pants invention methodologies in the garage will be very accessible to many people. To sum it up, I need to make something that I can't readily purchase, so I scour the garage in search of discarded parts and leftover materials from other projects, and bins of fasteners, and cobble something together by using the lathe, drillpress, grinder, saws, files, oxy-acetylene torch and arc welder.

But almost everyone has a refrigerator and stove.

I like food to taste good, and I get easily bored. So when I need to cook something, I look in the fridge for raw materials for my project. Meats, veggies, cheese and sauces are the fundamental building blocks of all meals. I've got lots of dried spices and herbs, and will sometimes employ a canned good that has something tomatoey. Even peanut butter is useful if you want to make something with a thai peanut sauce. No cookbooks, just an idea where you grab ingredients, mix sauces together, and in the end come up with some sort of dish or sandwich that isn't boring.

It's amazing what kind of variety you can get when you do that.

Until this week, I never really thought much about there being a relationship between the garage mad scientist/engineer/junk-yard warrior and the cook. But it's a consistent theme: see what resources you have available, and combine them to get the result you want.

What made me realize that this is a way of thinking that becomes a repeated theme no matter what you're doing, so long as you're making something, is: I realized how often I do this in scripting.

I'll get to that in a minute. First, I'll revisit last week's Animation Save/Load script.

It's not enough to get something working. You have to test it with the most dire conceivable scene you can find. In this case, I used our gold standard character (to which all other characters must be compared), and a 2000 frame mocap file with keys on every available channel on every joint. That's a lot of data.

Well, it worked. Sorta. Turned out it would take 15 minutes to save, and 45 minutes to load. Not very feasible in production. Since I anticipate using the base functional code in a custome character/shot/scene loader with multiple characters in a scene, this speed was highly impractical.

In the course of revision and restarting the interface, I had to do a lot of things repetitively. This led to refinements of the interface:
1. The animation file is now persistent. On repeated loads, it automatically loads the last file as long as the same Maya startup is still being used.
2. Clicking on the source, then the destination objects, and then hitting the "Map to New Character" button gets really tiresome when you have to do it 30 or more times for every animated object in the character. My first pass at resolving this led to making it a click, click and it's mapped solution: now the "Map to New Character" button was obsolete.
3. Clicking the "Get info from File" and "Get Character Controls" buttons was getting old too, so I made those load automatically when the interface starts. Now those buttons were obsolete.

So I had three obsolete buttons. And I still had to do a "click, click" for each object in the character. The couple of dozen times I had to do this, I pretty much thought the solution for the long file loads was just over the horizon. When I realized it wasn't, I created a "Save Map" and "Load Map" function in the interface. I used two of the old button positions and eliminated the third.

Here's where the cooking ingredients analogy comes in. I didn't want to spend a lot of time on this stuff. So I grabbed the code from the animation save/load global procs off the shelf and added some different sauces, and now I had a "Save Map" and "Load Map" function, with a text field with the file name. And because I solved the persistent file name problem, they'd automatically load and update the interface every time I started it.

This was a huge time saver.

So I was able, over repeated refinements, to speed up the code. I got it down to 10 minutes for a save, and 13 minutes for a load. Still unacceptable for production, but I was getting closer. By loading everything into an array, and working from the array, it sped things up and saved multiple iterations of code. Disabling the "undo cue" also saved a bunch of time and freed up some memory. That really adds up when you've got a 300,000 line file that you're saving/reading.

But I couldn't figure out how to get the time down from that. And if I repeated the operation with the same Maya interface open, that 13 minutes would jump up to 30 minutes. And suck up a lot of memory that never came back. Something was wrong, and I was thinking about learning to use the API. But I still had to solve this problem right away, and finally posted to CG Talk about it. Prospects of solving the problem seemed dismal at first, and then Rob The Bloke came to my rescue.

It turns out that printing to the script editor is unreasonably inefficient. It sucks up huge amounts of memory and time. Not really noticeable with small code, but very very noticeable with something large and highly iterative. For debugging, I had it print out the commands that were being used to set the keys on the new character. That's 300,000 lines of printing in this case. It would take up 500 megs of memory, and not give that back. Read Rob's post in the thread I posted at the top to see what he said.

This was a profound insight. Deleting the print commands on both writing and reading the files sped writes to 38 seconds (also using fflush as Rob advised contributed), and reads to 2 minutes and 30 seconds or less. Despite his advice to get rid of the array, I found that it consistently knocked 10 seconds off the read time, so I went back to it. I may be revisiting this in the future for further optimization, but WHAT A DIFFERENCE!

I'm now going to have to go back to all my highly iterative code that, for example, reads the data off every single vertex in a character, and eliminate the print commands for all of it. It's already made a huge difference in my "Save Weights" and "Read Weights" tools. Once a tool works and you don't have to debug anymore, get rid of the print commands!

Scripting is like cooking...Part 2

So the second bit of this Cooking/Scripting analogy...

This week I was instructed to start looking over a bunch of characters that were done before my time. What I noticed was inconsistencies in the hierarchies, a lot of transforms without children, and different naming conventions for the same joints.

If I'm going to employ tools in the future that can parse and deal with the various body parts, it would sure be nice if they were always named the same thing.

So I wrote a quicky tool in a few hours to deal with this. Doing it by hand was out of the question, because I had a big checklist and I was sure to miss some detail and continue propogating the problem. My current list has 17 characters, so the time to do this by hand would really add up.

I settled on a "gold standard" character whose hierarchy would be used to compare all the other hierarchies.

And I went to the "fridge" of previously written code to borrow code which could be redeployed in the tool. This is a real time saver, when you can look at how you handled a problem before, and not have to reinvent the wheel. I borrowed and modified parsers, cobbled quick interfaces together, pop-up windows and file writers. It does all the functions and adds them to a log file so I have a record of what was done.

What I wound up with:
1. A "promptDialog" window pops up asking me to name the character which leads all the info written to the log file.
2. It automatically renames a bunch of standard non-conforming names of joints like the pelvis, fingers and foot parts.
3. It deletes all transforms that have no children, but not joints, ikHandles or effectors.
4. It pops up a scroll window that gives me a summary of what was done. If it looks like something necessary was deleted, then I'll reload the character and fix the code. So far, I haven't had this problem since the third iteration of the code.
5. It pops up a scroll window that gives me a list on top of joints that exist in the "gold standard" character that don't exist in the character that is being fixed. On the bottom, it gives me a list of joints in the character being fixed that don't exist in the "gold standard" character. Sometimes it's because the character being fixed has joints that the "gold standard" doesn't have like a tail and ears. Or some joints are named with joints that have their lettered incrementers jumping sequence because at one point intermediate joints were removed for redundancy.

But this all gives me a visual list to investigate. It alerts me to name formats that are out of conformity with the general naming convention. It points out joints that were never renamed from the default. It's a very good summary of things that might need to be looked at, and appropriately targeted.
6. It pops up another window with a final checklist for hierarchy conformity that also includes what objects need to have "inherit transforms" turned off.

When all this is done, it writes to a log file. I go and hand edit the log file with any additional steps I take with the lines of hand edited items indented, and finally, I add the file name that the newly fixed character was saved under.

With this, I was able to conform 16 of the 17 characters in under 4 hours (which only took that long because I was updating the code as I found problems, and wrote a new sg_rig utility which I'll need for a different stage of zeroing rotations later).

And then there's the new sg_rig utility. This pops up a summary of the rotation, rotation axis, joint orient and rotation order for specific joints I need to be certain conform to standards we need to meet. I borrowed some of the window code from the naming and hierarchy cleanup of the earlier utility script for that.

I wrote the utility because, while I had the characters up, I wanted to get a preview of the work I had ahead of me and to anticipate some upcoming problems I see with correcting them. I'll have my work cut out for me next week. I'm getting some ideas for maybe logging old data and correcting animation files being loaded from earlier work based on that old data (just in case dealing with this becomes my problem).