Scripting is like cooking...Part 1
I've posted out of sequence this week, since this is a two parter, and I thought Part 1 should be posted on top for more natural reading. Part 2 is below.
Before I start a big THANK YOU to Robert "Rob The Bloke" Bateman. He posted a reply on my CG Talk Thread that gave me the insight to solve a major, show stopping problem.
I don't think an analogy to my rapid, seat of the pants invention methodologies in the garage will be very accessible to many people. To sum it up, I need to make something that I can't readily purchase, so I scour the garage in search of discarded parts and leftover materials from other projects, and bins of fasteners, and cobble something together by using the lathe, drillpress, grinder, saws, files, oxy-acetylene torch and arc welder.
But almost everyone has a refrigerator and stove.
I like food to taste good, and I get easily bored. So when I need to cook something, I look in the fridge for raw materials for my project. Meats, veggies, cheese and sauces are the fundamental building blocks of all meals. I've got lots of dried spices and herbs, and will sometimes employ a canned good that has something tomatoey. Even peanut butter is useful if you want to make something with a thai peanut sauce. No cookbooks, just an idea where you grab ingredients, mix sauces together, and in the end come up with some sort of dish or sandwich that isn't boring.
It's amazing what kind of variety you can get when you do that.
Until this week, I never really thought much about there being a relationship between the garage mad scientist/engineer/junk-yard warrior and the cook. But it's a consistent theme: see what resources you have available, and combine them to get the result you want.
What made me realize that this is a way of thinking that becomes a repeated theme no matter what you're doing, so long as you're making something, is: I realized how often I do this in scripting.
I'll get to that in a minute. First, I'll revisit last week's Animation Save/Load script.
It's not enough to get something working. You have to test it with the most dire conceivable scene you can find. In this case, I used our gold standard character (to which all other characters must be compared), and a 2000 frame mocap file with keys on every available channel on every joint. That's a lot of data.
Well, it worked. Sorta. Turned out it would take 15 minutes to save, and 45 minutes to load. Not very feasible in production. Since I anticipate using the base functional code in a custome character/shot/scene loader with multiple characters in a scene, this speed was highly impractical.
In the course of revision and restarting the interface, I had to do a lot of things repetitively. This led to refinements of the interface:
1. The animation file is now persistent. On repeated loads, it automatically loads the last file as long as the same Maya startup is still being used.
2. Clicking on the source, then the destination objects, and then hitting the "Map to New Character" button gets really tiresome when you have to do it 30 or more times for every animated object in the character. My first pass at resolving this led to making it a click, click and it's mapped solution: now the "Map to New Character" button was obsolete.
3. Clicking the "Get info from File" and "Get Character Controls" buttons was getting old too, so I made those load automatically when the interface starts. Now those buttons were obsolete.
So I had three obsolete buttons. And I still had to do a "click, click" for each object in the character. The couple of dozen times I had to do this, I pretty much thought the solution for the long file loads was just over the horizon. When I realized it wasn't, I created a "Save Map" and "Load Map" function in the interface. I used two of the old button positions and eliminated the third.
Here's where the cooking ingredients analogy comes in. I didn't want to spend a lot of time on this stuff. So I grabbed the code from the animation save/load global procs off the shelf and added some different sauces, and now I had a "Save Map" and "Load Map" function, with a text field with the file name. And because I solved the persistent file name problem, they'd automatically load and update the interface every time I started it.
This was a huge time saver.
So I was able, over repeated refinements, to speed up the code. I got it down to 10 minutes for a save, and 13 minutes for a load. Still unacceptable for production, but I was getting closer. By loading everything into an array, and working from the array, it sped things up and saved multiple iterations of code. Disabling the "undo cue" also saved a bunch of time and freed up some memory. That really adds up when you've got a 300,000 line file that you're saving/reading.
But I couldn't figure out how to get the time down from that. And if I repeated the operation with the same Maya interface open, that 13 minutes would jump up to 30 minutes. And suck up a lot of memory that never came back. Something was wrong, and I was thinking about learning to use the API. But I still had to solve this problem right away, and finally posted to CG Talk about it. Prospects of solving the problem seemed dismal at first, and then Rob The Bloke came to my rescue.
It turns out that printing to the script editor is unreasonably inefficient. It sucks up huge amounts of memory and time. Not really noticeable with small code, but very very noticeable with something large and highly iterative. For debugging, I had it print out the commands that were being used to set the keys on the new character. That's 300,000 lines of printing in this case. It would take up 500 megs of memory, and not give that back. Read Rob's post in the thread I posted at the top to see what he said.
This was a profound insight. Deleting the print commands on both writing and reading the files sped writes to 38 seconds (also using fflush as Rob advised contributed), and reads to 2 minutes and 30 seconds or less. Despite his advice to get rid of the array, I found that it consistently knocked 10 seconds off the read time, so I went back to it. I may be revisiting this in the future for further optimization, but WHAT A DIFFERENCE!
I'm now going to have to go back to all my highly iterative code that, for example, reads the data off every single vertex in a character, and eliminate the print commands for all of it. It's already made a huge difference in my "Save Weights" and "Read Weights" tools. Once a tool works and you don't have to debug anymore, get rid of the print commands!
Before I start a big THANK YOU to Robert "Rob The Bloke" Bateman. He posted a reply on my CG Talk Thread that gave me the insight to solve a major, show stopping problem.
I don't think an analogy to my rapid, seat of the pants invention methodologies in the garage will be very accessible to many people. To sum it up, I need to make something that I can't readily purchase, so I scour the garage in search of discarded parts and leftover materials from other projects, and bins of fasteners, and cobble something together by using the lathe, drillpress, grinder, saws, files, oxy-acetylene torch and arc welder.
But almost everyone has a refrigerator and stove.
I like food to taste good, and I get easily bored. So when I need to cook something, I look in the fridge for raw materials for my project. Meats, veggies, cheese and sauces are the fundamental building blocks of all meals. I've got lots of dried spices and herbs, and will sometimes employ a canned good that has something tomatoey. Even peanut butter is useful if you want to make something with a thai peanut sauce. No cookbooks, just an idea where you grab ingredients, mix sauces together, and in the end come up with some sort of dish or sandwich that isn't boring.
It's amazing what kind of variety you can get when you do that.
Until this week, I never really thought much about there being a relationship between the garage mad scientist/engineer/junk-yard warrior and the cook. But it's a consistent theme: see what resources you have available, and combine them to get the result you want.
What made me realize that this is a way of thinking that becomes a repeated theme no matter what you're doing, so long as you're making something, is: I realized how often I do this in scripting.
I'll get to that in a minute. First, I'll revisit last week's Animation Save/Load script.
It's not enough to get something working. You have to test it with the most dire conceivable scene you can find. In this case, I used our gold standard character (to which all other characters must be compared), and a 2000 frame mocap file with keys on every available channel on every joint. That's a lot of data.
Well, it worked. Sorta. Turned out it would take 15 minutes to save, and 45 minutes to load. Not very feasible in production. Since I anticipate using the base functional code in a custome character/shot/scene loader with multiple characters in a scene, this speed was highly impractical.
In the course of revision and restarting the interface, I had to do a lot of things repetitively. This led to refinements of the interface:
1. The animation file is now persistent. On repeated loads, it automatically loads the last file as long as the same Maya startup is still being used.
2. Clicking on the source, then the destination objects, and then hitting the "Map to New Character" button gets really tiresome when you have to do it 30 or more times for every animated object in the character. My first pass at resolving this led to making it a click, click and it's mapped solution: now the "Map to New Character" button was obsolete.
3. Clicking the "Get info from File" and "Get Character Controls" buttons was getting old too, so I made those load automatically when the interface starts. Now those buttons were obsolete.
So I had three obsolete buttons. And I still had to do a "click, click" for each object in the character. The couple of dozen times I had to do this, I pretty much thought the solution for the long file loads was just over the horizon. When I realized it wasn't, I created a "Save Map" and "Load Map" function in the interface. I used two of the old button positions and eliminated the third.
Here's where the cooking ingredients analogy comes in. I didn't want to spend a lot of time on this stuff. So I grabbed the code from the animation save/load global procs off the shelf and added some different sauces, and now I had a "Save Map" and "Load Map" function, with a text field with the file name. And because I solved the persistent file name problem, they'd automatically load and update the interface every time I started it.
This was a huge time saver.
So I was able, over repeated refinements, to speed up the code. I got it down to 10 minutes for a save, and 13 minutes for a load. Still unacceptable for production, but I was getting closer. By loading everything into an array, and working from the array, it sped things up and saved multiple iterations of code. Disabling the "undo cue" also saved a bunch of time and freed up some memory. That really adds up when you've got a 300,000 line file that you're saving/reading.
But I couldn't figure out how to get the time down from that. And if I repeated the operation with the same Maya interface open, that 13 minutes would jump up to 30 minutes. And suck up a lot of memory that never came back. Something was wrong, and I was thinking about learning to use the API. But I still had to solve this problem right away, and finally posted to CG Talk about it. Prospects of solving the problem seemed dismal at first, and then Rob The Bloke came to my rescue.
It turns out that printing to the script editor is unreasonably inefficient. It sucks up huge amounts of memory and time. Not really noticeable with small code, but very very noticeable with something large and highly iterative. For debugging, I had it print out the commands that were being used to set the keys on the new character. That's 300,000 lines of printing in this case. It would take up 500 megs of memory, and not give that back. Read Rob's post in the thread I posted at the top to see what he said.
This was a profound insight. Deleting the print commands on both writing and reading the files sped writes to 38 seconds (also using fflush as Rob advised contributed), and reads to 2 minutes and 30 seconds or less. Despite his advice to get rid of the array, I found that it consistently knocked 10 seconds off the read time, so I went back to it. I may be revisiting this in the future for further optimization, but WHAT A DIFFERENCE!
I'm now going to have to go back to all my highly iterative code that, for example, reads the data off every single vertex in a character, and eliminate the print commands for all of it. It's already made a huge difference in my "Save Weights" and "Read Weights" tools. Once a tool works and you don't have to debug anymore, get rid of the print commands!
posted by Stephen Greenberg at 4:46 PM
0 Comments:
Post a Comment
<< Home