Pics, as stated

One, we have the lovely 'Ryuou' / 'Valgirt' from Dragon Quest Swords (11k poly, 272 bones):



Two, we have the awesome robotic 'Kiryu' / 'Mechagodzilla' from Godzilla: Unleashed (6k poly, 69 bones):



These pic's don't show much except that the importing is pretty near flawless into blender, including weights and other nifty things.

Side note, I'm using the GLSL display option in blender, and have a script that converts I8A8 normal maps into RGB = XYZ normal maps, so that blender can use them in the shaders it has. Also, the light setup is a new one I'm experimenting with, that has two hemispherical lights always 180 degrees apart, but only one is applying specular components. It is similar to a ambient light, but seems to be a little nicer. Once I finish my CIE Lighting model shader, I'll try making that mess happen.

And, of course, both of these models are copyrighted to their respective owners. Go buy the damn game(1)(2) if you want them yourself!

-Z

Hacking Success; Dragon Quest Swords broken!

Add:

Godzilla: Unleashed
Rampage: Total Destruction
Dragon Quest Swords
Monster Hunter G
Monster Hunter Tri
Super Smash Bros. Brawl
Carnival Games *
Super Mario Galaxy
NiGHTS
ect...

to the list of games I can extract models from.

I wish I knew who to give credit to, but all I needed was this little function here to finally break apart the .fpk / Dragon Quest Swords compression format (I was really close too, ironically)


typedef unsigned int u32;
typedef unsigned short u16;
typedef unsigned char u8;


int endian=1;

u32 BE32(u32 data)
{
if(endian)
return ( (data<<24) | ((data<<8)&0x00ff0000) |
((data>>8)&0x0000ff00) | (data>>24) );
else
return data;
}

int blen;
int fbuf;

/* PRS get bit form lsb to msb, FPK get it form msb to lsb */
int get_bits(int n, char *sbuf, int *sptr)
{
int retv;

retv = 0;
while(n){
retv <<= 1;
if(blen==0){
fbuf = sbuf[*sptr];
//if(*sptr<256)
//{ printf("[%02x] ", fbuf&0xff); fflush(0); }
(*sptr)++;
blen = 8;
}

if(fbuf&0x80)
retv |= 1;

fbuf <<= 1;
blen --;
n --;
}

return retv;
}

int uncomp(char *dbuf, int dlen, char *sbuf, int slen)
{
int sptr;
int dptr;
int i, flag, len, pos;

blen = 0;

sptr = 0;
dptr = 0;
while(sptr < slen){
flag = get_bits(1, sbuf, &sptr);
if(flag==1){
//if(sptr<256)
//{ printf("%02x ", (u8)sbuf[sptr]); fflush(0); }
if(dptr < dlen)
dbuf[dptr++] = sbuf[sptr++];
}else{
flag = get_bits(1, sbuf, &sptr);
if(flag==0){
len = get_bits(2, sbuf, &sptr)+2;
pos = sbuf[sptr++]|0xffffff00;
}else{
pos = (sbuf[sptr++]<<8)|0xffff0000;
pos |= sbuf[sptr++]&0xff;
len = pos&0x07;
pos >>= 3;
if(len==0){
len = (sbuf[sptr++]&0xff)+1;
}else{
len += 2;
}
}
//if(sptr < 256)
//{ printf("< %08x(%08x): %08x %d> \n", dptr, dlen, pos, len); fflush(0); }
pos += dptr;
for(i=0; i< len; i++){
if(dptr < dlen)
dbuf[dptr++] = dbuf[pos++];
}
}
}

return dptr;
}



Thanks to that mess, I could use my own BRRESToOBM and the Pipeworks1.4ToOBM converters to completely extract the models and import them into blender.

Why do this, you ask?

Simple! Since NOT A SINGLE DAMN PERSON here is competent enough to help me, my only method of QA/QOS/QMS/verification is checking my work against commercial products. It's a very useful method as you often get to see techniques for animation, texturing and shading that are well known to industry but not well documented, or just plain hard to find.

Some of the cool things I've learned:

*Use additional bones inside of the arm joints on your characters; This allows you to retwist the arm visually while still keeping the IK chain unbroken. This is a classic character animation technique that I never had seen in practice.

*Use lots of matrices and bleed the weights out as much as possible, trying not to exceed 4 weights per vertex; Don't just use 0.5/0.5 blending factors (Valgirt has 272 bones...)

*If you are applying a bump/normal map to a character, use the I8A8 texture format and simply calculate the z component in the shader; This costs no additional instructions AND guarantees a normalized z component which you would have to do anyways.

*Use well painted textures, and don;t be afraid to use non-contiguous models (fins, talons, tongues, even neck/finger joints do not always have to be connected to the base mesh!

*Make sure you add in additional transform bones to characters, such as a Root Node and X/Y rotation nodes.

*for mesh modeling, targeting 3k verticies isn't really an issue.

*Always sort your meshes by: Material, Matrix Deformation Pool, Index array type.

*Always convert your final mesh pools into triangle strips if you can; Avoid quads and triangles. If you must use triangles, sort them in a indexed triangle list so that each next triangle shares adjacent verticies. Note that on file loading, you can always convert strips back to triangles; but not the other way around.

*Matrix deformation systems allow you to do things you could never do with bone animation systems; Since you just get a 3x4 float matrix per 'bone', you can apply translation, rotation, scale, and shear transformations. Since the hardware is unaffected by the values you put in those matrices (same number of ops) you're allowed to really have fun animating and be guaranteed consistent results anywhere.

*Mesh keyframes are A-OK and should be combined into your matrix mesh; for example facial poses and stuff. Simply apply those before running your mesh through the shader; You can get very awesome character animations this way.

*Try and avoid making your matricies match a 'skeleton' of you character; This often results in hard mechanical animation that looks not fun. It also makes your artists have to work harder, because a 'bone' does not behave like a matrix. Usually, with matrix deformations, you want the matrix to be at the 'ring center' of the mesh data, so that scaling and transforms will behave as you expect. Interestingly, since matrices have no constraints, you can always make this look like the hard mechanical, but you can't do that the other way around.

*Add additional target matricies; It only costs a small amount of CPU and gives your models flexibility for grapple animations, picking up objects, and easier cinematic animations

*I've seen 11,000 vertex models fully shaded on the Wii. No kidding. With 200 + bones. Think about that for a second. Think about how much more powerful your PC is than that.

*Most importantly, animate/design for fun. Don't make things realistic; It's a waste of your time because commercial studios can always do that better than you can. And who wants to play a realistic game anyways?

I'll post pics later, today I have more beer to acquire!

-Z

ZLB is zlib from Starfox Adventures

As a side note;

It turns out a lot of Nintendo titles use zlib compression streams for their data formats.

Most importantly, the infamous 'ZLB' format within the Gamecube's "Starfox Adventures" is nothing more than a normal zlib stream. Go download the zlib source and compile your own test to prove this.

If you see zlib compressed data, you generally see something like this:

78 9C - Wii games (sometimes)
58 85 - Starfox

Followed by apparently meaningless bytes. If you read how zlib works, you'll learn more about how that compression format really works, and as always, UTSL (Use The Source Luke)

I still have not cracked the Dragon Quest / Monster Hunter compression format. That one will be tackled next, it seems easier now that I have a decompressed block of data to compare & reverse engineer.

Amazing what you can do with a little perseverance and a lot of intelligence, eh?

Next up is trying to get some locals here to help me make XNA fun stuff. Just for kicks, since there aren't many people around here competent enough to want to make games for fun anymore.

No surprise. Some people actually value social time over working, and some people 'have a life' as they might say.

Peace!

-Z

Technically Competent Flash Game Engine

After playing with flash enough (Flash Professional 8)
I've reached a conclusion about how to build powerful, dynamic flash games.

I've worked with flash off and on for years, C/C++ and OpenGL are much cooler, but it has become difficult to manage my pipeline without additional tools that do not exist; So I am taking a break from that for a bit.

In order to properly make a flash game, you must first understand what a 'video game' is and how it works internally; They are full of interesting technical challenges as well as evil hacks; Because a game tends to be a simulator or emulation of some system, that means we'll never be able to truly make exactly what we want, either due to time or power constraints.

With that in mind, let's build up a 'megaman' example game.

We know from playing the megaman series on the NES that megaman is a little robot dude that can shoot other robots, some of which can move around and shoot back! And, if he enters a boss room, he can shoot a special enemy that can give him additional weapons!

Not much else too the game, really! Let's generalize it.

What kinds of things are there in the game?
-Sprites (2D Pictures)
-Sounds (Sound effects)
-Music (Longer strings of sounds! .nsf or some other mod format)

Well, that's kind of an aesthetic look. The actual game however, isn't quite as simple, as there are a variety of object types, like each type of enemy, each type of bullet, and so on.

When we look at a game from the programmers perspective, we can group all the things that work that same, and build a hierarchy:

Object
-Sprite
-Moving Object
--Bullet
--Controlled Moving Object
---Enemies
---Megaman

So, for each level down we go, we have to add more special code; There still is no magical way to code each type, that will ultimately have to be done. However, we can reduce the amount of work signifigantly by providing generic utilities for moving and updating those objects;

For example, we can have "MoveAsPlatformingCharacter" "MoveAsFlyingThing" "MoveAsBullet".

And, the ironic thing, is the game is still just moving pictures that can add and remove pictures and sometimes play sounds. There isn't anything more to it than that.


So, stepping back a bit, let's look at what flash can do for us:

-Automatic Heirarchy
-Display sprites, images, movies...
-AND can move, rotate and scale them!
-AND can colorize, shade, ect...
-AND can dynamically draw shapes!
-Automatic Sound playing
-Streaming music palying
-Input handling

So flash can do everything we need! But, flash is not assembly code, it is a SCRIPT LANGUAGE. That means that you cannot apply the same logic you use in C to flash; Script languages are not C, and C is not a script language and vice versa.

So in flash terms, we will have 1 movie clip called 'game' that has everything we need in it, as well as having ALL the functions we will use in it, so that you can do this:

game.moveObject( obj, 4, 0 ); //Move object obj right by 4 units

Inside of the game movie clip, we must have all the clips we want to duplicate in real time. Usually this is accomplised by:

game.dup_character;
game.dup_bullet;
ect...

Where, inside each of those dup_character, you have a lot of named frames for that specific character type, like "Heatman", "Iceman", "Megaman", and in each of those frames, you put down the instance of the clip you want to animate; you do not do the animations in the duplicate clip, you make a seperate layer;
This way, you can name each clip inside of the duplicate clip the SAME THING, ergo: "mc", so that the game can reference the mc identically for each instance, AND separate scaling and other properties from it.

[MovieClip: dup_robot]
[FrameName : "Megaman"]
mc [Instance of "Megaman" movie clip, which has all his named animations, like "run", "jump", ect...]
[FrameName : "Heatman"]
mc [Instance of "Heatman" movie clip, which has all HIS animations ]

so your 'game' clip now has this:

[game : Game clip]
dup_robot [Instance of "dup_robot" movie clip, which has all the possible robot types each in a specially named frame]
dup_bullet [Instance of "dup_bullet" movie clip, which has all possible bullet types each in a named frame]
ect...

Using this method, you can simply duplicate a dup_clip, then set it's frame, then once you do that, you can set the mc inside of it's frame, thus instantly allowing your game to be very flexible to add extra things later!

So this is all fine and dandy' Now in flash you can duplicate any clip that graphically represents what your game is, but, what about the actual game?

Here's where it get's difficult; If you trust flash's 1 collision function (hitTest) then you might know that it DOES NOT WORK on and point test that is NOT on the stage (any non-rendered pixel CANNOT be hit-tested against. SO DONT USE IT!)

That means yes, you have to make up your own collision scheme, and make up your own collision math, and your own physics system for motion.

Basically, that means that every object should have a .x, .y, .vx, .vy in it, so you can perform basic rectilinear motion. As for updating your objects, so long as you make a working rectilinear motion function, you can add any complex types of motion you want later.

This will be covered later, but the easiest solution is to add another dup_clip that creates boxes and ramps, which are very basic and mathematically simple collision primitives that you CAN make a game out of quickly.


But all this high level talk isn't much use; Because when you get started, you'll find you need a couple of basic tools:

Something to store this data,
A Map (dict for those python kidz)
A SpaceHashing/Broadphase test

Map is actually simple in flash,

var mymap = new Object();
mymap[ "any key as a string" ] = data;

So you can abuse flash's quick Log2(n) internal string search for object properties as a map. It's as fast as it get's folks.

A Spacehash is up to you, I prefer using a single axis sort and doing a map/array of bins, which have maps of catergories, each which stores an object UID

Yes, every object you make should have a non-repeating Unique ID. Don;t rely on the movieclip as one, you never know when depth levels can change or clips for that matter. Use a Number and just increment it, or make a map and reuse unused keys.

Blargh; What a mess. Too much information to disgorge for ya'll.

I'll post the flash engine later. Then you can just %#@*(& use it!

Happy fourth!

-Z

Pokemon Battle Revolution - h4x

Interesting theoretical question;

Let's say you make a video game with monsters such as Lizardon (Charizard for some);

And you want it to have different textures so you can make different colored versions.

The logical thing to do is to add the textures as references, or make a named texture set so your model can just quickly request a texture set switch, right?

Apparently, someone at nintendo didn't get the memo, and instead, they JUST MAKE AN ENTIRE NEW FILE which has the additional textures in it, ignoring the fact that the entire model set is identical and only the textures change...

For example,

Mr. Lizardon is ~ 800 kB, compressed.

When you decompress him, you get:

lizardon_0 ~ 1024 kB
lizardon_1 ~ 1024kB

So, the funny part, is the textures take up maybe 200 kB of each file.
If they would have just added the textures in, it would have saved nearly 800 kB, and who knows what that translates to in the compressed format!

Weird enough. Also, it explains why some of the colorings I had were dead wrong (is it?)



Oh well. Once I figure out where it tells me how many vertex arrays / vertex objects there are, I'll have it nearly 100% exported.

Peace!

-Z

Pokemon Battle Revolution - h4x

The first thing you probably said was "Pokemon!?? @#$%%(&!?" with more expletives.

On a side note; It was useful to decode the LZSS varient used in these games; It offers a serial form of compression that is very good at packed redundant data, and is byte driven instead of bitwise like a huffman. It seems to get pretty decent compression ratios for sparse or repetitive data; and is general & lossless. Also, learning more block compression schemes for pixel data is useful, PVRTC is the only one I care about at the moment, but STC3 and some others are handy for my own projects. Do they have a normal map compression scheme yet? hm.

On the actual note; I'm not sure if finding out how pokemon are made is useful; But it does show me just how crappy textures can be if you use shaders properly.

And, why would they use vertex keyframe animation? Makes it really really hard to integrate into any other system; Might explain the lack of Digimon/Monster Rancher type games. Oh well. Only theories.

Here's some random pokemon in Blender; No credence given since I never played the games:



-Z

You're fucking right

So I have yet another bad day out of my 27,375 available.

Solution Time.

1. Create my own 3D animation tool that does what I need it to;

2. Open it to the community via Sourceforge;

3. Integrate IGTL::Sys into the mix to improve both components;

4. Create 1 x demo project to showcase how it works in a pipeline.

I've named the project; Once I finish the design specifications it's on to coding it.

Planned:

*Support for full 3D Joint/Matrix based animation (Hardware level matrix joints, with bone display option for seamless transitions from your old tools)

*Support for the newly popular UVS animation (Using bones as 2D billboards that have x,y scale and can change uv coordinates per keyframe; Think Odin Sphere)

*Perfect Data Editor so that all data can be viewed/changed irrespective of present 3D view/mode, as well as locking and visible flags for all pieces of data.

*Keyframe based animation on a discrete timeline; So be sure to start with your base FPS set correctly (default is 50)

*Logical flow to data and animation; No 'global timeline' exists. Everything is animated from 'Strips', and 'Sequences' can be made from and combinations of strips

*Mode driven editing; Object mode for scene objects, Mesh Keyframe mode for mesh objects; Pose keyframe mode for Joint/UVS objects.

*Level Of Difficulty integration; Default system is super simple; Advanced users can tweak and expand editors as needed to suit their work mentality

*Entirely abstracted input with multitouch support; Any Human Input readable by SDL can be used and assigned to hotkeys/event keys. This also means you can record some macros within the program. (Automation tools)


Some obvious problems:

-Exporting will be limited initially to a text format for inter-compatibility; So long as the text format specification is rigid (like OGRE xml) then anyone can easily write a converter from text -> custom.

-Speed and User friendliness; If you do not have OpenGL 2.0 or a newer graphics card, the program will initially deny you the ability to use it. This will be fixed later because it is not a priority to write my own TnL for OpenGL 1.1 users. I have done so before, but this is the lowest priority item.

-Operating Systems; Windows and Mac and Linux don't seem to have multitouch support in the SDL I am using. Too bad; I'll have to store special mouse states for 'virtual mice' aka joysticks.

-Networking; I want this program to allow collaborative editing; This is always difficult to do and not a priority item because it is outside the scope of the first version, plus this would be mostly beneficial for scene editing and individual animations; Think cooperative moviemaking

-Rendering; This program is NOT a rendering tool; It can export frames, but onlt as good as your graphics card can make them. I do not want to write a rendering pipeline; But I should include exporters so you can dump a animation to a real rendering program like Blender. However, this is also a low priority item due to it being outside of the scope.

-Complexity; all good tools have reasons for complexity; Generally it is lazy programmers, but that's because some of the basic problems are very difficult, and they have timelines to meet so they do 'the dumb yet it works' solution. I am doing this for no profit, so there will be slowdowns in the development of this tool.

-3D Mesh Modeling; I do not want to make a mesh modeler; That is blender's job, not this tool. I want this tool to focus on making game animations with an existing mesh, as well as weighting the verticies of the mesh in the program. This means I will need some mesh tools; and that means the first thing people will demand is a mesh editor/generator. This is a low priority and not the scope of the program, though I may add some cool tools for it in the future; especially because re-meshing is common in real industry; which includes re-uv mapping and adding/removing some verticies. This will have to be supported and is a medium priority item.


I plan to have it work a lot like blender. I hope if I build it, indy developers and hobbyist animators can use it for their purposes and avoid the headaches with classical paid program nightmares.

This should take me at least 6 months to get a beta out.

-Z

Extremely Depressed

As stated, I am very sad.

(from http://garfieldminusgarfield.net/)

More than 10 years of programming, multiple jobs, and even an engineering degree later, I'm still not happy.

Here's why:

Blender, being the free opensource 3D wondertool had intrigued me from when I first found it. However, after years of playing, making animations, and games using this tool, now that I entered the realm of 'real' game development, blender is severely lacking in multiple areas.

1. Armatures

The concept of a armature is invalid; The 3D graphics hardware you have and have had since 1970's has always been of the 'projection matrix' * 'modelview matrix' => output raster position. Now, modern 3D hardware has the ability to be programmed, so, people like me can code in fully articulated characters by adding weights per vertex and writing a simple vertex shader that multiplies by each joints matrix.
Blender does not conform to this universal standard; IT instead tries to 'make it easy' by giving you a 'bone', which, here's the serious problem: It has a length. Matrices deform from their center, not an arbitrary point. This makes conversion to my game and from my game to blender impossible, thus, blender cannot be used for the animation pipeline. Any attempt to 'hack' blender into making this work is a waste of time; True, you can constrain your game a lot, but if you had 1/100th the experience I do, you would know better. Now for another point; Even with armatures, blenders animation system is designed for movies; That is, everything works on a global timeline via global IPO keys. No game works like this, so combining run + walk animations becomes very difficult, as well as keeping track of current animation track data. They have botched and fluffed over this for years; No positive results yet.
In conclusion, thanks to a broken bone system and incompatible animation keying system, I now no longer have a animation tool my artists can use for our pipeline.

2. Space conversions

Blender doesn't use math centric +x forward, +y left, +z up space consistently. This causes nothing but headaches for everyone. There is no justification for having inconsistent coordinate systems, pick a coordinate system and make your entire program be consistent.

3. Pipeline

When I make a model in blender, I use my character sketches and some quick concept coloring. Blender makes mesh modeling quick, which is nice. However, when I finish with my model, I want to take the data out, and put it into my game. There are many options for this, but, I usually have to write my own converter. Given, every single update of blender, guess what? My converter breaks somehow thanks to a undocumented python function or change in the way things work. Usually the breaks are not too large, but this is a lot of my time wasted for something that the program should do automatically; For instance, 'dump ascii' should export a large, concisely documented ascii file of all the data for the current selection, including it's linked data and so forth. If they wrote a game engine in blender, why can;t we dump that data out? And why do I keep having to make more converters to spit out a text file?

4. Data Model

Blender uses a older C-Data model. This is a good one to use, however, I would like to have more data model tools; For instance, if everything is reference counted and deleted on zero counts fine; But let me control that and show it to me in the OOPS or a special 'data tree' viewer. As a developer, I need control for that data to better improve the exporting I have to write for this tool. Also, sometimes blender files get junked up with bad chunks from older files. And, more importantly, where is the .blend to ascii converter? That would be very nice to have.

5. Next Gen Content

Blender currently is pathetic when it comes to this; Let's say I want to make a MGS4 snake. No problem you say, and model out a nice 3300 poly Pliskin and then build an armature for him. Now, you can bend and animate him with some ease, though, lookie here, his shoulder bends funny! Well, after about an hour of tweaking the armature, you got it to look better, but not commercial quality. Now you have to generate a mesh keyframe and link it to a python controller that listens to the armature. Okay, fine. But, how do you export that data out of the system? And how do you ever preview your animations if keyframes are global application? Hm, looks like you have a severe problem editing keyframes and armature actions. OH NO, you added visemes so snake could talk; Looks like there's no way to make animations except by manually entering times on the timeline; oh, and look, while he's talking the python controlled armature actuator is fuzzing the keyframes... Looks like you just wasted 8 hours fighting a system that wouldn't work anyways.
Enough bitching about that example; Point is, if you have a animation system, but 1 special component can have 'local' timelines (armature 'actions'), why can't mesh keyframes and other animation systems have 'time strips' that you can make, so that your main animation system can paste strips together? Oh what's that? NLA? only works with armature actions, sorry. Unless you're making a movie with no dynamic content, you're SOL here. And try writing an exporter for that. At least they finally added GLSL to the damn system.



I'm so depressed. What do I do, write my own tool like FrameGL3 (already solved all these problems myself btw; SDS, IK, anims, ect...) or do I just give up? This is a lot of work for anyone to undertake; Only because of the gruntwork required. More importantly, there has to be someone else who has this problem, but where is there solution?


Also, being unable to crack 'Dragon Quest Swords' funky LZSS type compression really has me down. But not down like Valgirt Nedlog has me down; fucker's hard!


Maybe I should give it all up for a while, like, a year or something...


I'm in the wrong fucking state/country/planet...


-Z


As a side note; I've hacked the graphics out of Primal Rage 2; Killer Instinct; Wario World; Super Smash Bros; Turok; and many other games just to learn how they built their data, as my ONLY FORM OF VALIDATION that what I have been doing is correct.