world modified

[KnightErrant]

Ah, excellent Argantonio, I see it! We're looking at the same
data. I can't get there by cutting and pasting, I just don't have
your feel for the data. I read the data using python and accumulate
the different blocks into arrays. But I have to be able to read through
the different .world formats. Right now I can only read the settlements
up to large_cities and castles up to wooden_forts and stone_keep but I
died on large_castle (somehow I missed bringing in castle to work).

The data is a lot like many mesh files glued together with the granny
bytes, the serie. Right now I use the .obj format using the
usemtl command to define a material for Milkshape and then go to
the materials tab to bring in the right texture (converted to .dds)
I read the triangles, the verts, then of the five quad blocks, I ignore
the first two, anims and lighting, (they need to be saved eventually,
I think, but not right now) the third block are the vertex normals so
I use these because the .obj format allows this, the fourth block are
vertex tangents so I throw them away. These can be calculated
from geometry for a back conversion. Caliban said the bitangents are
calculated by the game engine anyway be crossing the normals with
the tangents. The fifth block is the byte-encoded uvs. Now that
I got the upside-down part, (this should have been obvious, Max is
upside-down to Milkshape but I never saw this before in a CA model
so it took awhile to sink in).

Haven't tested beyond this:

village
town
large_town
large_city

(missed city)

wooden_fort
stone_keep

large_castle dies on the second settlement, the vertices are stored
in a strange way here. The above is for northern_european. One would
hope the others at the same level are readable; just haven't checked
because I can only get so many on my memory stick per day.

I think I'm answering the same question but
@ Yelu Dashi
@ Argent Usher
Yes, the uvs are a "block" the way Argantonio described. The blocks
in a "structure", probably "model" is a better word, go this way:
triangles organized in blocks or groups in the Milkshape terminology,
then verts, then five blocks of quad bytes: animations?, lighting?, vertex
normals RGB encoded, vertex tangents RGB encoded, and finally UVs
encoded as high bytes and low bytes. Second byte being the high byte
and 4*v-2; 4*v-2, and v -> 1-v as the last transformation.

I've changed my mind on this, with what Argantonio has shown us, we
can get to ANYTHING! If all we need are some simple conversion
scripts to get uvs, normals, whatever; over to what Milkshape needs
and back conversions to what the .world file wants; I don't see why anything can't be pulled out following Argantonio's methods. My enthusiam
runneth over, a script to do buildings maybe, but a hexedit method for
everything else.

[King Kong]

Though I don't understand much, this is fantastic news.

Thanks Argantanio, Knight Errant and everyone else who is working on this!

[Argantonio]

This works,
KE this image http://www.imaxenes.com/imagen/text11aw278f.jpg.html
is the block 7 of ne-citadel(1) (coords UV) , Can you show us ? the block2 of the mesh with the correct teture, or in his default your ms3d ne-citadel(1) . For I understand high and low byte.

[S-te-Fan]

Does this means I put my landwalls of constantinople ingame? Sounds pretty cool .

[Argantonio]

Here I show you an image http://www.imaxenes.com/imagen/world11by5002.jpg.html
similar to box_lod0.mesh, but it is a file world, ne_stakes_aVacio, http://www.twcenter.net/forums/showp...1&postcount=83
we go to see his blocks, and after will learn who to exchanged.
Orange. two common blocks all files world , in study , but are some type of exhibition of all and each of the parts, objects or subdivisions of the file. It is not important for the moment , no changes must be do now.
(If someone remembers the stone- elevators, and this file observes that the sequences are substituided for zeros this it is the reason of which only see a rock and not see all the original bridge(ne_wooden_bridge64x64)).

Red. In red our first block, the triangles, is like that of the mesh. 12 00 = 18 . size 18x 6 = 108
The same size like block1 mesh with equality of number of triangles, if the exchange the block and "four initial bytes" with the number of triangles , "new model" is strange, but it works.

Blue, second block, the position of the verts , size 15 00 = 21 , 21x12= 252 The same size like block 4 mesh with equality of number of verts repeats. The number of verts 15 00 00 00 it repeats two times (is a particularity) . if we replace this block with "four initial bytes" we have completed conversion world-mesh in the direction that let's let's want.
Try and replacing the blocks 1 y 4 mesh for 1 y 2 world and see the result a box.world in the game.

The following blocks are empty( I have emptied them,for demostred that works) and the number of verts is not the same, but it does not matter now, we concentrate on the last one.
Green . The block 7 , this block came preceded from byte 4 , The coords UV, His size is 2A 00 = 42 . 42 x 4 = 168 .This one is the problem , this blocks is like block 2 file mesh , but they have diferent size , it is the half of small that that of the mesh, ( mesh x8 world x4) They are not interchangeable game crashes , if KE has found the way of exchanging them, resolved topic.
In others blocks are indicated the byte of head-board ( 3 , 8 , 9 ,10) with a yellow circle, your size are equal in both types of files.( are exchanged)

[KnightErrant]

@Argantonio
Sorry, didn't have citadel at work so could not check today. Just
going to do a short post tonight.

@All
The v -> 1-v is only for .obj files. Got the .ms3d export working today
and .ms3d doesn't want the v -> 1-v transform.

Bad news for Milkshape modellers. Was running ne_castle (not even
large_castle) and ran across this bad boy.



This had 475 groups; Milkshape was only designed to do characters, not
architectural behemoths like this, so the upper limit on groups is 255.
This is a wicked pisser, I had put the anim data in quad block 1 into the
first weight, and the lighting data from quad block 2 into the extra int
in the weighting section and it worked. Even if I saved the file the
info was still there. But the castle result says you can't use Milkshape
for any of the big boys. Max can handle any number of groups but the
.obj file format won't let you put the information in via an import (nothing
available in the .obj file format specification for extra vertex info) so we'd
have to make a custom file format with an import/export script to handle
these cases.

I'll clean up some problems with the interface and release the extractor
tomorrow. With more eyeballs on the problem we'll probably run across
some other things like this (the number of verts in that model was
over 58,000. Milkshape only goes up to 65534. Already getting close
to that limit here too.)

[Redmeth]

Would 3dsMAX handle it?

[AnthoniusII]

You guys open new fields in moding art!!!

[Lewagal]

that's right!
(Just think of it Anthonius: tons of hoplites fighting in front of Olympia )

[SigniferOne]

Nice KE and Argantonio...

[KnightErrant]

@Redmeth
3dsMax definately would work, that's how I made the picture.
I thought about the format. Just make a new format with extension
.ms3dbig. It's exactly the same as .ms3d except where words and bytes
are used for number of vertices, faces, and groups one uses 4-byte ints.
Then the import/export script just needs two more buttons, Import ms3dbig
and Export ms3dbig. The changes required for reading and writing .ms3dbig
files would be trivial. And Max has a 100 vertex data channels for storing
extra info on a per vertex basis so storing quad block 1 and 2 data
should be easy too.


@all
I uploaded the settlementextractor.py file and a small material file for the obj format here:

http://www.twcenter.net/forums/downl...o=file&id=2019

Make a directory where you are going to do settlement work and put the
python file, material file, and any settlement or castle .world files you want
to look at in here. Double click on settlementextractor.py in explorer to
run it. You'll get a dialog that looks like this:



The listbox is populated with whatever .world files it sees in the local
directory. The first time you run it on a given world file just select structure 1 and then the
desired .world file. After it runs you will get a small summary file showing you how
many complexes and structures (models) are in the file. You can then rerun it and pick
your structures. The edit box will accept a single number like 1 or 29 or a range
like 11-21. It WON'T work with multiple ranges like "1-5,7,21-24".

To keep things organized, for each world file, it creates a subdirectory
with that world file's name and outputs the obj, ms3d, and summary files in that
subdirectory. It also creates a large .txt file that has all the headers and footers but
no data. You can edit this in Notepad to find the names of the texture files. The textures
come after the structures.

The summary file looks like this for north_european_village_a.world:

There are 31 complexes in this file.

Complex 1 has 2 structures: 1 2 
Complex 2 has 3 structures: 3 4 5 
Complex 3 has 6 structures: 6 7 8 9 10 11 
Complex 4 has 4 structures: 12 13 14 15 
Complex 5 has 1 structures: 16 
Complex 6 has 3 structures: 17 18 19 
Complex 7 has 3 structures: 20 21 22 
Complex 8 has 3 structures: 23 24 25 
Complex 9 has 4 structures: 26 27 28 29 
Complex 10 has 2 structures: 30 31 
Complex 11 has 4 structures: 32 33 34 35 
Complex 12 has 3 structures: 36 37 38 
Complex 13 has 6 structures: 39 40 41 42 43 44 
Complex 14 has 5 structures: 45 46 47 48 49 
Complex 15 has 5 structures: 50 51 52 53 54 
Complex 16 has 5 structures: 55 56 57 58 59 
Complex 17 has 4 structures: 60 61 62 63 
Complex 18 has 2 structures: 64 65 
Complex 19 has 6 structures: 66 67 68 69 70 71 
Complex 20 has 5 structures: 72 73 74 75 76 
Complex 21 has 5 structures: 77 78 79 80 81 
Complex 22 has 4 structures: 82 83 84 85 
Complex 23 has 2 structures: 86 87 
Complex 24 has 1 structures: 88 
Complex 25 has 1 structures: 89 
Complex 26 has 2 structures: 90 91 
Complex 27 has 2 structures: 92 93 
Complex 28 has 2 structures: 94 95 
Complex 29 has 1 structures: 96 
Complex 30 has 1 structures: 97 
Complex 31 has 1 structures: 98

The big .txt file has parts that look like this (look for the red part to see the texture names):

------------------------
Complex 10
------------------------
16 0 [1321] (nstruct =  2) 20 0 [1322]                                      odd seq. number [1339], even seq. number [1340]

Structure Number 30
23 0 [1323] (num_groups =  3)                                                     odd seq. number [1341], even seq. number [1340]

Triangle data section:
27 0 [1324] 0 0 0 0 0 0 0 0 (416 tris) 0 0 0 0 0 26 0 [1325] 27 0 [1324]          odd seq. number [1343], even seq. number [1342]
27 0 [1326] 0 0 0 0 0 0 0 0 (184 tris) 0 0 0 0 0 26 0 [1327] 27 0 [1326]          odd seq. number [1345], even seq. number [1344]
27 0 [1328] 0 0 0 0 0 0 0 0 (300 tris) 0 0 0 0 0 26 0 [1329] 27 0 [1328]          odd seq. number [1347], even seq. number [1346]
vertex block:
22 0 [1330] 23 0 43 5 0 0 {1727} 0 0 0 0 1 0 0 0 35 0 [1331] 0 0 0 0 
                    (1727 verts)  0 0 0 0 34 0 [1332] 35 0 [1331] 5 0 0 0         odd seq. number [1349], even seq. number [1350]
quad byte blocks:
40 0 [1333]  8 0 0 0 (1727 quads) 0 0 0 0 39 0 [1334] 40 0[1333]                  odd seq. number [1351], even seq. number [1352]
40 0 [1335]  9 0 0 0 (1727 quads) 0 0 0 0 39 0 [1336] 40 0[1335]                  odd seq. number [1353], even seq. number [1354]
40 0 [1337]  3 0 0 0 (1727 quads) 0 0 0 0 39 0 [1338] 40 0[1337]                  odd seq. number [1355], even seq. number [1356]
40 0 [1339] 10 0 0 0 (1727 quads) 0 0 0 0 39 0 [1340] 40 0[1339]                  odd seq. number [1357], even seq. number [1358]
40 0 [1341]  4 0 0 0 (1727 quads) 0 0 0 0 39 0 [1342] 40 0[1341]                  odd seq. number [1359], even seq. number [1360]

bounding sphere:
0 0 0 0 -103.40617 +1.06102 +13.06679 +33.55326 0 0 0 0 19 0 [1343] 20 0 {1322} 20 0 [1344] odd seq. number [1361], even seq. number [1362]

Structure Number 31
23 0 [1345] (num_groups =  9)                                                     odd seq. number [1363], even seq. number [1362]

Triangle data section:
27 0 [1346] 0 0 0 0 0 0 0 0 ( 18 tris) 0 0 0 0 0 26 0 [1347] 27 0 [1346]          odd seq. number [1365], even seq. number [1364]
27 0 [1348] 0 0 0 0 0 0 0 0 ( 26 tris) 0 0 0 0 0 26 0 [1349] 27 0 [1348]          odd seq. number [1367], even seq. number [1366]
27 0 [1350] 0 0 0 0 0 0 0 0 ( 12 tris) 0 0 0 0 0 26 0 [1351] 27 0 [1350]          odd seq. number [1369], even seq. number [1368]
27 0 [1352] 0 0 0 0 0 0 0 0 ( 12 tris) 0 0 0 0 0 26 0 [1353] 27 0 [1352]          odd seq. number [1371], even seq. number [1370]
27 0 [1354] 0 0 0 0 0 0 0 0 ( 44 tris) 0 0 0 0 0 26 0 [1355] 27 0 [1354]          odd seq. number [1373], even seq. number [1372]
27 0 [1356] 0 0 0 0 0 0 0 0 ( 12 tris) 0 0 0 0 0 26 0 [1357] 27 0 [1356]          odd seq. number [1375], even seq. number [1374]
27 0 [1358] 0 0 0 0 0 0 0 0 ( 12 tris) 0 0 0 0 0 26 0 [1359] 27 0 [1358]          odd seq. number [1377], even seq. number [1376]
27 0 [1360] 0 0 0 0 0 0 0 0 ( 12 tris) 0 0 0 0 0 26 0 [1361] 27 0 [1360]          odd seq. number [1379], even seq. number [1378]
27 0 [1362] 0 0 0 0 0 0 0 0 ( 60 tris) 0 0 0 0 0 26 0 [1363] 27 0 [1362]          odd seq. number [1381], even seq. number [1380]
vertex block:
22 0 [1364] 23 0 65 5 0 0 { 333} 0 0 0 0 1 0 0 0 35 0 [1365] 0 0 0 0 
                    ( 333 verts)  0 0 0 0 34 0 [1366] 35 0 [1365] 5 0 0 0         odd seq. number [1383], even seq. number [1384]
quad byte blocks:
40 0 [1367]  8 0 0 0 ( 333 quads) 0 0 0 0 39 0 [1368] 40 0[1367]                  odd seq. number [1385], even seq. number [1386]
40 0 [1369]  9 0 0 0 ( 333 quads) 0 0 0 0 39 0 [1370] 40 0[1369]                  odd seq. number [1387], even seq. number [1388]
40 0 [1371]  3 0 0 0 ( 333 quads) 0 0 0 0 39 0 [1372] 40 0[1371]                  odd seq. number [1389], even seq. number [1390]
40 0 [1373] 10 0 0 0 ( 333 quads) 0 0 0 0 39 0 [1374] 40 0[1373]                  odd seq. number [1391], even seq. number [1392]
40 0 [1375]  4 0 0 0 ( 333 quads) 0 0 0 0 39 0 [1376] 40 0[1375]                  odd seq. number [1393], even seq. number [1394]

bounding sphere:
0 0 0 0 -94.23789 +0.01352 +13.91741 +28.65560 0 0 0 0 19 0 [1377] 20 0 {1344} 2 0 0 0 odd seq. number [1395], even seq. number [1396]

About to read texture or bounding sphere header: tell says 1815580
Number of bytes from getheadersize = 18
header: 49 0 [  82] 48 0 [  83] 49 0 [1378]                                       odd seq. number [1397], even seq. number [1398]
18  buildinglitterlod0
17
40  BlockSet/Textures/NE_litter_pass.texture
47  BlockSet/Textures/NE_litter_pass_normal.texture

{====>>> 428 bytes of the standard big granny string omitted here <<<====}

About to read texture or bounding sphere header: tell says 1816147
Number of bytes from getheadersize = 12
Bounding sphere:
48 0 [1379] 49 0 [1378]   -100.72674 +0.00183 +15.21055 +34.82426
15 0 [1380] 16 0 {1321} { 315} { 326} 1 0 0 0 2 0 0 0                             odd seq. number [1399], even seq. number [1400]

For 2 structures:
Preceding integers:  716  208   16 
[34] Integers: (plus the final 1)
 109   27   92   93   94   95   96   97   98   99  100  101  102  103  104  105 
 106  107  108  163  164  165  166  167  168  169  170  171  173  174  175  176 
 177  178    1

So far the script fails on:

ne_huge_city_a
ne_large_castle_a

but at least it fails in the same way; something non-standard in the header
of the verts block and the way the verts are organized. Presumably it will
fail on the corresponding non-Northern European files as well. Don't have
fortress, or large_city, or whatever else I may have forgotten, to test on.

ne_city_a has 11 structures too large: 2, 12, 32, 50, 55, 67, 88, 93, 100, 105, and 107
ne_stone_keep_a has 1 structure too large: 38
ne_castle_a has 4 structures too large: 24, 41, 66, 69

[Argantonio]

Spectacular, It works and very speed,


You are a "crack" I was sure that you ,do it .The method of blocks is to slow and not extract coords uv. I remain forced in the problem of the coords uv , but your settlementextractor , opens infinity of possibilities.

[Argent Usher]

KE and Argantonio: Good Job and i hope your heads will bear the heavy weight of the crowns.

[Argantonio]

Using your settlementextractor , I can see in the file mesh (wooden_stakes_b) the block 2 that corresponds to the coords uv, http://www.imaxenes.com/imagen/mesh1sm98j8.jpg.html
if I compare it with the original one of wooden_stakes_b, http://www.imaxenes.com/imagen/world1dw83s8.jpg.html
they are different, we knows it , the mesh block 2 is double of big in relation with the world block 7 , ¿Can you interpret the differences? The coords are the same but the codification is different ¿ Who can exchange or convert one in other?
I think that resolved , this problem , Let us to introduce new objects , buildings , castles... but with the correct texture.

[KnightErrant]

Thanks to you both.
Well, now we can take the watch apart.
Let's see if we can put it back together.
I'll start on a script to read from a selected
.world file in the base directory and write a
new one in the subdirectory substituting the
data from any .ms3d files it finds there. After
the last .ms3d file it finds, it will just read and write
all remaining bytes.

Edit: Just saw your 2nd post Argantonio. I didn't try it on
non-settlement or non-castle files. So the original one has twice
as many uvs as verts? I would have thought that would kill the
script.

[Argantonio]

I do not have words...
The solaristica has done a science , and the extractor became a converter.
Nice day is today ( In Spain is Night).

[KnightErrant]

Hi Argantonio,

I went and looked at the wooden_stakes.world file. My apologies, I did not
realize there were so many .world files or what you were doing. I was
so locked-in to working with settlements and castles I did not appreciate
the scope of your work. I ran my script on it and got out files even though
the script died. This does raise the question, what is the right thing to work
on next? The small .world files can probably be understood and parsed to
the end. That might help understand the settlements and castles.

I expected more downloads and involvement in this question, but perhaps
we are destined to explore this ourselves. Do you have a feeling for the
next step? I can refine the extraction script to work with the small .world
files and maybe learn from that. This pushes back the general solution
for cities and castles.

I'm not sure if this is correct but I'll raise these two objections or possible
constraints on modelling structures here.

(1) As I see it, you can't add in new complexes or structures into the
settlement and castle .world files. There's just too much other stuff we
don't understand in the tables at the top or at the bottom.

(2) Even more restrictive to modellers, you can add in all the verts and
triangles you want. You just can't change the number of groups. This
would change the even and odd serial numbers in the bottom half of
the world file. I don't know how to read this part of the file. I was just
planning to copy it once the last changed .ms3d file was processed.

Do you agree? With your experiments, would extra triangle groups
that changed the block counting in the even and odd serial numbers
cause the game to crash. These are the sort of questions I'm wondering
about when making a repacker.

Regards,

KE

[AnthoniusII]

Keep on guys...moders counting on you...
Finish it ...make a tutorial...win the golden price and make yoyr selvs famous!

[Basil 11]

Keep up the good work!!!!!!!!!
This will be amazing!

[Shigawire]

You two are gods! And you should be deified for this. You deserve it more than any milksop egotistical forum-politician at TWC. But I suppose, you two are "Knights" and are too chivalrous to care about such things. You will probably be more satisfied and honored by all the great mods that can be made thanks to your work.

When you two are done with this, you will be opening a field of modding that wasn't even available in RTW... that is: editing settlements!

+rep! but I can't rep you any more right now