Release of the CUF (font & text rendering) mod tool

[Tellos Athenaios]

The Europa Barbarorum Team hereby releases the results of its efforts to open up the CUF file format for modding. The “CUF” project provides modders with tools to create their own CUF fonts as well as incorporate basic support for “rich text” in their mods. Originally developed for manipulating the CUF fonts bundled with these games, the “CUF” project has progressed so far we believe that others would want to use this tool for their own purposes as well. A screenshot or two may help to explain why we feel that such confidence is justified:



The tool allows you to convert TrueType or Type1 fonts to a CUF file; as well as make more advanced use of the strings.bin format in tandem with the CUF format to manipulate the way descriptions are rendered and produce multiple font styles alongside each other in the same “body” of text. The screenshot above shows how that might look.

The project can be downloaded from the TWC download service right here: http://www.twcenter.net/forums/downl...o=file&id=2856. The download consists of a single compressed 7-zip archive containing:

• Java source code (requires Java 1.6 JDK to build) and resources
• A COPYRIGHT file containing the licence under which the project is made available (see below)
• A readme.html file containing instructions for building from source, running it and using it to get some modding done. The file also contains links to download additional required software: it is very much recommended that you read this file.
• Ant build scripts to make building from source easy (requires Ant 1.7.1 or higher to build)
• A POSIX sh script which you can use as example for creating your own scripts that run the program under specific settings. Note that it is not required to use such a script; also note that it is not a batch file.

Programs such as 7-zip can be used to extract these contents. (You can get 7-zip from here: http://www.7-zip.org/ or commandline equivalent p7zip from here: http://p7zip.sourceforge.net/ )

People who do not want to build from source can download the following compressed 7-zip archive instead: http://www.twcenter.net/forums/downl...o=file&id=2857 (524KB). This version contains:

• The tool in the form of a single JAR file (dist/CUF.jar)
• The help bundle (dist/cuf-help.jar). JavaHelp is required to use it, you can get JavaHelp from here: https://javahelp.dev.java.net/
• A COPYRIGHT file containing the licence under which the project is made available (see below)
• A readme.html file containing instructions for building from source, running it and using it to get some modding done. The file also contains links to download additional required software: it is very much recommended that you read this file.
• A sample lib/nblibraries.properties file you can use to configure the tool to link against additional libraries (JAR files) at start up. This may be necessary for using the Help feature (depending on how you install JavaHelp).
• Code documentation for use as reference material when building more complex scripts to automate the tool. (dist/help/javadoc)

Both versions (source and binary only) are made available under the following licence:

Copyright (c) 2010, The Europa Barbarorum Team
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
* Neither the name of The Europa Barbarorum Team nor the
names of its contributors may be used to endorse or promote products
derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL The Europa Barbarorum Team BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Our knowledge of the CUF file format is by no means infallible; as is evident from the screenshot marking most CUF properties as `purpose unknown'. This obviously reflects on how the end result looks compared to a preview generated by the tool: while the preview is roughly accurate, it is necessarily a simplified render and thus does not reflect the precise rendering method of the game engine. Consequentially precise text alignment is at the moment somewhat hit & miss and best results come from using font sizes that correspond with the fonts used by CA.[*] Further research in the details of CUF font rendering as it is performed by the M2TW engine will be required to make the results more robust in this regard.

Comments, criticism, improvements and bug reports are welcome. Regards,

The Europa Barbarorum Team.

* Note that alignment does not refer to how individual glyphs align with each other; rather it refers to how a line of text aligns to its bounding box (e.g. how a label aligns on a button).


A manual is available in PDF form at TWCenter's download section.

[WImPyTjeH]

Excellent tool! I'm so tired of those modern blue letters...

[Augustus Lucifer]

Very cool, I should check this area more often.

DLed the file, subscribed to thread, and will hopefully be able to use the tool and offer some feedback.

[Tellos Athenaios]

Excellent tool! I'm so tired of those modern blue letters...

Thanks! Note the colour of the font is not controlled in the CUF file itself; rater there are a few data/*_db (plain text) files which do this. (The screenshots you see were all rendered using exactly one same CUF file).

[Tellos Athenaios]

Updated the main post to use the TWC downloads section instead of the EB FTP server.

[Gigantus]

I am bit daft when it comes to fancy programmer's language. Underneath what happens when I double click the jar file:

Starting the CUF shell. Omitting the preload phase (there is no interpreter available for use yet) for text/javascript. Warning at: Shell.preload: no preload script for: ecmascript. File does not exist: C:\Users\myusername\.cuf\preload\ecmascript\text/javascript. Welcome to the CUF shell. Your current interpreter language is: ECMAScript, version 1.6. (Name of the interpreter: Mozilla Rhino [version: 1.6 release 2], extension js, MIME application/javascript). To exit, type shell.exit() or shell.exit(<code>) if you wish to return a specific exit code. To change interpreter language type shell.load(<arg>), where <arg> is a MIME type (e.g. text/javascript), a file extension (e.g. py), a language (e.g. sleep) or the name of an interpreter (e.g. jruby). Note that an engine may be available under numerous aliases, extensions, or MIME types. You can always get a handle on your interpreter through shell.getEngine() and use shell.getFactory() to query for more details. [0]: Thomas Kuck@C:\Users\myusername\Downloads>

I downloaded the other file as well (the project) and the archive contains a single file named CUF (font & text rendering) mod tool (source distr without extension.

Somebody enlighten me a bit,please, I really would like to give it a spin!

[Tellos Athenaios]

This is normal behaviour: the program gives you a prompt at which you type commands. The warning you received is innocent as you have not set up preload scripts. What preload scripts are (and how to use the program) is explained in the readme.html file.

If you are running it in a GUI there should be an option under Help which reads "README". It pops up a window which renders the readme.html file. It is not a fully functional render because the support for rendering HTML and CSS in Java is limited to HTML 3.2 or so (hey, it was written in the mid 90's...) ... and hyperlinks don't work.

Alternatively if you want a more fully functional readme, you can extract this file from the JAR archive (simply open it in a tool capable of handling ZIP files). Then open the readme.html file in a browsers and make yourself a cup of tea or something because it's a fairly big read (contains plenty of sample script snippets). ~

For a nice starter try a preview of one of M2TW's own fonts (this assumes you have unpacked your packs with CA's unpacker tool, so you will have an unpacked version of the CUF files in <M2TW>\data\fonts\ ):

importPackage(org.europabarbarorum.cuf.shell);
/* adjust path as necessary, backslashes need to be escaped */
var font="C:\\path\\to\\M2TW\\data\\fonts\\kpl_60.cuf";
var file=shell.pathParser().parsePath(font);
var tk= new FontToolkit();
tk.snapshot(tk.fromFile(file), "test preview title").preview();

You could save that as something like: font.js where you downloaded the JAR file (e.g.: "C:\Users\myusername\Downloads\font.js"), then run that script from the prompt by issuing the following command:

shell.interpret("font.js");

And it will pop up a preview window as show in the screenshots where you can preview the contents of a font.

[SigniferOne]

Nice job.

Is it possible to change the color of the texts in various windows, for example the color of the text descriptions in the image below?

[Tellos Athenaios]

To an extent, see one of the screenshots for canary yellow text.

But that is not the job of the tool because colours are controlled by a plain text _db file (colors_db or something like that). Search your unpacked <M2TW>\data\ folder for it (as well as fonts_db or something similar which controls the files of the fonts used for the various fonts in the game). The colours file specifies a number of categories and I have not checked if all individual `UI' elements are accounted for so it might be that some UI elements must share the same text colour.

The CUF file format uses alpha bitmaps to store a glyph; the rendering engine in the game simply combines these with 8bit colour components (red, green and blue) from the colours file and this is what you see as the result in the game. The tool itself is unaware of the colours (they are not at all relevant to its job).

[Gorrrrrn]

I look forward to using this tool.

correction - I look forward to having a version of this tool I can use.
(far too complicated at moment.)

word to the wise: there are a vast number of appalling bad fonts out there!
before using any fonts check that EVERY letter or frequent combination of letters is legible in both upper case and lower case.
You might also want to check for the availability of "special" characters - accents etc.

the existing default font is dull but does its job well.
Use fancier fonts at your peril.

(I'm going to see if I can get kelmscott to work.)

[Tellos Athenaios]

Pushed out an update in the TWC tools section (since the links in post #1 point to those, these are automatically updated as well).

A summary of the changes:

New Features

• Use escape characters as a more robust alternative to source indentation or newlines which means less risk of damage by braindead text-editors and other ills. Also allows you to type in multi-line preview texts.
• Tab base text alignment and text flow calculations. The tool is now able to generate Strings files which horizontally align pieces of text on individual lines.
• FontStyle: removes the need to know about lower level Java packages in order to be able to work with font attributes (e.g. shape width and stroke weight, underline and strikethrough)
• Support listening for compile events so it becomes easier to write the glue to manage scripts run through fork.
• Made the Strings compiler a more modular structure so it becomes possible to compile Strings from other things than XML files.
• Added a few different StringSources, the most notable of which is the EditorSource which allows you edit an existing Strings file.
• (Improved) support for order strings files. Previously the only format that could be read & written was something I call “keyed”: it stores string keys and values, and it is used for most descriptions (e.g. export_units.strings.bin). However some files use a slightly different Strings file format, which omits the keys. The CUF program/libraries now support this variation, too.

Bugfixes

• Fixed a fromFile() method which returned null when loading system fonts.
• Fixed the same fromFile() method to strip the font family attribute from the delta map it uses. This is required for correctly handling font files that are not installed in the system environment.
• Report script file names correctly when an error occurs.
• Fixed some outdated Javadoc, added some more for previously undocumented elements.
• Fixed misleading error message from SAXHandler. This bug would not ordinarily manifest itself in the previous version of the tool because the conditions for it to occur were not yet supported anyway... However with the introduction of tab support/layout you can now make additional errors which highlighted this bug during testing.

Misc changes

• Changed structure of strings package. There is now a strings.impl package as a result of making the Strings compiler more modular.
• Moved some parsing code around to enable better re-use, and improve modularity.
• Changed CUFSource interface so it no longer requires you to work with ChartableEntry objects, if you don't want to. (Of course the built-in implementations still do.)
• Enabled cuf.sh script to work when invoked from arbitrary locations.
• FontToolkit/Shell/StringsToolkit etc. use string pathnames rather than java.io.File objects. The pathnames still expand as if processed with shell.pathParser().parsePath().
• Some restructuring/fat-removal in the internals of the shell package. This should not affect existing scripts because existing scripts have about zero use for those internals anyway!
• Changed how compilation is done. The StringsToolkit/FontToolkit now return an object which encapsulates a compilation task, rather than scheduling this task directly. This is required to take advantage of the ability to listen for compile events.
• Removed FontSnapshot, it no longer had a purpose.
• Updated readme.html file.

... and anything I might have forgotten. Also a screenshot that showcases tab support:

[Tellos Athenaios]

correction - I look forward to having a version of this tool I can use.
(far too complicated at moment.)

What is too complicated about it? (What version are you using, btw: the source distribution or the binary only?) Compiling? Running? Reading/using the readme? Other?

I can't help you or improve the tool if I don't know what the problem is.

[Tellos Athenaios]

Please explain, I would like to do that job for the German 'unconvertable' string.bin files.

The tool contains a library which deals with the .strings.bin file format. It allows you to write both “tagged” (as alpaca calls it: I call it “keyed” because the file defines a mapping between key and data records) and “untagged” (I call it ordered because the only way to make sense of the individual strings records is by their order in the file).

For instance one can trivially build the converter for tagged files in JavaScript like this:

// read a tagged strings file
var stringsFile=new java.io.File("/path/to/input/file");
var reader=new org.europabarbarorum.cuf.strings.StringsReader(stringsFile);

// check if the file can be read
if(reader.isPrepared()) {
  // get an iterator so we can do a python-like for x in file() construct
  var iterator=reader.iterator();
  // open a file for writing 
  var writer=shell.pathParser().writer("/path/to/output/file");
  while(iterator.hasNext()) {
     // read the next strings key+data record
     var record=iterator.next();
     var key=record.key();
     var value=record.value();

     // write a line of text in the converted format
     writer.println("{" + key+ "}" +value);
  }
  writer.close();
  shell.out("done!");
}
else {
  shell.error("unable to read: "+stringsFile);
}

And converting untagged files requires only a little more work: you need to supply list of keys to produce the format. The easiest way to do that is by using a KeyList which generates keys as needed:

// read an untagged strings file
importPackage(org.europabarbarorum.cuf.strings);
var stringsFile=new java.io.File("/path/to/input/file");
var keylist = new StringsReader.KeyList();
var reader=new StringsReader(stringsFile, keylist);

// rest of the code same as in above example

[Gigantus]

I am completely thumbs when it comes to using something like this. Could you give a step by step guidance? Like from after installing your tool? (Link please)

[Tellos Athenaios]

Tool (CUF) is in the Miscellaneous section of the Tools, Tutorials etc. forum; it contains screenshots to wet your appetite, too.

A quick step by step would be:
(a) Make sure you have Java 1.6 (Standard Edition) installed on your system.
(b) Download the version of choice (source, which you can compile and build documentation from; or binary only which is just the compiled program + libs in a single JAR file)
(c) Read the readme file. If you download the binary only you will need to extract it from the JAR file using some tool capable of opening ZIP files (JARs are a kind of ZIP files). You say you are no great shakes at this kind of thing, so it will probably not illuminate much more -- except that it contains more snippets you can try and it explains a thing or two you will want to know for more advanced use.
(b) Run the program (if you build from source you will need to compile it first). This is just to check that things work as they should.
(c) Copy the example to notepad or some other editor of your choice, edit the /path/to/file paths to point to the right files and save it somewhere. (E.g.: C:\Examples\converter.js)
(d) Run the program again if you closed it at (b), and execute the script like this: shell.interpret("C:\\Examples\\converter.js"); (the double backslashes are required because you are typing a bit of JavaScript code).

The script should now either run and print the done message when done; or blow up if an error occurs.

Also since you are a moderator and this is getting a bit off-topic for alpaca's thread, perhaps it is a good idea if you could please split off the last few posts and append them to my CUF tool thread?

[Gigantus]

Three posts moved

[Gigantus]

I just discovered my main problem: after downloading the executable and running it, I can't type anything in the shell!
I checked my java updater and I have this version: (build 1.6.0_20-b02) which it claims to be the latest.

What could be the problem?

[Tellos Athenaios]

I am not able to re-create that program behaviour with the same Java version on either Windows 7 or Kubuntu 10.10. Are you sure you tried to type in the single-line text-field below the multi-line, scroll area which displays only program output? (See the screenshots in post #1 for an example.)

I have (both on Windows 7 and Kubuntu 10.10):

java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

As well as (on Kubuntu 10.10 only):

java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9pre) (6b20~pre1-0ubuntu2)
OpenJDK 64-Bit Server VM (build 17.0-b16, mixed mode)

[Gigantus]

Mine looks like the first pic on the left, that is the one I get when click on the 85888-CUF.jar file. I can do what I want, but I don't get a cursor. Nor a response if I click anywhere on the window.

[Tellos Athenaios]

That's odd?

Try running & posting output of the following batch file?
Save the following code as a .bat (in the same directory where you have the 85888-CUF.jar file, also check the file name at the end of the command):

java -Dcuf.shell.chrome=yes -jar 85888-CUF.jar
pause

If that doesn't work you can try disabling the GUI like so (you'll have to type on the batch prompt so it'll be a little less comfortable as things like backspaces won't work):

java -Dcuf.shell.chrome=no -jar 85888-CUF.jar
pause