[GUIDE/DOC] EA/make/decomp wizardry

Tutorials
, , , ,
1

EA/make is something I’ve been messing with for a while now, but recently I had the idea of including the decomp projects into the mix! In this guide-ish thread, we are going to attempt to do the following:

  • Setup a EA/make hacking project
  • Setup a decompilation project for use as reference for wizardry in that make/EA hacking project

I’m going to mostly assume you have read through my guide to doing C wizardry (with EA). Here’s a quick rundown on what you should have learned from it:

  • Install DevkitARM
  • Build C or ASM for the GBA using makefiles
  • Use lyn to convert compiled objects to EA events
  • Use lyn reference objects (which I refer in the C+EA guide as “libraries”)

I’m also going to assume you are familiar with EA buildfiles and how they work.

Tools and resources used in this:

I’m going to be doing this on FE6 (not FE8!), but this should be applicable to really any GBA game for which there is a decompilation project (even if it is nowhere near completed).

Setting up our basic project

This is not something that is particularly hard to do for seasoned buildfile users, but I’m going through it anyway just so we are on the same page.

We are going to be starting from a fresh, empty directory. Mine is going to be ~/Documents/code/fegba/fe6-hax, but of course mostly anything will be fine.

The first thing we are going to do is initialize a git repository inside that directory. Use you favorite client or just use the command line interface like this:

git init

Using git (or really any version control system) is generally good practice, but in this case we will want it because of how we will integrate the decompilation project in our hacking project later.

I will now make a Tools subfolder to put all of the tools and otherwise references for our project.

The first tool we will want to include will be, of course, Event Assembler: I will dump the contents of the event assembler distribution into a Tools/EventAssembler folder. Note the lack of spaces between Event and Assembler: that’s important (make doesn’t like spaces). I will also put ColorzCore in that folder.

Note: you may want to add the Tools/EventAssembler folder to a .gitignore, otherwise you will “pollute” your git repository with Event Assembler files, as they are not sources from your project. In general it is considered good practice to gitignore any kind of binary from your git repository (thus keeping only the source).

Now that’s done, let’s make our main buildfile. Traditionally, it was called ROM Buildfile.event, but make doesn’t like spaces, so I’m going to call it just Main.event. Let’s fill it with some basic junk:

ORG 0x5C4A34 + 8*5 // ProcScr_GameController + 8*5
    WORD 0x0004000C 0 // PROC_GOTO(4)

This will make it skip the FE6 intro movie, this to have a quick and easy way of knowing our thing works.

We now want a clean FE6 ROM, which I will put into our folder under the name fe6.gba.

Note: .gitignore your ROMs! If you do end up pushing your git repository to something akin to GitHub, you do not want to accidentally distribute ROMs, even if it was long ago in your commit history. The easy want to do this is to add a *.gba line in your .gitignore.

Finally, we are going to be writing a quick, temporary “MAKE_HACK.cmd” to get us started. It will quickly get replaced by make shenanigans in the next section but it will do for now.

copy "fe6.gba" "hack.gba"
Tools/EventAssembler/ColorzCore.exe A FE6 "-input:Main.event" "-output:hack.gba"

If we run “MAKE_HACK.cmd” we should get a hack.gba ROM which, when run, will get us directly to the FE6 title screen (skipping the intro movie, as planned).

We could git commit our progress here if we want to.

Enter make

To replace “MAKE_HACK.cmd” with make, we are going to need to create a file named Makefile, and put it at the root of our project folder.

The most basic makefile we can write would be one that defines a target to build the hack.gba ROM from the clean rom and the main event, using the same commands from “MAKE_HACK.cmd”:

hack.gba: fe6.gba Main.event
    cp -f fe6.gba hack.gba
    Tools/EventAssembler/ColorzCore.exe A FE6 -input:Main.event -output:hack.gba

Note: When copy-pasting makefile snippets, make sure to replace the 4-space indentations with tabulations! That is important as make detects those to identify recipes.

Note: make doesn’t use Windows cmd (batch) for recipes. Instead it uses a variant of unix sh, which is why we replaced the Windows copy with the standard unix cp -f.

Now if we open a unix shell in our folder and run make, we should have the same hack.gba as before.

Before we get any further, we are going to clean this up. While this works (under Windows), there are some things we can improve upon.

filename variables

First, it is generally considered unsound to put filenames in verbatim, especially when they show up multiple times. We want instead to be using make variables. So let’s do this:

EA_DIR := Tools/EventAssembler
EA := $(EA_DIR)/ColorzCore.exe

FE6_GBA := fe6.gba
HACK_GBA := hack.gba
MAIN_EVENT := Main.event

$(HACK_GBA): $(FE6_GBA) $(MAIN_EVENT)
    cp -f $(FE6_GBA) $(HACK_GBA)
    $(EA) A FE6 -input:$(MAIN_EVENT) -output:$(HACK_GBA)

Note that I have made an intermediate EA_DIR variable. This doesn’t look too useful now but it will be when we will be defining variables for EA tools later.

Remove hack.gba on error

One of the issues we have right now is that hack.gba will still be updated even in case of EA errors (leaving us with a hack.gba that is identical to the vanilla ROM).

We can fix that simply by appending the following to the EA invocation like:

<...> || rm -f $(HACK_GBA)

This will remove hack.gba in case of an EA error.

non-Windows supports

This is for the non-Windows users out there… like me! Indeed the above makefile only works fine on windows. That is because of the ColorzCore.exe invocation: under Windows, it can be run directly, but on unixes, it needs to be run through either wine or mono (mono as it is a special .NET exe. regular exes won’t work with mono).

To fix this, we are going to replace the EA variable definitions with this:

ifeq ($(OS),Windows_NT)
  MONO :=
  EXE  := .exe
else
  MONO := mono
  EXE  :=
endif

EA_DIR := Tools/EventAssembler
EA := $(MONO) $(EA_DIR)/ColorzCore.exe

Note: don’t worry about the EXE definition just yet. That’ll be useful later.

That way, on Windows, the EA variable will expand to Tools/EventAssembler/ColorzCore.exe while on non-Windows, it will expand to mono Tools/EventAssembler/ColorzCore.exe. This allows the invocation of EA later on to work on all systems!

EA/make event dependencies

If you try to re-run make after running it once, it will output something like this:

make: 'hack.gba' is up to date.

Make checks for a target to be up to date by comparing how old the file is, and how old its dependencies are. In this case, the file is hack.gba and its dependencies are Main.event and fe6.gba.

So here this makes sense, as neither of the dependencies were edited. If you edit Main.event in any way and then re-run make, it will rebuild the ROM as expected.

However, if you #include a file in Main.event (say, Constants.event), re-run make, edit that new file, and then re-run make, you will get:

make: 'hack.gba' is up to date.

Uh-oh, we now have a problem, as we do want the ROM to be rebuilt (since we modified a file that is used in its build process), but make doesn’t detect that it needs to be updated, so it does nothing.

We can work around that issue by forcing the target to remade using either make -B, or by making the target phony, but both of these options are suboptimal, and we’ll see later that it won’t work as desired either.

The only solution that will work properly is to make the hack.gba target depend on all the files its main event includes and any files those include as well.

Ideally, what we would do, is scan the event files for includes and add all of resulting files to the target’s dependencies. And you know what? I wrote a tool to do just that!

It’s called ea-dep, and it will scan for #includes and #incbins from an event file, and print the list of files it and its includees include.

Let’s add ea-dep to our Tools folder (this we have Tools/ea-dep). And add a variable to our Makefile to define its name and location:

EA_DEP := Tools/ea-dep$(EXE)

Note: this is where the EXE starts seeing use. Indeed, under Windows, executable files all end with “.exe”, while on other systems they typically do not have extensions. We defined EXE to only be “.exe” on Windows earlier.

We now will run it from our makefile and put the result into a variable

MAIN_DEPENDS := $(shell $(EA_DEP) $(MAIN_EVENT) -I $(EA_DIR) --add-missings)

I added $(EA_DIR) as an include path as we want ea-dep to detect standard includes (such as EAstdlib.event), and I added --add-missings as it will allow us to make rules for generating missing files later on (this is very, very useful! and will serve as a replacement for the traditional #incext/#inctext directives).

Now that we have all our included files into a variable, we only need to add them as dependencies to our main target!

$(HACK_GBA): $(FE6_GBA) $(MAIN_EVENT) $(MAIN_DEPENDS)

Aand we’re done! This should fix our dependency issue from earlier.

We now have the basic backbone of a EA/make project. Anything more will be a matter of adding additional targets and pattern rules for building “missing files” included in our buildfile.

C Wizardry

This is where we bridge the gap between the make shenanigans from the C+EA guide and the make shenanigans from here. I’ll go a bit more quick here as I explained most of what I’m doing in that guide.

First, I’ll go ahead and make a quick, temporary “library” for lyn (from now on, I will refer to it as the “reference”), and call it fe6-reference.s:

    .global GetGameTime
    .type   GetGameTime, function
    .set    GetGameTime, 0x08000EED

It will serve us for the example, but we’ll replace it later with decomp shenanigans.

Next, we need to add some makefile rules. I’ll make a new makefile “segment” named wizadry.mk and include it in our main makefile:

include wizardry.mk

I’ll put lyn into Tools/EventAssembler/Tools and define a variable accordingly. I’ll also add a CACHE_DIR variable to the main makefile which will define which directory to put temporary files in. Any name that starts with a dot will do (I use .cache_dir).

CACHE_DIR := .cache_dir
$(shell mkdir -p $(CACHE_DIR) > /dev/null)

I’ll just copy most of my wizardry makefile from FE-CHAX (see wizardry.mk in FE-CHAX). If you followed the C+EA guide it should mostly make sense to you (if you have any questions feel free to ask).

wizardry.mk 
# ====================
# = TOOL DEFINITIONS =
# ====================

# making sure devkitARM exists and is set up
ifeq ($(strip $(DEVKITARM)),)
  $(error "Please set DEVKITARM in your environment. export DEVKITARM=<path to>devkitARM")
endif

# including devkitARM tool definitions
include $(DEVKITARM)/base_tools

LYN := $(EA_DIR)/Tools/lyn$(EXE)

# ==================
# = OBJECTS & DMPS =
# ==================

LYN_REFERENCE := fe6-reference.o

# OBJ to event
%.lyn.event: %.o $(LYN_REFERENCE)
    $(LYN) $< $(LYN_REFERENCE) > $@

# OBJ to DMP rule
%.dmp: %.o
    $(OBJCOPY) -S $< -O binary $@

# ========================
# = ASSEMBLY/COMPILATION =
# ========================

# Setting C/ASM include directories up
INCLUDE_DIRS := Wizardry/Include
INCFLAGS     := $(foreach dir, $(INCLUDE_DIRS), -I "$(dir)")

# setting up compilation flags
ARCH    := -mcpu=arm7tdmi -mthumb -mthumb-interwork
CFLAGS  := $(ARCH) $(INCFLAGS) -Wall -Os -mtune=arm7tdmi -ffreestanding -mlong-calls
ASFLAGS := $(ARCH) $(INCFLAGS)

# defining dependency flags
CDEPFLAGS = -MMD -MT "$*.o" -MT "$*.asm" -MF "$(CACHE_DIR)/$(notdir $*).d" -MP
SDEPFLAGS = --MD "$(CACHE_DIR)/$(notdir $*).d"

# ASM to OBJ rule
%.o: %.s
    $(AS) $(ASFLAGS) $(SDEPFLAGS) -I $(dir $<) $< -o $@

# C to ASM rule
# I would be fine with generating an intermediate .s file but this breaks dependencies
%.o: %.c
    $(CC) $(CFLAGS) $(CDEPFLAGS) -g -c $< -o $@

# C to ASM rule
%.asm: %.c
    $(CC) $(CFLAGS) $(CDEPFLAGS) -S $< -o $@ -fverbose-asm

# Avoid make deleting objects it thinks it doesn't need anymore
# Without this make may fail to detect some files as being up to date
.PRECIOUS: %.o;

-include $(wildcard $(CACHE_DIR)/*.d)

To test this setup, I have made this C code (named NoTime.c):

unsigned GetGameTime(void)
{
    return 0;
}

Now we can make NoTime.lyn.event to generate the object and then the event file for it! Even better yet, we can include NoTime.lyn.event anywhere in our buildfile and we will have it built automatically when we run make!

If you’re curious as to what this does: this makes use of lyn’s “auto-hooking” feature to replace the GetGameTime function which was defined in the reference. Making it always return 0 will stop a bunch of time-related events from happening in-game. For example, standing map sprites will stop being animated (this should be easy enough to notice).

Integrating the decompilation

Integrating the decompilation into our Wizardry pipeline will allow us to not have to manually write references and headers for our C wizardry.

First, let’s actually include the decompilation in our project. To do so, we will be using git submodules. This is why I wanted to make our project a git repository earlier.

We will add the decompilation project as a submodule under Tools/fe6 (remember that I am using FE6 for this). Using commandline git, we do this:

git submodule add https://github.com/StanHash/fe6.git Tools/fe6

Now, before we integrate the decompilation into our project, let’s make sure the decompilation can be built. See the installation instruction for the decompilation project you’re using to set it up.

Once our decompilation is ready, we can build it from our project folder using this:

make -C Tools/fe6

Now that we have that done, we can start working to the integrating part.

To make things simpler in the makefile, I will add some make variables that point to the decompilation’s directory and important files:

FE6_DIR := Tools/fe6

FE6_GBA := $(FE6_DIR)/fe6.gba
FE6_ELF := $(FE6_DIR)/fe6.elf

Note that we are replacing the old FE6_GBA definition with this now, so you can remove that one (as well as the fe6.gba ROM in the root directory). We will use the decompilation’s FE6 ROM from now on.

The ELF file will be useful later.

Adding the decomp’s include directory to the C include path

This is the easy part. We just have to edit the INCLUDE_DIRS make variable defined in wizardry.mk to include it:

INCLUDE_DIRS := Wizardry/Include $(FE6_DIR)/include

And we’re pretty much done for the include files. Now on to the harder part.

Generating a reference from the decomp’s resulting ELF

We’re going to go off the resulting ELF as it contains all symbol information we need and is relatively easy to parse.

I wrote a python script to basically print out a reference source from an input ELF file, using pyelftools:

elf2ref.py 
import sys
from datetime import date

def iter_elf_symbols(f):
    from elftools.elf.elffile import ELFFile
    from elftools.elf.sections import SymbolTableSection

    elf = ELFFile(f)
    section = elf.get_section_by_name('.symtab')

    if section == None or not isinstance(section, SymbolTableSection):
        return

    for sym in section.iter_symbols():
        if sym.entry.st_info.bind != 'STB_GLOBAL':
            continue

        yield (sym.entry.st_value, sym.name, sym.entry.st_info.type == 'STT_FUNC')

def main(args):
    try:
        elfname = args[0]

    except IndexError:
        sys.exit("Usage: {} <ELF>".format(sys.argv[0]))

    with open(elfname, 'rb') as f:
        elf_symbols = { addr: (name, is_func) for addr, name, is_func in iter_elf_symbols(f) }

    print("")

    print(f"@ generated by elf2ref on {date.today()}")
    print("")

    print(".macro fun value, name")
    print("    .global \\name")
    print("    .type \\name, function")
    print("    .set \\name, \\value")
    print(".endm")
    print("")

    print(".macro dat value, name")
    print("    .global \\name")
    print("    .type \\name, object")
    print("    .set \\name, \\value")
    print(".endm")
    print("")

    addr_list = sorted(elf_symbols.keys())

    for addr in addr_list:
        if addr < 0x02000000:
            continue

        sym = elf_symbols[addr]

        print(f"{'fun' if sym[1] else 'dat'} 0x{addr:08X}, {sym[0]}")

if __name__ == '__main__':
    main(sys.argv[1:])

Since this is using a non-standard python module, we need to install it using pip:

pip install pyelftools

I will put this file into Tools/elf2ref.py for now. Maybe later you’d want a Scripts subdirectory for small scripts like this one.

Now, you can invoke it as such:

python Tools/elf2ref.py Tools/fe6/fe6.elf > fe6-reference.s

Note: This process may take several seconds.

And your fe6-reference.s file should contain definitions for all public symbols from the FE6 decomp (that is, all non-static functions and all global variables).

Let’s now automate the process of making the reference by adding it to the makefile.

First, since this script uses python 3, we need to make sure we are using a python 3 executable. This snippet should take care of it:

# Making sure we are using python 3
ifeq ($(shell python -c 'import sys; print(int(sys.version_info[0] > 2))'),1)
  export PYTHON3 := python
else
  export PYTHON3 := python3
endif

Then, we just add a rule to our makefile for making the reference source.

ELF2REF := $(PYTHON3) Tools/elf2ref.py

$(LYN_REFERENCE:.o=.s): $(FE6_ELF)
    $(ELF2REF) $(FE6_ELF) > $(LYN_REFERENCE:.o=.s)

And there we go! Our reference will now be rebuilt everytime the decomp ELF is updated.

Automatically keep the decompilation up to date

We now are basically done integrating the decomp in our project, but we can make it so that we don’t have to manually rebuild the decomp when we make changes to it.

For that, I’m going to add the following rules to our makefile:

$(FE6_GBA) $(FE6_ELF) &: FORCE
    @$(MAKE) -s -C $(FE6_DIR)

FORCE:
.PHONY: FORCE

Now there’s a bit of make trickery going on here so I’ll try explaining:

Firstly, the & in the target list. This it so tell make the the recipe will make both targets. Otherwise, the recipe will be run for each target (thus, twice).

Secondly, the FORCE target. This is simply to force the targets to be rebuilt everytime. Effectively this means that we will make the decompilation every time we need the ROM or ELF. This isn’t as bad as it sounds, as the decompilation makefile will itself still only build the files if it needs to, but we prefer to leave that to the decomp rather than our project.

The reason we are using FORCE and not making $(FE6_GBA) $(FE6_ELF) phony directly is a bit of a weird one. If we make those phony, (GNU) make will not consider the actual files at all, and always consider them new, while the FORCE method, while still running the recipe every time, will still take the target’s age into account. Without this, all targets that depend on those files wound be rebuilt every time, even if the actual files don’t change.

Testing

Now that we finally have our EA/make/decomp, let’s try to make use of it. I’m going to make a new C file called NewAid.c and have this in it:

#include "unit.h"

int GetUnitAid(struct Unit* unit)
{
    if (UNIT_ATTRIBUTES(unit) & UNIT_ATTR_MOUNTED)
        return 25;

    return unit->pow;
}

This will replace the GetUnitAid function to always return 25 for mounted units, otherwise return the unit’s power (strength/magic) stat. This makes use of a bunch of things defined in the unit.h header from the decomp. Again, lyn auto-hooking allows us to not have to manually hook into the game.

If we include the corresponding .lyn.event file into our buildfile, and run make, it should apply to the game. It’s as simple as that!

Final cleanup

Now we are technically done. We have a working EA/make/decomp setup. This section is here to document the organization changes I’m making from this tutorial to the repository linked below.

First, I’m moving all tool related makefile commands to a separate tools.mk file.

Next, I’ll make sure fe6-reference.s is actually inside the cache directory. I’m defining a variable called FE6_REFERENCE, and redefining LYN_REFERENCE in wizardry.mk to be based on that.

That’s it for the makefile changes. Now let’s rearrange files a bit:

  • Added Wizardry directory and Wizardry/Wizardry.event file.
  • Moved NewAid.c to Wizardry/NewAid/Src/NewAid.c and added Wizardry/NewAid/NewAid.event
  • Moved NoTime.c to Wizardry/NoTime/Src/NoTime.c and added Wizardry/NoTime/NoTime.event
  • Removed Constants.event
  • Moved intro-skipping events to Wizardry/Misc/SkipIntro.event
  • Removed MAKE_HACK.cmd
  • Removed fe6-reference.s
  • Removed fe6.gba
  • Moved Tools/elf2ref.py to Tools/Scripts/elf2ref.py

The end

We’re done! Hopefully that was somewhat useful to somebody. If you have any questions feel free to ask.

The source repository for this is available on GitHub under StanHash/fe6-wizardry (“guide” branch).

:wave:

20 Likes
2

They have compatibility issues on macOS 10.15 Catalina (OK until 10.14). I built it for non-Windows platform.

The link is dead because I deleted my fork but anyone who need it can rebuild it.

3

I have the same problem but with Big Sur. Is there any way to get Wine working on Big Sur?