[V2.0, V0.3] Another Text Parser, by Crazycolorz5

Requirements: Anti-Huffman Patch

Usage:
ParseFile:

  • Download ParseFile.exe. Either download ParseDefinitions.txt or create your own text file for it. Write out your text in alltext format. Example input provided below. Drag your text file to ParseFile.exe and say Run With. A .dmp file should be output; if not it means you messed up your format somewhere.
    Then paste the .dmp somewhere, and repoint an entry in the text table to that location. make sure to add 0x80000000 to your pointer to tell the Anti-Huffman patch that the text is unencoded. (This all can be done automatically with EA)

ParseFolder (recommended):

  • Download ParseFolder.exe. Either download ParseDefinitions.txt or create your own text file for it. Put ParseFolder.exe into a directory with your text files (they may be in a subdirectory), and run it. All text files must be in .txt and AllText format. If there’s an error in parsing, it will tell you the error.

Using ParseDefinitions.txt:

  • On separate lines, write definitions like [ToDefine] = (definition). The space is necessary, and (definition) may be any combination of already defined directives or plain text. For example, [Silent...] = [ToggleMouthMove]...[ToggleMouthMove]. You can then use the definitions in your text files.

Notes:

  • Your line breaks don’t matter; a text’s new line must be given by [NL] (or [2NL]). This is so you can separate out control codes and movement with new lines without fear of messing up your text.
  • I provide the .hs source. Thus if you run Mac or Linux, you can recompile from source with a good Haskell compiler (read: ghc).

Remember that all text has to end with a [X]. I have it automatically adding this if you forget.

[FastText], which is a diffrent code in FE8 and FE6, is [FastText] for FE8 and [FastTextFE6] for Fe6. Same for the other code(s) that re like this.

Source:
FEParser.hs
ParseFile.hs
ParseFolder.hs
CFG model I based parsing off of

Download:
ParseFile V2.1.exe
ParseFolder V0.3.exe
ParseDefinitions.txt

Example input:

#This should be inserted as text id 0xASDFGHJKL
[OpenRight][LoadPortrait][0x01][0x01]
[OpenLeft][LoadPortrait][0x02][0x02]
[OpenRight]
Eliwood, let me tell you[NL]
about a lost art.[A]
[OpenLeft]
What is it, o wise[NL]
question-mark man?[A]
[OpenRight]
Once upon a time, before the[NL]
time of FEditor, we did text[A][NL]
a different way. The anti-[NL]
huffman way. This involved[A][NL]
hex editing. In fact, FEditor[NL]
only automates this method.[A][NL]
[OpenLeft]
I see. Why are you telling me this?[A]
[OpenRight]
Since it's what FEditor is based on,[NL]
people should know how it works.[A][NL]
That way new ways of editing text[NL]
based on it can be created.[A][NL]
For example, imagine if we could[NL]
insert text through EA...[A][NL]
[OpenLeft]
That sounds like a[NL]
useful option to have.[A]
[OpenRight]
By the way, there are bandits[NL]
behind you, Eliwood.[A]
[OpenLeft][MoveLeft]
Ahh![A]
[ClearFace]
[OpenRight]
Bye![A]
[ClearFace][X]

Example Output:

0B 10 01 01 0A 10 02 02 0B 45 6C 69 77 6F 6F 64 2C 20 6C 65 74 20 6D 65 20 74 65 6C 6C 20 79 6F 75 01 61 62 6F 75 74 20 61 20 6C 6F 73 74 20 61 72 74 2E 03 0A 57 68 61 74 20 69 73 20 69 74 2C 20 6F 20 77 69 73 65 01 71 75 65 73 74 69 6F 6E 2D 6D 61 72 6B 20 6D 61 6E 3F 03 0B 4F 6E 63 65 20 75 70 6F 6E 20 61 20 74 69 6D 65 2C 20 62 65 66 6F 72 65 20 74 68 65 01 74 69 6D 65 20 6F 66 20 46 45 64 69 74 6F 72 2C 20 77 65 20 64 69 64 20 74 65 78 74 03 01 61 20 64 69 66 66 65 72 65 6E 74 20 77 61 79 2E 20 54 68 65 20 61 6E 74 69 2D 01 68 75 66 66 6D 61 6E 20 77 61 79 2E 20 54 68 69 73 20 69 6E 76 6F 6C 76 65 64 03 01 68 65 78 20 65 64 69 74 69 6E 67 2E 20 49 6E 20 66 61 63 74 2C 20 46 45 64 69 74 6F 72 01 6F 6E 6C 79 20 61 75 74 6F 6D 61 74 65 73 20 74 68 69 73 20 6D 65 74 68 6F 64 2E 03 01 0A 49 20 73 65 65 2E 20 57 68 79 20 61 72 65 20 79 6F 75 20 74 65 6C 6C 69 6E 67 20 6D 65 20 74 68 69 73 3F 03 0B 53 69 6E 63 65 20 69 74 27 73 20 77 68 61 74 20 46 45 64 69 74 6F 72 20 69 73 20 62 61 73 65 64 20 6F 6E 2C 01 70 65 6F 70 6C 65 20 73 68 6F 75 6C 64 20 6B 6E 6F 77 20 68 6F 77 20 69 74 20 77 6F 72 6B 73 2E 03 01 54 68 61 74 20 77 61 79 20 6E 65 77 20 77 61 79 73 20 6F 66 20 65 64 69 74 69 6E 67 20 74 65 78 74 01 62 61 73 65 64 20 6F 6E 20 69 74 20 63 61 6E 20 62 65 20 63 72 65 61 74 65 64 2E 03 01 46 6F 72 20 65 78 61 6D 70 6C 65 2C 20 69 6D 61 67 69 6E 65 20 69 66 20 77 65 20 63 6F 75 6C 64 01 69 6E 73 65 72 74 20 74 65 78 74 20 74 68 72 6F 75 67 68 20 45 41 2E 2E 2E 03 01 0A 54 68 61 74 20 73 6F 75 6E 64 73 20 6C 69 6B 65 20 61 01 75 73 65 66 75 6C 20 6F 70 74 69 6F 6E 20 74 6F 20 68 61 76 65 2E 03 0B 42 79 20 74 68 65 20 77 61 79 2C 20 74 68 65 72 65 20 61 72 65 20 62 61 6E 64 69 74 73 01 62 65 68 69 6E 64 20 79 6F 75 2C 20 45 6C 69 77 6F 6F 64 2E 03 0A 80 0C 41 68 68 21 03 11 0B 42 79 65 21 03 11 00

Which is, by HxD’s rendering of non-ASCII characters:
.........Eliwood, let me tell you.about a lost art...What is it, o wise.question-mark man?..Once upon a time, before the.time of FEditor, we did text..a different way. The anti-.huffman way. This involved..hex editing. In fact, FEditor.only automates this method....I see. Why are you telling me this?..Since it's what FEditor is based on,.people should know how it works...That way new ways of editing text.based on it can be created...For example, imagine if we could.insert text through EA......That sounds like a.useful option to have...By the way, there are bandits.behind you, Eliwood...€.Ahh!...Bye!...

4 Likes

Updated to version ParseFile 1.0. This mostly makes its --to-stdout flag compatible with EA’s #incext.

Updated to version ParseFolder 0.2. This fixes a few issues with folder traversal. Namely, if you recompile on a UNIX machine, it should work properly there now.

1 Like

Updated to version ParseFile 2.0. It changes up the language a bit, so this is actually important:

  • Whenever a # character is encountered in text, the parser treats the rest of the line as a comment. For example,
asdfg#hjkl

gets parsed the same as

asdfg

To insert a literal #, you can use the control code [#]. The TextRipper WIP should support this as well once I update its thread.

  • Trailing newlines and comments are discarded.
  • Added a -o outputfilename option to allow for a custom name for the output.

Updated ParseFolder 0.3 to support the same parser changes.

I made a version of ParseFile that outputs in the Shift-JIS encoding. The input file must be in Unicode/UTF-8.
Here is a download link. For most users it probably won’t be useful but I’m sure there are some people who may find this useful. For example, it can be used to parse unicode text for DSA,

This is a [Open]test[Close].[NL]
日本語、[0xc0][0xc1]かわいい!![X]

can be turned into a binary output.

Source for this version is on the sjis branch of the EA Formatting Suite Github.

6 Likes

Hello, genius colorz!
I am considering inserting more characters into FE8U using Parse. In fact, through FEB’s Draw-UTF8 patch, FE8U can already correctly recognize the unicode. I hope to insert cn or jp text through TextProcess, But Parse cannot identify UTF8 correctly. Is there anyway to make characters in unicode?
For example:

い[X]

to

E3 81 84 00

Did you see the post just above? Does that solve the problem or am I misunderstanding?

Well it did nt solve the problem…
The above tool can only handle Japanese characters,not all unicode or UTF8 characters.
As an example, it can not handle Chinese character “我”,which should be encoded into E68891.

I am not sure whether the encoding operation of Chinese characters is the same as Japanese. but the above version of Parse cannot handle Chinese characters well.

I think the following is what I need:

我[N]い[X]
to
E6 88 91 01 E3 81 84 00

in which,0xE68891 and 0xE18184 is just UTF8 code if you push .txt file to Hxd.

I hope if there can be a version of Parse that just read the binary data from the text document directly, and then put these values in .dmp file, just handle with FE style control codes like carriage return, character appearance, etc… But Parse seems to have some judgment process, which prevents me from transferring directly.

Version 2.1 “Add non-english punctuation and force reads from utf-8 to support non-english characters better.”, check OP for download link

2 Likes