MODULE: jsonparser 0.1.0

[eri0o]

jsonparser version 0.1.0


Get Latest Release jsonparser.scm | GitHub Repo | Demo Windows | Demo Linux | Download project .zip

I recently found a cool unrelated tool that I wanted to add to my workflow, but the tool output was a json file. There's no good object to represent a JSON object in AGS and I was too lazy to make a plugin.
So I made a parser! This is a JSON minimal parser for Adventure Game Studio. It's based on JSMN, which is a C JSON parser that had an easy to read code.

It's not well tested, but I thought to "formally" release it here in case someone had a need that it would be enough for it. I probably need to work on a better demo and write better docs for it too. But hey, first release!

Usage

If you wish to handle things more manually, you can use a thiner parser that is also faster:

Code: ags

String json_string = "{ \"name\":\"John\", \"age\":30, \"car\":null }";
JsonParser* parser = new JsonParser;

int token_count = 8;
JsonToken* t[] = JsonToken.NewArray(token_count);

int r = parser.Parse(json_string, t, token_count);

// now that you have the Tokens, you can use them to parse as you wish!
if (r < 0) Display("Failed to parse JSON: %d\n", r);
if (r < 1 || t[0].type != eJSON_Tok_OBJECT) Display("Object expected\n");

for(int i=0; i<r  ; i++){
  JsonToken* tok = t[i];
  Display(String.Format("%d ; %s ; %d ; %s ; %d ; %d ; %d", 
    i, tok.ToString(json_string), tok.size , tok.TypeAsString,  tok.start ,  tok.end ,  tok.parent ));
}

Display("JSON Parsing has FINISHED for string\n\n%s", json_string);

You can find here an example in-game usage of the parsing, which may be useful for giving ideas.

This module also packs a more approacheable (but less tested, probably buggy) parser:

Code: ags

String json_string = ""; json_string = json_string.Append("{\"squadName\":\"Super squad\",\"formed\":2016,\"active\":true,\"members\":[");
json_string = json_string.Append("{\"name\":\"Molecule Man\",\"age\":29,\"secretIdentity\":\"Dan Jukes\",\"powers\":[\"Radiation resistance\",\"Radiation blast\"]},");
json_string = json_string.Append("{\"name\":\"Madam Uppercut\",\"age\":39,\"secretIdentity\":\"Jane Wilson\",\"powers\":[\"Million punch\",\"Super reflexes\"]},");
json_string = json_string.Append("{\"name\":\"Eternal Flame\",\"age\":100,\"secretIdentity\":\"Unknown\",\"powers\":[\"Immortality\",\"Heat Immunity\",\"Interdimensional jump\"]}]}");

MiniJsonParser jp;
jp.Init(json_string); // parse json_string and internally generate the tokens

while(jp.NextToken()) // advance the current token and exit when there are no tokens left
{    
  if(jp.CurrentTokenIsLeaf)  // usually the interesting information is on the leafs
  {
    Display(String.Format("%s: %s", jp.CurrentFullKey, jp.CurrentTokenAsString));
  }    
}

Display("JSON Parsing has FINISHED for string\n\n%s", json_string);

Script API

JsonParser
JsonParser.Parse

Code: ags

int JsonParser.Parse(String json_string, JsonToken *tokens[], int num_tokens)

Parses a JSON data string into and array of tokens, each describing a single JSON object. Negative return is a JsonError, otherwise it's the number of used tokens.

You need to preallocate a number of tokens to be used by this method before calling it. Use JsonToken.NewArray(count) to help you.

JsonParser.Reset

Code: ags

void JsonParser.Reset()

Marks the parser for reset, useful if you want to use it again with a different file. Reset only actually happens when Parse is called.

JsonParser.pos

Code: ags

int JsonParser.pos

offset in the JSON string

JsonParser.toknext

Code: ags

int JsonParser.toknext

next token to allocate

JsonParser.toksuper

Code: ags

int JsonParser.toksuper

superior token node, e.g. parent object or array

JsonToken
JsonToken.NewArray

Code: ags

static JsonToken* [] JsonToken.NewArray(int count)

Static helper to ease Token Array creation. Ex: JsonToken* t[] = JsonToken.NewArray(token_count);

JsonToken.ToString

Code: ags

String JsonToken.ToString(String json_string)

pass the json_string that was parsed and generated this token to recover the string this token refers to

JsonToken.type

Code: ags

JsonTokenType JsonToken.type

The type of the token: object, array, string etc.

JsonToken.start

Code: ags

int JsonToken.start

The start position in JSON data string.

JsonToken.end

Code: ags

int JsonToken.end

The end position in JSON data string.

JsonToken.size

Code: ags

int JsonToken.size

The size tells about the direct children of the token, 0 if it's a leaf value, 1 or bigger if it's a key or object/array.

JsonToken.parent

Code: ags

int JsonToken.parent

If it's a child, is the index position of the parent in the token array.

JsonToken.TypeAsString

Code: ags

readonly attribute String JsonToken.TypeAsString

Utility function for debugging, returns the type of the token in a String format.

JsonTokenType
- eJSON_Tok_UNDEFINED, a valid token should never have this type.
- eJSON_Tok_OBJECT, an object, it holds keys and values, values can be any other type.
- eJSON_Tok_ARRAY, an array, the token will contain direct ordered children.
- eJSON_Tok_STRING, the token is a string, could be a key, could be a value, context is needed.
- eJSON_Tok_PRIMITIVE, the token is either a number (float or integer), a boolean (true or false) or null.

JsonError
Used to check parse results.
- eJSON_Error_InsuficientTokens, Not enough tokens were provided. Please use more tokens.
- eJSON_Error_InvalidCharacter, Invalid character inside JSON string.
- eJSON_Error_Partial, The string is not a full JSON packet, more bytes expected.

MiniJsonParser
MiniJsonParser.Init

Code: ags

void MiniJsonParser.Init(String json_string)

Initialize the parser passing a JSON as a string. Common usage is: MiniJsonParser jp; jp.Init(json_string);.

MiniJsonParser.NextToken

Code: ags

bool MiniJsonParser.NextToken()

Advances to the next token. Returns false if no tokens left.

MiniJsonParser.CurrentTokenAsString

Code: ags

readonly attribute String MiniJsonParser.CurrentTokenAsString

The current token content, as a String.

MiniJsonParser.CurrentTokenType

Code: ags

readonly attribute JsonTokenType MiniJsonParser.CurrentTokenType

The current token type.

MiniJsonParser.CurrentTokenSize

Code: ags

readonly attribute int MiniJsonParser.CurrentTokenSize

The current token size, 0 if it's a leaf value, 1 or bigger if it's a key or object/array.

MiniJsonParser.CurrentState

Code: ags

readonly attribute MiniJsonParserState MiniJsonParser.CurrentState

The current state of our mini parser. Helps understanding the JSON tokens we got when parsing.

MiniJsonParser.CurrentFullKey

Code: ags

readonly attribute String MiniJsonParser.CurrentFullKey

Gets the current dot separated key.

MiniJsonParser.CurrentTokenIsLeaf

Code: ags

readonly attribute bool MiniJsonParser.CurrentTokenIsLeaf

Checks if the state and key type currently are a leaf. True if it's, usually leafs are the interesting tokens we want when parsing.

MiniJsonParserState
- eJP_State_START, The parser just started.
- eJP_State_KEY, The current token is key in an object.
- eJP_State_VALUE, The current token is a value in an object.
- eJP_State_ARRVALUE, The current token is a value in an array.
- eJP_State_STOP, Don't parse anything in this state, but the parser is not necessarily done.


License

This code is made by eri0o and is licensed with MIT LICENSE. The code on this module is based on Serge's JSMN, which is also MIT licensed and is referenced in the license.

[Joseph DiPerla]

That's pretty cool! Thanks.

[Snarky]

Hi @eri0o, this is good stuff!

I'm trying to use this parser, and I think I've come across a couple of bugs.

The first is that the parser chokes on horizontal tab (ASCII value 9). You have a case that supposedly treats "\t" as whitespace (two places in the code, lines 60 and 266), but it uses ASCII value 20 instead, which in the ASCII tables I see is "Device Control 4" (whatever that might be). (Edit: The codes for "\r" and "\n" also appear to be wrong, given as 18 and 14 instead of 13 and 10.)

The other is in MiniJsonParser. Basically, it seems that calling MiniJsonParser.Init() repeatedly on the same instance will not fully reset the parser.

If I call Init() once, do NextToken() until I hit a particular token, and then stop, if I then call Init() again, the parser now prepends the CurrentFullKey from the token I stopped at to the CurrentFullKey of every token on this parse.

If I on the first parse make sure to call NextToken() until I reach the end before calling Init() to do another parse, it crashes at some point during the second parse with an "Array index out of bounds" error on jsparser.asc:421 (from :492 in NextToken()).

I can't figure out exactly what the problem is, but clearly some state from the first run is preserved even after calling Init() again.

Actually, would it be possible to add an API call to reset the parser to start at the first token again without re-parsing the JSON string?

The background for my question is that I'm trying to build a more convenient layer on top of the parser, something like jq, where you can have random-access lookup of any key path, and not depend on a certain order in the JSON file. The simplest (though inefficient) way seemed to be to simply do a linear search for each key you try to look up, but that means resetting the parser each time. Hence my issue.

(Edit: I forgot I added some lines to the module script in order to get more helpful error messages. Corrected the line number references to match the official version.)

[Snarky]

OK, I spent some time delving into the code, and I found the source of the second bug. In MiniJsonParser::Init() we need to add:

Code: ags

  this._stk_keyidx_idx = 0;

I also think I probably should build the features I have in mind directly on top of JsonParser instead of MiniJsonParser, since JsonParser provides direct access to the parse tree. (Though it would be easy for MinJsonParser to do the same, simply by having MiniJsonParser.Init() return the _Tokens[] array.) My current thinking is that most of the functionality I'm looking for can be implemented as JsonToken extension methods.

[eri0o]

Hey @Snarky , first, thank you for using the module, you are my first costumer in three years.

Now, the miniparser I kinda wanted to remove since I thought it was badly designed and too buggy and wasn't making things easier to use at all - but this was my own experience as sole user and developer. I just used the regular parser when I needed.

Now about the tab issue thanks for the explanation I filed an issue in the repository and will look into fixing it.

I guess I probably should bring the tap module and update the CI of it and add a bunch of json tests there.

Anyway, this may take a little bit I will get there, sorry for the wait.

[Snarky]

Anyway, this may take a little bit I will get there, sorry for the wait.

No worries! I've made the fixes for the issues I encountered in my local copy, so they're not showstoppers. (I thought I'd even experiment with forking the repo and trying to submit my edits as pull requests.)

Ultimately I'm hoping to be able to do something like:

Code: ags

// In Player.asc, for a custom "Player" struct 
void LoadFromJson(this Player*, JsonParser* json, String path)
{
  if(json.KeyExists(path, this.Id))
  {
    JToken* t = json.Select(path, this.Id);
    this.name = t.StringValue("name");
    this.age = t.IntValue("age");
    this.leftHanded = t.BoolValue("left-handed");
    this.hairColor = HexToAgsColor(t.StringValue("hair-color"));
  }
}

// In GlobalScript.asc, on startup
bool LoadJsonConfig(String jsonString)
{
  JsonParser* json;
  // Omitted: parse the string

  // Load player config
  String path = "players";
  for(int i=0; i<playerCount; i++)
    player[i].LoadFromJson(json, json.SelectArrayIndex(path,i).path);

  // TODO: Load other stuff
  // ...
}

(I haven't thought through the precise API, but essentially I want to be able to freely select/search for particular keys within a particular token and its children and read the corresponding values.)

[Snarky]

I went ahead with a rewrite of the module to fit my requirements and preferences. The implementation is still based on the same JsonParser (discarding MiniJsonParser), and it should produce essentially the same parse trees, but the user-facing API is rather different.

I still need to do some code cleanup, I haven't thoroughly tested it, and there are some minor features missing (there should be a way to dispose of the parsed data if you no longer need it, for example), but it seems to be working, so this could be considered a pre-release:

Json.ash
Json.asc

The API looks like:

Spoiler

Code: ags

/// A Json node/token, with parent/child/sibling links forming a Json tree
managed struct Json
{
  /// ID of this Json token (for internal/debugging use)
  import readonly attribute int Id;
  /// ID of this Json tree (for internal/debugging use)
  import readonly attribute int TreeId;
  
  /// Type of this Json node
  import readonly attribute JsonType Type;
  /// String representation of the Json Type
  import readonly attribute String TypeAsString;
  /// Key for this Json node (null if none, implies this node is the root)
  import readonly attribute String Key;
  /// Value of this Json node,  as a string (i.e. all its contents; if node is not a leaf, its full subtree)
  import readonly attribute String Value;
  /// Value as int
  import readonly attribute bool AsInt;
  /// Value as float
  import readonly attribute bool AsFloat;
  /// Value as bool
  import readonly attribute bool AsBool;
  
  /// Path from the root to this Json node
  import readonly attribute String Path;
  
  /// The root of the Json tree this node belongs to (if node is root, returns itself)
  import readonly attribute Json* Root;
  /// The parent of this Json node (null if none, i.e. this node is root)
  import readonly attribute Json* Parent;
  /// The direct children of this Json node
  import readonly attribute Json* Child[];
  /// The number of direct children of this Json node (size of .Child[])
  import readonly attribute int ChildCount;
  /// The next sibling of this Json node (so if this node is this.Parent.Child[i], returns this.Parent.Child[i+1]; null if none)
  import readonly attribute Json* NextSibling;
  
  /// Whether this Json node is a leaf node
  import readonly attribute bool IsLeaf;
  /// Whether this Json node is the root of the Json tree
  import readonly attribute bool IsRoot;
  
  /// Select a node in this node's subtree (using the subpath to navigate)
  import Json* Select(String path);
  
  /// Convert this Json node to a flat dictionary (making each leaf in its subtree a key/value pair)
  import Dictionary* ToDictionary(Dictionary* dictionary=0);
  
  /// Parse a String into a Json tree, returning the root node (null if error, see Json.ParserState)
  import static Json* Parse(String jsonString);
  /// Parse a text file into a Json tree, returning the root node (null if error, see Json.ParserState)
  import static Json* ParseFile(String fileName);
  /// Status of parser after parsing (if successful >0, count of tokens parsed; if error, JsonError value)
  import static readonly attribute int ParserResult;
  /// Number of Json trees parsed and stored
  import static readonly attribute int TreeCount;
  /// Retrieve a Json node directly (for internal/debugging use)
  import static Json* GetNode(int id);
  /// Retrieve a Json tree
  import static Json* GetTree(int treeId);
};

[close]

And you can use it like this, for example:

Code: ags

  Json* jsonData = Json.Parse("{ "ags-games": [ {"title": "Rosewater", "author": "Grundislav", "released": true} , {"title": "The Flayed Man", "author": "Snoring Dog Games", "released": true}, {"title": "Old Skies", "author": "Wadjet Eye Games", "released": false}] }");
  // Or to read from a file: Json* jsonData = Json.ParseFile("$INSTALLDIR$/ags-games.json");
  if(jsonData)
  {
    Json* agsGames = jsonData.Select("ags-games");
    for(int i=0; i<agsGames.ChildCount; i++) // Loop through "ags-games" array
    {
      Json* agsGame = agsGames.Child[i];
      AgsGameData* gd = AgsGameData.Create();  // Assume we have this managed struct defined

      // Set fields
      Json* title = agsGame.Select("title");
      if(title) gd.Title = title.Value;
      Json* author = agsGame.Select("author");
      if(author) gd.Author = author.Value;
      Json* released = agsGame.Select("released");
      if(released) gd.IsReleased = released.AsBool;
    }
  }

My plan is to use this to let developers and players import (and eventually export) style sheets for the TextField and SpeechBubble modules. I also think the approach of having styling specified in external data files rather than hardcoded in script is generally useful for other modules and templates, particularly in light of the accessibility considerations we discussed some time ago. (In many cases it will probably be more convenient to use INI files and Dictionaries, but JSON is better for more complex, structured data.)