Plugin API

[void]

Hello, fellow AGS-Users,

while using the AGSs Plugin API quite intensive during the past time, I noticed there're some things that may be missing right now. For example, there is no clean way of retrieving all the objects of the game, not only the current room. I know this is more likely due to the resource management of the engine (I guess it only loads the current room and discards all previous objects), but a possibility to load a room on demand would be great.

Also, the implementation of the callback functions back to the scripting language is rather inconvenient. Having three integers as arguments just isn't enough and requires a good portion of workarounds that are not very comfortable. There should be a way of at least pass a String object back to the engine.

I also noticed that the handling of the data types isn't very consistent. Sometimes you use an AGSCharacter struct, sometimes an integer as ID, but there's no way getting the ID from the struct (or at least I don't remember any).

Those things are just my quick thoughts on that, I guess there may be more, but I mainly intended to start a discussion about the API and its further development. Your thoughts on that?

[sonneveld]

I would be nice if other platforms were included in this discussion.

[monkey0506]

The plugin API is essentially nothing more than a publicly exposed (from the perspective of an external library) portion of the main engine code. This would (should) definitely be cleaned up in the refactoring/rewriting of the engine code. There are various things like this where CJ just patched in this requested functionality or that, but he himself admitted that due to spending more time implementing new features and bug fixes than optimizing and maintaining a consistent codebase led to the current state of the engine's source.

Various people are definitely working on it, but it would be good to know what people would like to have exposed to the plugin API (ideally most plugin functionality would be feasible using an AGS-native module instead, but the plugin API is definitely useful in the meantime).

A lot of the more recent changes aren't available to the plugin API, such as the AudioClip and AudioChannel types. That could be useful.

[Crimson Wizard]

It' been a year! wow.
Anyway.

Unfortunately, it seems that we won't be able to maintain (engine's) Plugin API in its current form. By saying "not able" I mean it will be
pain in the... head to do so.
What's wrong with current plugin system?

1) It grants free access to internal engine data. Which...
   a) allows plugin to easily corrupt engine's memory;
   b) prevents from changing internal object implementation;
   c) prevents from changing data location in memory (like, dynamic reallocation on new place, which is common when arrays of objects grow or shrink);

2) It is based on classes with virtual inheritance. Which is not so bad on its own, but it appears that different compilers pass class objects slightly different, and that "slightly" is enough to screw things up when there's virtual table pointer in them.


Keeping those points in mind, those are requirements for (engine's) Plugin API:
1. It should not provide addresses (pointers, references) to real engine objects or parts of those objects.
Instead, engine may return interface objects that serve medium between plugin and game object, and, in certain cases, temporary PODs used to
read/write game object's state.
2. It should not have interfaces with virtual functions. This means that interfaces may only have normal functions and member variables.
3. Additionally, we may consider to let people write plugins in C, in which case the only option for interfaces are C structs with function pointers in them.

In fact, unless I forget some other opportunitues, it seems that C-style interfaces (structs with function pointers) are the only option, because if we can't use virtual functions, we won't be able to create extending classes in plugin to pass their objects to engine (like for the font renderer) :-/.

[Wyz]

Yes I guess that simple structs and function pointers is a safe bet for compatibility. There could perhaps be a wrapper API for people that implements OO-style functionality for people that prefer this. Such wrapper would be compiled along with compiling the actual plugin so it will not suffer from the same cross-compiler issues.

Personally I don't really mind exposing internal representations to plugin developers but I see how this would limit efforts to change internal structures and perhaps hinder progress. It would be nice if there was a more general way to probe and change game variables and since the scripting environment is already OO it would be great if it would make use of this. This way you can tap into any of the default game objects like Characters and Rooms, but also objects managed by other plugins or perhaps something for the future: user built classes.

Something I think is of great use is a callback system (events, functions pointers, whatever you want to call it); I'd like to see this in general for the scripting environment but for plugins it is a great start and something that would be quite useful.

One short note on managed objects and the way they work now:
They register objects with virtual parts; something that will break as already mentioned. What always struck me is that it is a bit inconsistent with the way script functions are registered: this is simply done with a 'register' call specifying the parameter count. For managed objects you need the object interfaces. I guess this could be replaced with another 'register' call but this time specifying callbacks for the serialization, memory allocation and freeing functions (and a string for the object name). I guess this is a simple fix though for backwards compatibility I guess there needs to be a version check.

[Crimson Wizard]

One short note on managed objects and the way they work now:
They register objects with virtual parts; something that will break as already mentioned. What always struck me is that it is a bit inconsistent with the way script functions are registered: this is simply done with a 'register' call specifying the parameter count. For managed objects you need the object interfaces. I guess this could be replaced with another 'register' call but this time specifying callbacks for the serialization, memory allocation and freeing functions (and a string for the object name). I guess this is a simple fix though for backwards compatibility I guess there needs to be a version check.

As a note, to make this more understandable, I believe this is what Wyz meant:

Code: cpp

typedef struct tagUserScriptObject
{
   int         (*Dispose) (void *user_object, bool force);
   const char *(*GetType) ();
   int         (*Serialize)(void *user_object, char *buffer, int bufsize);
} IUserScriptObject;

Code: cpp

// Somewhere in AGSEngine interface
void RegisterScriptObject(void *object, IUserScriptObject *interf);

Usage in plugin:

Code: cpp

int MyClassDispose(void *user_object, bool force)
{
  // something
}

const char * MyClassGetType()
{
  // something
}

int MyClassSerialize(void *user_object, char *buffer, int bufsize)
{
  // something
}

IUserScriptObject myClassInterface = {
  MyClassDispose,
  MyClassGetType,
  MyClassSerialize
};

// Somewhere else...
MyClass *my_object = new MyClass();
engine->RegisterScriptObject(my_object, &myClassInterface);

//-----------------------------------------------------------------------
Considering possible implementation in the engine:

Code: cpp

// base class for built-in and custom script objects
class ScriptObject
{
   AddRef, Release, etc
};

class ScriptUserObject : public ScriptObject
{
   void              *user_object;
   IUserScriptObject *user_interf;
};

//=======================================================
EDIT:
Or, as an alternative:

Code: cpp

void RegisterScriptObject(IUserScriptObject *object);

Code: cpp

class MyClass : public IUserScriptObject
{
}

[Crimson Wizard]

This may take time, but I started the big change for the 3.4.0 branch, which would alter the way managed objects are represented, passed and stored.
This should be quite similar to what fuzzie did in ScummVM. There are differences though, because ScummVM implementation does not support plugins... at all (they built-in some of the plugins as an integral engine part).
In short, there will be no managed pool to store objects and keep record on their reference count, instead there will be an object hierarchy, derived from the base ScriptObject class, which counts references on itself.

It is already done that the script API inside the engine defines a formal script function type, as opposed to original AGS, which literally did not have any specification, meaning function may have any return value (usually int, or 32-bit pointer) and any number of any parameters (usually ints or 32-bit pointers).
Currently (in 3.3.0) the engine API functions are declared as:

Code: cpp

typedef RuntimeScriptValue ScriptAPIFunction(const RuntimeScriptValue *params, int32_t param_count);
typedef RuntimeScriptValue ScriptAPIObjectFunction(void *self, const RuntimeScriptValue *params, int32_t param_count);

The first is to support legacy non-OO functions, the second is a more formal way to call OO-like functions (although internal implementation is usually still non-OO).
I think in future 3.4.0 updates this may be changed to just

Code: cpp

typedef ScriptValue ScriptAPIFunction(ScriptObject *self, const ScriptValue *params, int32_t param_count);

(with 'self' = NULL for static or legacy functions)

The ScriptValue is a simple struct, which may store any type of values the script works with. In general it may look like:

Code: cpp

struct ScriptValue
{
    ScriptValueType Type;
    union
    {
        int32_t     IValue;
        float	    FValue;
    };
    // There are two distinct unions, because some types require having both pointer to an object and offset written as a 32-bit integer;
    // (this is for backwards-compatibility)
    union
    {
        const char          *Str; // as a cast to C-string
        ScriptObject        *Obj; // access ptr as a point to Script Object
        ScriptAPIFunction   *Pfn; // access ptr as a pointer to Script API Function
        // there are also very special internally used pointer types, which I do not mention here
    };
};

Now, this struct may be exposed to PluginAPI, accompanied by typedefs that "hide" meaning of ScriptObject and show the meaning of ScriptAPIFunction (because plugins should be allowed to call one):

Code: cpp

typedef void ScriptObject;
typedef ScriptValue ScriptAPIFunction(ScriptObject *self = NULL, const ScriptValue *params = NULL, int32_t param_count = 0);

The plugins do not really need to see the contents of ScriptObject, because they get or set data by calling script functions. So, they just take the returned pointer and use it as an argument to whatever they call (this is, in general, how they do it now):
(Following may not be an example of efficient code)

Code: cpp

int room_object_graphic = 0;
ScriptAPIFunction *pfn = engine->GetScriptFunction("Object::GetAtScreenXY");
ScriptValue params[2];
params[0].SetInt(100);
params[1].SetInt(200);
ScriptValue retval = pfn(NULL, params, 2);
ScriptObject *room_obj = retval.Obj;
if (room_obj){
  pfn = engine->GetScriptFunction("Object::get_Graphic");
  retval = pfn(room_obj);
  room_object_graphic = retval.IValue;
}

[Wyz]

I like your approach on the script functions; that would be quite an improvement.

I'm not sure if I get your proposals for registering managed objects in Plugins. Could you give an example of how that would look from the plugin's point of view?

(Btw, I show my face here way too little, less then I also would like. I just wanted to say you're doing a great job moving things along, and I applaud that.)