Wrapping C++ APIs for use in KL

This guide is intended to help developers wrapping C++ extensions for use in KL to use the best practices and follow common conventions when wrapping their own APIs using the EDK.

Overview

The KL language is designed to provide a low barrier to entry for developers migrating from other languages. KL features high level systems such as reference counted containers and interfaces.

The goal of wrapping an API for KL is to enable developers working with the KL language to utilise the API in a similar way to how they would work with the API in C++ or other languages where the API is exposed. The consistent mapping of an API ensures that users familiar with the API can transition to KL and quickly become productive without needing to learn new concepts.

C++ API Code:

class A {
  std::string getName();
}

C++ Example Code:

A *val1 = new A();
sprintf(val1->getName());
delete val1;

KL Example Code:

A val1 = A();
report(val1.getName());
val1 = null;

If an API has been well wrapped, then code written in C++ can be migrated to KL with only minor modifications. Classes defined in C++ can be exposed in KL with the same interfaces, meaning that users can create KL type and invoke methods on them as they would in C++. The differences between the KL and the C++ API should mainly be derived from the simpler memory management exposed in KL. Concepts such as pointers and memory allocation are not exposed in KL.

Where there is direct overlap with the provided KL types, It is preferable to convert the API types directly to the KL types in the API wrapping layer. An example of this is basic math types such as Integers, Scalars, Vectors and Quaternions. Fabric Engine comes with a complete math library, so to provide a seamless integration the KL Math types should be generated by the API unless some good reason exists to propagate custom Math types in to KL.

Type Conversion

Providing a set of utility functions to convert between common data types is the first step in mapping an API from C++ into KL. Conversion functions should be implemented for both directions enabling values to be pushed into the C++ API from KL, and pulled out of the API layer into KL.

For convenience, you can use the use namespace functionality of C++ to avoid prefixing the types and functions:

Example Code from the Bullet Integration

inline bool vec3ToBtVector3(const KL::Vec3 & from, btVector3 & to) {
  to.setX(from.x);
  to.setY(from.y);
  to.setZ(from.z);
  return true;
}

inline bool btVector3ToVec3(const btVector3 & from, KL::Vec3 & to) {
  to.x = from.getX();
  to.y = from.getY();
  to.z = from.getZ();
  return true;
}

These utility methods become a key component of the API wrapping, and should be as simple and performant as possible. These methods are usually used to map arguments provided to methods from KL to the C++ API types and therefore may be called many times per graph evaluation.

Note

In some circumstances where the KL type’s memory layout can be guaranteed to match the memory layout of the C++ API types, the KL type can simply be cast to the C++ type. Using the ref:EAG.kl2edk utility, you can validate the exact memory layout of your KL types by checking the generated header files. See The kl2edk Utility

Mapping Classes to Objects

An API might define a collection of C++ classes organized into a hierarchy. The user of the API can construct these classes and invoke methods on them. Conceptually, the classes defined in the API should map to KL Objects. An instance of a KL Object should represent an instance of the C++ class. This relationship is managed via mapping methods on KL objects to static methods in the C++ wrapper layer.

Memory Management

In KL, Objects are ref counted objects. Only once all references to an object are removed will an object be destroyed. KL objects provide constructors and destructors that are called when the object is created or destroyed. The lifetime of the KL object should be used to manage the lifetime of the instantiated C++ classes that the object represents.

object ObjectA {
  Data pointer;
}

function ObjectA() = "ObjectA_construct";
function ~ObjectA() = "ObjectA_destroy";

KL objects must be referenced by at least once by an owning class or node to ensure that they are not destroyed. By maintaining references to objects, you can guarantee that they will not be destroyed, and can control when they will eventually be freed. For systems of interdependent classes see ‘Managing Data ownership and bidirectional relationships’ below.

Mapping Public methods

Mapping methods from KL to C++ starts by defining the KL method, and the name of a static method that would be invoked when that method is called from C++. Only the public methods that define the interface of a class need to be mapped from KL to C++. These methods are the methods that clients of the class must invoke when working with the class. The KL objects represent a mapping of the public interface of the class, rather than a complete mapping of all protected and private methods.

object ObjectA {
  ...
}
function ObjectA.methodA() = "ObjectA_methodA";

Public Members

Mapping of public members from KL to C++ is not automatic. There is no facility in KL to detect when a KL Object’s or Struct’s member value changes to automatically synchronize values to the mapped C++ class or struct. Ideally all interactions with a struct or class occur through public methods(except in the ‘Simple Data-Container Structs’ example provided below).

Pure data container structs

In some cases, a C++ API may define simple structs that are used to pass a large number of variables into a constructor or method. In this case an equivalent KL struct can be defined, complete with all the same members. When the KL struct is passed into a method, the C++ mapping of the method can handle a manual conversion of the KL struct to the C++ struct. In the same way the Math types are mapped, each of the members is simply converted to the C++ API types.

struct  ClassAConstructionInfo
{
  Scalar      foo;
  Xfo         xfo;
  Vec3        bar;
};
FABRIC_EXT_EXPORT void ClassA_construct(
  Fabric::EDK::KL::Traits< Fabric::EDK::KL::ClassA >::IOParam this_,
  Fabric::EDK::KL::Traits< Fabric::EDK::KL::ClassAConstructionInfo >::INParam constructionInfo
)
{
  ClassAConstructionInfo info;

  scalarToFloat(constructionInfo.foo, info.m_foo);
  xfoToTransform(constructionInfo.xfo, info.m_xfo);
  vec3ToVector3(constructionInfo.bar, info.m_bar);

  this_->pointer = new ClassA(info);
}

Passing Arrays from KL to a C-Style API

Some C++ API’s, typically APIs developed for use in game runtimes, avoid using C++ array representations(e.g. std::vector), or any other higher level array representations, and instead use a combination of pointers and count values. Because pointers are not exposed in KL, the exposed KL API must operate at a slightly higher level, rather than expose the C++ method arguments directly in KL. A single array value can be passed into the method and in the C++ wrapping code, the arguments expanded for the C-style API. This provides a slightly higher level, but easier to use API in KL than the C++ API. It is up to the developers discretion to adopt the higher level function signatures when required and provide the mapping of the arguments in the C++ wrapping code. Kl2edk cannot automatically expand the KL arguments required for these functions, so usually these methods must be manually implemented.

function A.setWeights(Vec3 weights[]) = 'A_setWeights';
FABRIC_EXT_EXPORT void A_setWeights(
  Fabric::EDK::KL::Traits< Fabric::EDK::KL::A >::IOParam this_,
  Fabric::EDK::KL::Traits< Fabric::EDK::KL::VariableArray< Fabric::EDK::KL::Float32 > >::INParam weights
)
{
  A* cThis_ = 0;
  if(!KLObjectToCPP<KL::A, A>(this_, cThis_)){
    setError("Error in A_setPositionsArray. unable to convert: this_");
    return;
  }

  this_->pointer.setPositionsArray(weights.size(), &weights[0]);
}

Note

If the C++ class merely uses the passed in array values to extract some data, then as soon as the call stack unwinds the array may be safely destroyed. If the C++ API class stores the pointer to this array, then the memory associated with this array must be referenced by the KL object to ensure that it is not freed before the class is destroyed. The reference in KL to the array will ensure that the array is not freed by KL before the KL class is destroyed. Whenever a C++ class has a dependency on the existence of memory allocated elsewhere, a KL reference to the data must be set. See ‘Managing Data Ownership and bidirectional relationships’ below.

function A.setWeights(Vec3 weights[]){
   this.__weights = weights;
   this.__setWeights(weights);
}
function A.__setWeights(Vec3 weights[]) = 'A_setWeights';

Note

The provided Bullet extension implements this slightly higher level wrapping of the Bullet API, enabling KL arrays to be passed into methods that, in the C++ API expect a count and pointer to be passed. Look at the provided source code of the Bullet extension for examples of how this has been implemented.

Mapping Class Hierarchies

Often C++ APIs are structured as hierarchies of classes that inherit from each other. Objects in KL cannot currently inherit from base objects, and so a direct mapping of a C++ class hierarchy is not possible.

KL provides a system of interfaces which enable the specification of a set of methods that a given class must implement. Interfaces are similar to pure-virtual classes in C++, and therefore provide no implementations of methods or member values. Objects can support multiple interfaces, enabling class hierarchies to be supported through the implementation of interfaces, one for each of the inherited classes in the class hierarchy.

Each KL object must then support the interfaces defined for each of the classes it inherits from in the class hierarchy.

C++ API Code

// A base class that implements a method called ‘getName’.
class A {
  std::string getName();
}

// A derived class that inherits from A.
class B : public A {
  method2(A a);
}

KL Wrapping Code

// The ‘A’ interface declares a method called ‘getName’.
// Objects that support the ‘A’ interface must implement ‘getName’
interface A {
  String getName();
}

// Object B supports the ‘A’ interface (can be automatically cast to A)
// and so must implement all methods defined in ‘getName’.
object B : A {
  Data pointer;
}

// Object B must implement its own methods, and all the methods inherited
// from its interfaces.
function String B.getName() = "b_getName";
function B.method2(A a) = "b_method1";

C++ Wrapping Code.

FABRIC_EXT_EXPORT void KL::String b_getName(...){
  return KL::String(this_.pointer->getName());
}

FABRIC_EXT_EXPORT void b_method1(
  KL::B::IOParam this_,
  KL::A::INParam a
){
  ...
}

KL Example Code

// An instance of B can be created, and assigned to a reference of
// type A interface.
A val1 = B();
report(val1.getName());

// A new instance of type B is created and passed to the first.
// method1 accepts a value of type A interface, so the B object is
// automatically cast.
A val2 = B();
val1.method1(val2);

For deep class hierarchies, many interfaces may be required, and all public methods exposed through the inheritance chain must be implemented directly by the leaf objects.

Note

Full support for Inheritance ok KL objects scheduled for an upcoming release which greatly simplify the mapping of C++ hierarchies.

Managing Data Ownership and bidirectional relationships

In some APIs, you may have a collection of classes whose lifetimes are related, and must be destroyed in a specific order. KL can be utilized to manage the lifetimes of the objects in the system, such that objects will always be destroyed in a predictable order.

The Ref<> feature in KL is a raw unmanaged pointer. Ref<> pointers will not affect the lifetime of an object, and so can be used in cases where backpointers are required.

Note: If 2 classes reference each other, then neither class will ever be destroyed due to the cyclic reference. A Owner object should reference its ‘owned’ objects, and those ‘owned’ objects should maintain simple ‘Ref’ pointers back to the owners. Ref pointers must be manually maintained. If the pointer is not nulled and the owner is destroyed, the pointer will become garbage and cause a crash if accessed. Cleanup is required to ensure your code is stable under all conditions. .

object Slave {
  Data pointer;
  // The slave maintains a raw pointer to the master(not a reference)
  // Only if this pointer is valid is the slave in a valid state.
  Ref<Owner> master;
};
function Boolean Slave.isValid(){
  return this.master != null;
}

object Master {
  Data pointer;
  Slave slaves[];
};
function Master.addSlave(Slave slave){
  slave.master = this;
  this.__addSlave(slave);
  // Maintain a reference to the slave so that it is not destoyed.
  // The calling code may have only a stack-allocated reference to the slave.
  this.slaves.push(slave);
}
function Master.__addSlave(Slave slave) = "Master_addSlave";

function ~Master(){
  // By nulling the back-pointer on the slaves, they become invalid.
  // This can be used to protect against evaluation
  for(Integer i=0; i<this.slaves.size; i++){
    this.slaves[i].master = null;
  }
  this.__destroy();
  // removing all the references from the master to the slaves may cause all
  // the slaves to be destroyed, unless another class references the slaves.
  this.slaves.resize(0);
}
function Master.__destroy() = "Master_destroy";

Handling Dependencies between Classes

When a class hierarchy is being mapped to KL, the dependencies between the class declarations needs to be mapped to kl. The Place where these dependencies are mapped, is the fpm.json file that loads the kl files for the extension.

C++ Code Class A

class A {
  std::string getName();
}

C++ Code Class B

#include <A.h>;
class B {
  A* m_a;
}

KL Code object A

object A {
  Data pointer;
}

KL Code object B

object B {
  Data pointer;
  A a;
}

Resulting MyExt.fpm.json file

{
 "libs": [
  "MyExt"
 ],
 "code": [
  "A.kl",
  "B.kl",
 ]
}

The dependencies between the C++ classes needs to be reflected in the definition of the KL objects, and in the order that the classes are loaded by the fpm.json file.

Note: Object B maintains a KL reference to Object A, to ensure that its lifetime lasts longer than B. Only once A is destroyed or releases its reference to B(as long as only A reference B), B will be destroyed.

Other Tips and Tricks

Mapping Const Functions to KL as const functions

By default, kl methods that return values are const. You may have a class that declares methods that take non-const reference arguments and perform computation returning the results in the args. To declare a KL function that is also cost, simply append a ‘?’ as the end of the function name.

C++ Code

class MyCPPClass {
   void doStuff( Scalar& value) const;
}

KL Code

object MyCPPClass {
  Data pointer;
}
function MyCPPClass.doStuff?(io Scalar value) = "MyCPPClass_doStuff";

C++ Code

class MyCPPClass {
   int computeDataReturnCode();
}

KL Code

object MyCPPClass {
  Data pointer;
}
function MyCPPClass.computeDataReturnCode!(io Scalar value) = "MyCPPClass_computeDataReturnCode";

Cloning Objects in KL

When a KL object that references some C++ data is cloned in KL, then the expected behavior is that the C++ class is also closed, ensuring that the 1-1 mapping of objects to C++ classes.

KL Code

object MyObj {
  String s;
  // …
};
function MyObj MyObj.clone() = "MyObj_clone";

C++ code

FABRIC_EXT_EXPORT void MyObj_clone(
  KL::Traits< KL::MyObj >::Result result,
  KL::Traits< KL::MyObj >::INParam other
{
  // The wrapped class must be cloned,
  // and the cloned wrapper KL object must
  // reference the newly constructed class.
  result->pointer = MyObj(other->pointer);
};