Custom Data Models

Using Master Data

Concept

In case and process applications that are created with Flowable, there is often a need to reference lists of data, which do not change daily. A good example is a list of countries that are shown in a task or work form to select the country of origin for an incoming order.

The data object engine in Flowable provides the possibility to define a data object definition that specifies which fields are needed for the master data values. The name and key fields are required because they are needed for every master data value. Optionally, an external id and any number of additional values can be configured. Coming back to the country example the name refers to the country name, the key to the three-character alpha code of the country. Also, you could also have a two-character alpha code value for a country and a numeric value that is used to order the list of countries differently or is a lookup value that can be used to retrieve additional information about the country for example. This guide shows how to define a data object definition and how the master data values are used in a task form.

Deploying a New Data Object Definition

The data object engine in Flowable both follow the same pattern as the other engines and allows us to deploy definitions with the Java or REST API or with auto deploying from the file system. The deployed definition is then versioned in the data object engine and is used to create master data values. Let us look at an example data object definition that provides the country master data example that was discussed in the previous section.

{
  "key": "md-country",
  "name": "Country",
  "description": "The master data definition for countries.",

  "dataObjectType": "masterData",
  "type": "internal",
  "subType": "country",

  "sourceId": "JSON",
  "supportsNameFiltering": true,

  "keyField": "alpha3Code",
  "idField": "alpha3Code",
  "nameField": "name",

  "variables": {
    "alpha2Code": "alpha2Code",
    "numericCode": "numeric"
  }
}

The key in the data object definition is important from a versioning and deployment perspective. The value in the key field is used to identify the data object definition uniquely and if there is already an existing data object definition with the same key value, a new version for this data object definition is created. The same logical key concept is used for the other Flowable engines as well. Most of the other fields are used for metadata and definition query reasons. The key and name field values are important for master data definitions and refer to the JSON fields in the countries JSON document we discuss in the next section. In addition to the key, name and id fields also an unlimited number of variables can be defined that are stored together with the master data instance as well and can be used to show data in the master data selection form field or to query on the master data instances for example.

The data object definition JSON file needs to have a file suffix of .data because the data object engine filters on files with that suffix. Now the data object definition is configured, it can be deployed on the data object engine and this is done with the autodeploy feature by adding the .data file to the com/flowable/dataobject/default or com/flowable/dataobject/custom folder. When rebooting Flowable the data object definition is automatically deployed. Using the REST API, this can also be done with needing the reboot of the application. When the application is running on localhost and port 8080 using the flowable-work context root, a deployment can be done by making a multipart form data POST request to the following URL:

http://localhost:8080/flowable-work/dataobject-api/dataobject-repository/deployments

The .data JSON file needs to be added as a file in the multipart form data POST request. After the POST request is executed, the data object definition is deployed and can be retrieved using the following REST GET request:

http://localhost:8080/flowable-work/dataobject-api/dataobject-repository/dataobject-definitions

The data object definition REST response looks like this for the single country data object definition that is deployed.

{
    "data": [
        {
            "id": "0048baf2-50c0-11e9-859b-acde48001122",
            "name": "Country",
            "key": "md-country",
            "version": 1,
            "type": "internal",
            "subType": "country",
            "sourceId": "JSON",
            "sourceType": "JSON",
            "resourceName": "country-masterData.data",
            "deploymentId": "DOE-0047a980-50c0-11e9-859b-acde48001122",
            "tenantId": "",
            "creationTime": "2019-03-27T18:41:58.022Z",
            "creatorId": "admin",
            "description": "The master data definition for countries.",
            "dataObjectType": "masterData",
            "keyField": "alpha3Code",
            "idField": "alpha3Code",
            "nameField": "name",
            "variables": {
                "alpha2Code": "alpha2Code",
                "numericCode": "numeric"
            }
        }
    ],
    "total": 1,
    "start": 0,
    "sort": "id",
    "order": "asc",
    "size": 1
}

Importing Master Data Values

With the data object definition deployed it is now possible to import master data values into the data object engine so it can be used later on when designing a form with a countries select form field. The default way to import master data is with a JSON file. With the data object definition that was deployed in the previous section the data is expected to have a format similar to the following example.

[
  {
    "name": "Australia",
    "alpha2Code": "AU",
    "alpha3Code": "AUS",
    "numeric": 36,
    "sortOrder": 0,
    "favorite": true
  },
  {
    "name": "Bermuda",
    "alpha2Code": "BM",
    "alpha3Code": "BMU",
    "numeric": 60,
    "sortOrder": 1
  },
  {
    "name": "Haiti",
    "alpha2Code": "HT",
    "alpha3Code": "HTI",
    "numeric": 332,
    "sortOrder": 2
  }
]

In general, the JSON properties are all mapped to either a name, key or id field directly in the master data instance or to a variable. But in addition also the sort order can be defined with the sortOrder property and a favorite master data value can be configured with the favorite is true property value. The favorite master data values will by default be shown at the top of the master data options.

A REST API call similar to the data object definition deployment example is used to import the countries JSON data. A multipart form data POST request is executed to the following URL (again using the localhost with port 8080 and flowable-work context root as an example):

http://localhost:8080/flowable-work/dataobject-api/dataobject-runtime/import-masterdata-instances?dataObjectDefinitionKey=md-country

After this call is executed, the data object engine creates three master data instance entries. These values can now be queried, like with the following GET request:

http://localhost:8080/flowable-work/dataobject-api/dataobject-runtime/masterdata-instances?dataObjectDefinitionKey=md-country&nameLikeIgnoreCase=a%

This request searches for all master data instances with a name starting with a within the data object definition list of md-country. A response similar to the following JSON document is expected:

{
    "data": [
        {
            "id": "DOE-9ada08f7-5173-11e9-b617-acde48001122",
            "name": "Australia",
            "key": "AUS",
            "externalId": "AUS",
            "dataObjectDefinitionId": "ea151a16-516c-11e9-b617-acde48001122",
            "dataObjectDefinitionKey": "md-country",
            "state": "ACTIVE",
            "sortOrder": 0,
            "favorite": false,
            "creationTime": "2019-03-28T16:07:37.257Z",
            "creatorId": "admin",
            "updateTime": "2019-03-28T16:07:37.257Z",
            "updaterId": "admin"
        }
    ],
    "total": 1,
    "start": 0,
    "sort": "name",
    "order": "asc",
    "size": 1
}

With the data object definition and the master data instances created, the master data can now be used in a task form.

Auto Importing Master Data Values

It is possible to automatically deploy master data values when the application starts. By default when the application starts it will import all master data values from all JSON documents from the following classpath directories:

• master-data
• com/flowable/master-data/custom

This location can be customized with the flowable.dataobject.master-data.resource-locations property. For example in order to import from the my-master-data classpath directory. The property should be set to classpath*:/my-master-data/.

The format of the JSON documents should be the following

{
  "dataObjectDefinitionKey": "md-country",
  "masterData": [
    {
      "name": "Australia",
      "alpha2Code": "AU",
      "alpha3Code": "AUS",
      "numeric": 36
    },
    {
      "name": "Bermuda",
      "alpha2Code": "BM",
      "alpha3Code": "BMU",
      "numeric": 60
    },
    {
      "name": "Haiti",
      "alpha2Code": "HT",
      "alpha3Code": "HTI",
      "numeric": 332
    }
  ]
}

When the server starts every JSON element from the masterData array would be mapped to a master data instance. If a master data instance for the configured key based on the master data model already exists, then it would be updated with the one in the document. Elements which are no longer part of the document will not be deleted by default. Read below for how to configure that.

Apart from the format given above. The JSON document can additionally contain the following optional elements:

• tenantId - Optional tenant id that should be used to lookup the data object definition and the tenant id of the created master data instance
• tenantIds - Optional array of tenant ids. Every tenantId from the that array would be should be used to lookup the data object definition and the tenant id of the created master data instance. E.g. If there are 2 tenantIds, then there would instances created in every single one of them.
• overwrite - Boolean flag indicating whether to delete the master data instances which are not part of the document.

::note

In multi-tenancy environments either tenantId or tenantIds should be provided.

:::

It is also possible to force the overwrite (delete master data instances not in the documents) for all documents by setting the property flowable.dataobject.master-data.force-overwrite to true.

In case you need to completely disable the auto import of master data documents. You need to set the property flowable.dataobject.master-data.auto-import to false.

Implementing a Task Form with a Master Data Select Field

In the last section of this guide, a simple BPMN process with one user task is created in Flowable Design with a start and task form that includes a select field that has master data as it's data source. The first step is to create a very simple BPMN process like the following example.

Now let us add a new start form to the start event. In the following form, a Data entry>Text field has been added and then a Data entry>Select (single) form field. The following screenshot shows the necessary settings:

The Select (single) form field needs to be configured to reference the country data object definition that was deployed earlier. Set Data source to master and Data source>Master data key to md-country. For large lists of metadata (not the three value example here).

It is important to note that the Autocomplete>Enable autocomplete form field should be checked and the Autocomplete>Input min. length field should be set to the minimum number of characters that need to be typed before activating autocomplete. You can set it to 0 to trigger the autocomplete logic immediately.

After finishing the form design, let us go back to the example process and also reference the same form for the user task. The next step is to create an app and publish it to Flowable. When starting a new process instance, the start form appears, and the country selection field is filled with the three countries that were imported into the data object engine. When choosing a country and completing the start form, the user task becomes available, and the selected country is shown in the task form.

Master data default tenant fallback

v3.16.1+

In Flowable, the concept of accessing definitions by keys is highly beneficial in multitenant environments. This setup allows you to deploy master data object definitions to the default tenant, making them accessible to all other tenants through their keys. We have further enhanced this concept by enabling master data instances to be requested by key, with a fallback mechanism to the default tenant.

The fallback occurs under the following conditions:

• no master data instance exists for a requested data object definition key within the tenant

• the model is configured to map by key

  

note

The fallback will not trigger if you access any master data instance by ID.

Data Objects

When using forms and data in Flowable you can use basic values like an account number or a customer name. But in some cases you would like some specific values to be grouped together in a logical data entity and manage these entities in a separate database table or REST service and not as individual process or case variables. This is the purpose of the Data Object Engine (in conjunction with the Service Registry). It provides the ability to store the data separate from the process and case variables using either a database table, a REST service or an expression (which can be implemented with a Spring bean for example).

Data Object Definition

Like with most functionality in Flowable, using the data objects feature starts with designing a so-called Data Object definition. The Data Object definitions can be created in Flowable Design and use JSON as the technical format. The foundation of a data object definition looks like this:

{
  "key": "customerDataObject",
  "name": "Customer data object",
  "description": "Customer data object description",

  "dataObjectType": "serviceRegistryDataObject",
  "sourceId": "flowableServiceRegistryDataSource",

  "type": "customer",
  "subType": "orders"
 }

• key: Is the Unique human-readable identifier for the data object. The key is used for versioning like with other definition types.

• name: The name for the data object.

• description: An optional description for the data object.

• dataObjectType: The type of the data object definition. The currently supported values are:

  • masterData: Use the data object definition to manage the master data instances of a list of values.

  • serviceRegistryDataObject: Use the service registry engine to manage the data object in a database table / REST service or expression.

• sourceId: The source for this data object. The currently supported values are:

  • JSON: For master data definitions the values can be imported in JSON format.

  • flowableServiceRegistryDataSource: For data objects that use the service registry engine to manage the data, this value should be used.

Defining the fields for a data object

With the foundation of the data object definition format defined, let’s look at how to define the fields / attributes for a data object. Because we are using the service registry to implement the actual logic to manage the data in a database table, REST service or expression, the data object definition only defines the fields for a data object. Let’s look again at a basic customer example.

{
  "key": "customerDataObject",
  "name": "Customer data object",
  "dataObjectType": "serviceRegistryDataObject",
  "sourceId": "flowableServiceRegistryDataSource",

  "fieldMappings": [
    {
      "name": "customerName",
      "type": "STRING"
    },
    {
      "name": "accountNumber",
      "type": "STRING"
    },
    {
      "name": "eligibleForDiscount",
      "type": "BOOLEAN"
    }
  ]
}

The field mappings array defines the fields / attributes a data object has. A field mapping has the following properties:

• name: The mandatory name of the value that can be used to reference the data object property.

• type: The mandatory type of the value. Currently supported types are: STRING, INT, LONG, DOUBLE, BOOLEAN, and DATE.

• label: A logical name to reference the data object property.

• defaultValue: The optional default value that is used if there is no value set during the creation of the data object.

• description: The optional description for the field mapping. This field is only used for documentation purposes.

Defining the service registry definition

With the customer data object defined, the next step is to design a service registry definition to connect the data object to an implementation type. Again, this can be done using Flowable Design, but in this developer guide we focus on the structure of the definitions and the logical combination of the model types. Let’s assume the customer data object needs to be managed in a database table named CUSTOMER_DATA. The following service registry definition can be created for this.

{
  "key": "customerService",
  "name": "Customer service",
  "type": "database",
  "referenceKey":"customerDataObject",
  "tableName":"CUSTOMER_DATA"
  "lookupId":"accountNumber",
  "columnMappings": [
    {
      "name": "customerName",
      "type": "STRING",
      "columnName": "NAME_"
    },
    {
      "name": "accountNumber",
      "type": "STRING",
      "columnName": "ACCOUNT_NUMBER_"
    },
    {
      "name": "eligibleForDiscount",
      "type": "BOOLEAN",
      "columnName": "ELIGIBLE_"
    }
  ],
  "operations": [
    {
      "name": "Lookup",
      "key": "findById"
    },
    {
      "name": "Create",
      "key": "create"
    },
    {
      "name": "Update",
      "key": "update"
    },
    {
      "name": "Delete",
      "key":"delete"
    }
  ]
}

When a service registry definition is connected to a data object definition that persists the data in a database table, the type is set to database. And the referenceKey points to the key value of the data object definition. The tableName property is used to defined the database table name and the lookupId refers to the data object field that is used to uniquely lookup a data object instance. In this case the account number is used, because that’s a unique value to identify a customer data object.

For each field mapping that is defined in the data object definition a column mapping is defined in the service definition with a reference to the database column name in the columnName property.

Lastly, there are 4 fixed operations that are required for a service definition that implements the data object persistence:

• findById: find a single instance of the data object using the lookup id value.

• create: creates a new instance of the data object.

• update: updates an existing data object instance.

• delete: deletes an existing data object instance.

The service registry definition can also have a type value of REST or expression for a data object implementation. The same operations are used for these types as well, only the mapping to a database table will then change to a REST service endpoint or an expression. In other developer guides the usage of these other types is described using Flowable Design to define the data object, service registry and Liquibase changelog models.

Defining the Liquibase changelog model

Flowable also allows you to manage the database table to be used to store the data object instances as an optional feature. In this way the Liquibase changelog can be automatically generated from the service registry definition, and future changes to the database table are managed by Liquibase.

In Flowable Design, a new Liquibase changelog model can be created, referencing an existing service registry model. The changelog XML content is automatically generated based on the table name and column mappings of the service registry model. The changelog model can then be deployed to Flowable together with the other case / process artefacts.

Using data objects in Flowable

With the data object, service registry and the optional Liquibase changelog definitions in place, we can now start using the data objects in case, process and form models. The starting point is in most cases a start form or task form that creates a new data object instance based on a number of form fields. In a form model a list of data objects can be referenced with a specific scope name. The scope name can then be used in the form field binding like it’s also done for the out-of-the-box root and parent scopes for example.

{
  "dataModel": {
    "customer": {
      "type": "dataObject",
      "dataObjectDefinitionKey": "customerDataObject",
      "mode": "create"
    }
  }
}

• type: The type of the data model. Currently the only supported type is dataObject.

• dataObjectDefinitionKey: The key of the data object definition that is used to create or modify the data object.

• mode: The mode for the variable. The following modes are supported:

  • create: It is only allowed to create a data object, but not modify the value.

  • modify: It is only allowed to modify a data object, but not create the object.

  • view: It is only allowed to view a data object, but not create or modify the object.

In the form now the following binding can be used for example: {{customer.customerName}} or {{customer.accountNumber}}. When submitting the form, the Flowable logic will automatically fill the values in a newly created customer data object instance, and use the Service Registry to store it in the database table in this case (but that could also be calling a REST service or expression).

The customer variable is then available in the case / process scope where this start form is used in. So in for example a condition expression on a sequence flow the following expression could be used: ${customer.eligibleForDiscount} to validate if the customer data object instance has a boolean value of true for the eligible for discount field.