DATA Phase Reference

A Data phase transforms instance variables without any human involvement. The phase runs deterministically on Nembl's serverless executor, writes its output to an instance variable, and advances immediately. Use Data phases to clean up incoming payloads, compute derived values, validate inputs, or pivot array shapes between steps.

Every Data phase config has:

• Sub-type — one of the 7 listed below (filter, sort, aggregate, transform, enrich, validate, transpose)
• Source variable — an instance variable (dot-notation supported)
• Output variable — where the result lands on the instance

The workflow editor's Data Test Panel lets you paste sample input JSON and preview the output before publishing — no instance launch required.

Filter

Remove array elements that don't match a condition set.

Config:

• sourceArray — array to filter (dot-notation OK)
• conditions — array of { field, operator, value }
• outputVariable — filtered array lands here

Supported operators: eq, neq, gt, lt, gte, lte

Multiple conditions combine with AND.

Example: keep active adults.

{
  "sourceArray": "users",
  "conditions": [
    { "field": "active", "operator": "eq", "value": true },
    { "field": "age", "operator": "gte", "value": 18 }
  ],
  "outputVariable": "eligibleUsers"
}

Sort

Sort an array by one or more columns.

Config:

• sourceArray
• sortColumns — array of { field, direction }
• outputVariable

Directions: asc, desc.

Columns are applied in order (first column is primary sort).

Example: sort by priority descending, then created date ascending.

{
  "sourceArray": "tickets",
  "sortColumns": [
    { "field": "priority", "direction": "desc" },
    { "field": "createdAt", "direction": "asc" }
  ],
  "outputVariable": "sortedTickets"
}

Aggregate

Compute a single numeric value from an array. One sub-type covers four operations.

Config:

• sourceArray
• operation — min, max, sum, or average
• field — numeric field name. Leave empty if the array is a flat list of numbers (e.g. [1, 2, 3])
• outputVariable — scalar result

Example: total order value.

{
  "sourceArray": "lineItems",
  "operation": "sum",
  "field": "amount",
  "outputVariable": "orderTotal"
}

Transform

Map and convert field values one at a time.

Config:

• mappings — array of { source, target, transform, transformArg }

Supported transforms (10 total):

Example: normalize names and set a default region.

{
  "mappings": [
    {
      "source": "name",
      "target": "normalizedName",
      "transform": "uppercase"
    },
    {
      "source": "region",
      "target": "region",
      "transform": "default",
      "transformArg": "US"
    },
    {
      "source": "tags",
      "target": "tagList",
      "transform": "split",
      "transformArg": ";"
    }
  ]
}

Enrich

Compute a new value from a math expression evaluated against instance variables.

Config:

• expression — a mathjs expression referencing instance variables by name
• outputVariable — where the computed value lands

Engine: mathjs (opens in a new tab) in sandboxed mode. Standard math functions are available (abs, round, ceil, floor, sqrt, pow, min, max, trigonometry, statistics helpers, etc.). The following are blocked for security: import, createUnit, evaluate, parse, simplify, derivative.

Examples:

// Compute discounted total
"expression": "quantity * unitPrice * (1 - discountPct / 100)",
"outputVariable": "totalAfterDiscount"

// Compute elapsed days
"expression": "floor((now - createdAt) / 86400000)",
"outputVariable": "elapsedDays"

Validate

Evaluate a set of rules and report which pass. Can optionally block the workflow on any failure.

Config:

• rules — array of { field, operator, value, message }
• failAction — error (throw and stop the phase) or setVariable (continue and surface results in the output variable)
• outputVariable — when failAction is setVariable, this contains an array of { field, message, passed }

Operators: same set as Filter — eq, neq, gt, lt, gte, lte.

Example: check minimum age and ensure email present, without blocking the workflow.

{
  "rules": [
    {
      "field": "age",
      "operator": "gte",
      "value": 18,
      "message": "Applicant must be 18+"
    },
    {
      "field": "email",
      "operator": "neq",
      "value": "",
      "message": "Email is required"
    }
  ],
  "failAction": "setVariable",
  "outputVariable": "validationResult"
}

Output lands in validationResult as an array of rule evaluations; a DECISION phase can route on validationResult[0].passed etc. Use failAction: "error" instead when a failed rule should halt the workflow immediately (the instance moves to error state and responsible parties are notified).

Transpose

Pivot an array of objects into a key-value map. Useful for turning tabular data into a lookup shape downstream phases can index directly.

Config:

• sourceVariable
• keyField — which field supplies the map key
• valueField — which field supplies the map value
• outputVariable

Example:

// Input: [{ "name": "env1", "value": "prod" }, { "name": "env2", "value": "dev" }]
{
  "sourceVariable": "settings",
  "keyField": "name",
  "valueField": "value",
  "outputVariable": "settingsMap"
}
// Output: { "env1": "prod", "env2": "dev" }

Chaining Data Phases

Data phases are designed to chain. A common pattern:

Filter (keep active users)
  → Sort (by name)
  → Transform (uppercase names)
  → Aggregate (count)

Point each phase's sourceArray (or sourceVariable) at the previous phase's outputVariable. No workflow-level glue required — it's just variable wiring.

Testing

Open a DATA phase in the editor and click Test to bring up the Data Test Panel:

1. Paste sample JSON input (what the instance variables would contain)
2. Click Run Test
3. See the computed output and, for Validate, the rule-by-rule pass/fail

The test runs client-side for filter/sort/validate/enrich, and server-side for transform/aggregate/transpose. Identical logic to runtime — if the test passes, the phase will pass in production.

Errors at Runtime

• Missing source variable → phase errors with Unknown source variable: <name>. Responsible parties are notified.
• Invalid operator or missing required config field → phase errors at validation time (before the executor even runs).
• validate with failAction: "error" and any failing rule → phase errors with the first failing rule's message.

Errors are recorded in the instance's audit log under workflow.phase_error and are visible on the instance viewer.

Related

• Phases and Transitions — where DATA fits into the phase-type palette
• Decisions and Parallel Paths — how to route on DATA phase outputs
• Workflow Execution — how phases activate and advance at runtime