docs
/
AppEngine API

Bulk, export and migration

Importing many records, streaming exports, and moving data between collections.

Bulk create

POST/repository/bulk-createJWT

Creates many records in one call. The caller is passed through as the author.

For imports that must skip the usual machinery there is createNoChecks, described in its own decorator as bypassing "all collection/workflow/schedule pre/post processing". It backs bulk import, history rewrite and migration.

Records written without checks are not indexed

Skipping pre/post processing also skips search indexing. After a bulk import, run createSearchIndex for the datatype or the records will not appear in full-text results.

Export

POST/repository/export/:datatypeJWT
{ "query": { "data.status": "active" }, "options": { }, "format": "csv", "fields": ["name", "data.email"], "limit": 5000 }

format is csv (default) or json. Anything other than "json" produces CSV.

The response is a download, not a JSON body:

  • Content-Disposition: attachment; filename="<datatype>-<YYYY-MM-DD>.<format>"
  • Transfer-Encoding: chunked — the row count is not known until the last page is read
  • CSV output begins with a UTF-8 BOM (), because Excel otherwise reads a bare UTF-8 CSV as the local codepage and mangles non-ASCII text

It is streamed and paged internally, so it is not limited to one page of results and does not buffer the export in memory. Whole-collection exports are fine.

Migration

Move or copy records between collections, optionally reshaping them on the way.

POST/repository/migrateJWT
{
  "sources": [
    { "datatype": "lead", "query": { "data.status": "converted" },
      "fieldMap": { "data.company": "data.organisation" } }
  ],
  "targetDatatype": "customer",
  "targetCollection": { "name": "customer", "title": "Customers", "schema": { } },
  "mode": "auto",
  "deleteFromSource": false,
  "skipDuplicates": true,
  "batchSize": 50
}
FieldTypeDescription
sources requiredarrayOne or more { datatype, query?, fieldMap? }. query selects records; fieldMap maps source field paths to target field paths. Multiple sources can merge into one target.
targetDatatype requiredstringDestination collection.
targetCollectionobject{ name, title, schema } — creates the target collection if it does not exist.
mode`'instant''batch'
deleteFromSourcebooleanMove rather than copy. Off by default.
skipDuplicatesbooleanSkip records that already exist in the target.
batchSizenumberRecords per batch. Default 50.

Tracking a migration

GET/repository/migrate/status/:jobIdJWT
GET/repository/migrate/jobsJWT
POST/repository/migrate/cancel/:jobIdJWT

migrate/jobs accepts ?status=, ?page= and ?pageSize=.

deleteFromSource is destructive

deleteFromSource: true removes records from the source collection once they are copied. Run the migration without it first and check the result, then re-run to move.

Aggregations

There is no raw MongoDB API: every read and write goes through the repository, so BaseModel stamping, history, search indexing, requiredRole and trash always apply. For queries a plain find cannot express, run an aggregation pipeline against one datatype:

POST/repository/aggregate/:datatypeJWT

The tenant's orgid selects the database, so a pipeline cannot cross tenants.