Transfer · Do Tasks · do_capture_profiles

Purpose

The do_capture_profiles task is used to synchronize the profiles table in the analytical database with the profiles in the profile database. Only when this step is done, the profiles are available in realtime.

Example usage

task:
    type: do_capture_profiles

extract:
    table_id: profiles
    dataset_id: transfer_testdata
    project_id: factbase-augment
    limit: 10

deduplicate:
    type: truncate

load:
    project_id: factbase-7dj2
    log_result: no
                

Root properties

Properties: Extract

propertytyperequireddescription
project_idstringnoBiqQuery only. Default is project_id of client_cloud
dataset_idstringyesBiqQuery only. Dataset ID of the profiles table.
table_idstringyesBiqQuery only. Table ID of the profiles table.
databasestringyesSnowflake only. Default is database of client_cloud. Database of the profiles table.
schemastringyesSnowflake only. Schema the profiles table.
databasestringyesSnowflake only. Table of the profiles table.
limitintegernoLimits the amount of profiles to be synced with the profile server. Use in development mode.

Properties: Deduplicate

propertytyperequireddescription
typeenumerator(replace, truncate)noEither `truncate` or `merge`. If set to truncate the complete profiles table will be truncated, before the profiles will be imported. This basically means a reset of the profile table. If not set or set to replace the profiles of the analytics database will replace the profiles in the profiles database.

Properties: Load

propertytyperequireddescription
project_idstringnoBiqQuery only. Default is project_id of client_cloud
log_resultyesno (boolean)noIf set to yes, the task will count all profiles in the profile database and log the result. Extra charges by your cloud may apply. Use no in production situation.