Tasks refer the individual data tasks that form a job. Onesecondbefore divides the world in three parts: from, do and to.
The `task` part of the configuration considers task related settings. It can be configured for all task types.
start_date: yesterday -3 days
|Contains type of task. Must be one of:
The items below are discussed in more detail in the Do section
The items below are discussed in more detail in the To section
|Default value is the filename without extension. Unique name for the task.
|Timestamp when the job (not the task) was triggered in the local timezone. Useful in deduplicating tables and in SQL templates.
|Unique id per run. Every time a task runs, it receives a unique 8 character alphanumeric string.
|Temporary folder on the worker machine where data will be stored during it's lifetime. Once the task is done, the worker and all data on it will be irreversibly deleted.
|relative or absolute date or date & time
|Start date of the period that will be selected in the datasource. Can be filled with an absolute or relative date. Read more about relative date and time here.
|string, date or date & time
|End date of the period that will be selected in the datasource. Can be filled with an absolute or relative date. Read more about relative date and time here.
|enumerator (year, month, week, day, hour, minute, second, file, list)
|Loop the task depending on the enumerator value. If year, month, week, day, hour, minute or second the loop will add an equal time frame to the start_date until the end_date is reached. If list the loop will cycle through the values in the loop_list. If file the loop will cycle through each file on a data source. This is especially useful when downloading a large data files in many different chunks.
|Contains a list of values to loop through.
|string or int
|Contains the actual value of the loop when loop_by is used. Automatically set by Workflows.
|Starts at 0. Contains the number of the loop. Use in combination with loop_by
|date or datetime
|Set to task.start_date. Only available when loop_by=hour or loop_by=day. When used, task.start_date will be overwritten with the timeframe of the current loop.
|date or datetime
|Set to task.end_date. Only available when loop_by=hour or loop_by=day. When used, task.end_date will be overwritten with the timeframe of the current loop.
|enumerator (0, 1, 2, 4, 8, 16, 32, 64, 128, 256)
|Default is 0. Resource size to use for the task. Number corresponds with the amount of CPU (0 being 0.25). The memory of the instance is 8 x resource_size Gib. E.g. a resource_size of 16 means 16 CPU with 8 x 16 = 128 Gib.