Your Transfer account comes with a Github repository. This section explains the structure.
Repository +- global.yml +- global.sandbox.yml +- includes +- sql_template_1.sql +- sql_template_2.sql +- jobs +- job_name_1 +- job.yml +- some_task_1.yml +- etc. +- job_name_2 +- job.yml +- some_task_1.yml +- etc.
The includes folder contains all template files. These files can be called by the different tasks. They are accessible by all jobs. You are allowed to create folders to structure your template files. Just make sure you use the same folder names in the inclusion of template files.
The template files are used for SQL templating. The templating engine used is Jinja2.
The jobs folder contains all jobs. You add a job simply by adding a folder. The job will appear in Transfer with the exact same name as the folder. Please only use alphanumeric characters and an underscore(_). Every job needs at least 1 job.yml file. This file contains important meta information about your data job and is explained in the jobs section. Please refer to the Jobs documentation for more information.
The global.yml file contains properties that will be set in all jobs and tasks. Typical properties to be set in the global.yml file are below. Please refer to the Jobs documentation for more information about the properties.
job: depends_on_past: yes backfill: no owner: onesecondbefore email: - email@example.com email_on_failure: no email_on_retry: no retries: 0 retry_delay: 5 schedule_interval: '@daily' max_active_runs: 2 environment: production log_level: 20 task: resource_size: 0 slack_conn_id: slack slack_on_failure: no client_cloud: timezone: Europe/Amsterdam