Workflows · Repository structure

Your Workflows account comes with a Github repository. This section explains the structure.

Repository structure

Repository
+- global.yml
+- global.sandbox.yml
+- includes
    +- sql_template_1.sql
    +- sql_template_2.sql
+- jobs
    +- job_name_1
        +- job.yml
        +- some_task_1.yml
        +- etc.
    +- job_name_2
        +- job.yml
        +- some_task_1.yml
        +- etc.

Includes folder

The includes folder contains all template files. These files can be called by the different tasks. They are accessible by all jobs. You are allowed to create folders to structure your template files. Just make sure you use the same folder names in the inclusion of template files.

The template files are used for SQL templating. The templating engine used is Jinja2.

Jobs folder

The jobs folder contains all jobs. You add a job simply by adding a folder. The job will appear in Workflows with the exact same name as the folder. Please only use alphanumeric characters and an underscore(_). Every job needs at least 1 job.yml file. This file contains important meta information about your data job and is explained in the jobs section. Please refer to the Jobs documentation for more information.

global.yml and global.sandbox.yml

The global.yml file contains properties that will be set in all jobs and tasks. Typical properties to be set in the global.yml file are below. Please refer to the Jobs documentation for more information about the properties.

job:
    depends_on_past: yes
    backfill: no
    owner: onesecondbefore
    email:
        - admin@onesecondbefore.com
    email_on_failure: no
    email_on_retry: no
    retries: 0
    retry_delay: 5
    schedule: '@daily'
    max_active_runs: 2
    environment: production
    log_level: 20
    task:
        resource_size: 0
        slack_conn_id: slack
        slack_on_failure: no

    client_cloud:
        timezone: Europe/Amsterdam