Workflows · From Tasks · from_bigquery


Imports data from Google BigQuery. If you have an Google Cloud configured as your client_cloud you should probably use the do task.

Method of use

Before you can download data from Google BigQuery you have to give Workflows acces. Take the steps below to download data with a service account.

  1. Create a service account or re-use a service account of a Google Cloud environment.
  2. Give the service account BigQuery Editor permissions to the BigQuery datasets.
  3. Have the service account added to the Workflows connections by a Onesecondbefore staff member
  4. If you use Google Cloud, you're done. If you use another cloud, add the connection string to conn_id


Example usage

    query: SELECT 1 AS int_field, '{{ task.run_id }}' AS string_field, CURRENT_TIMESTAMP() AS timestamp_field
    db_conn_id: google_cloud_default
    storage_conn_id: google_cloud_default
    project_id: my-project-id
    dataset_id: my-dataset-id-for-temp-table
    bucket: bucket-to-extract-tmp-files-to
    folder: folder-in-bucket-to-extract-tmp-files-to


querystringnoUse either query or template (below). Query to be executed, whose results will be uploaded to the destination.
templatestringnoUse either query(above) or template. Contains a link to a file in the `includes` folder in your repository that contains the SQL statement. This query will be executed and the results will be uploaded to the destination.
db_conn_idstringnoConnection to use to connect to the BigQuery database
db_locationenumerator(bigquery query locations)noQuery location. Default is EU. Currently only supported in Google BigQuery.
storage_conn_idstringnoConnection to use to connect to Google Cloud Storage.
project_idstringnoProject ID for table extract to Google Cloud Storage.
dataset_idstringnoDataset ID in which the temporary table will be created. The table id will be _tmp_{{ task.run_id }}
bucketstringnoBucket to which the data will be extracted from the temporary table.
folderstringnoFolder to which the data will be extracted from the temporary table.


Pre-formatted schemaBigQuery uses the schema of the query. Use config schema if you want to add descriptions.