Archive - April 1, 2020

1
Scheduling a Singer pipeline on Google Cloud – Part 3 Adwords to Bigquery

Scheduling a Singer pipeline on Google Cloud – Part 3 Adwords to Bigquery

If you have followed the tutorial you will have a docker image with your stitch pipeline in Google container registry. For running this setup we will be implementing the following setup in Google Cloud. Here are the tasks we will need to complete the setup:- Create a VM instance to get the historical data Create a pub/sub topic to trigger cloud functions Create Google cloud functions to start and stop the instance Setup cloud schedular jobs Create a VM instance to get the historical data The strategy I am following, in this case, is that I am downloading all the historical data till the current data and store the current date in the state.json file and everyday I run the cronjob to get data of the previous data into Bigquery. We will set up a containerized compute engine using the docker image we created in the previous article, the reason[…]

Read More

Copyright © 2020. Created by Meks. Powered by WordPress.