Getting Data From Google Adwords to Google Bigquery using Singer- Part 2
Okay now we have the pipeline running, we want to make sure that the state.json file is stored somewhere safe since we might opt to destroy a VM instance at some time, it is best to store the files in google cloud storage. Here are the steps we will take in this file:- a) Creating a Docker file b) Create an ENTRYPOINT script, that will run every time the docker image is run and will run the pipeline and store/load the state.json file from google cloud storage. Creating the Entry.sh file Here is the Bash file that will run our pipeline. Here is the explanation about the code:- a) We authenticate gcloud service account so that we can use gsutil to copy state.json file from google cloud storageb) If the file exists we set the state environment variable to point to the file otherwise it is kept emptyc) Run the[…]
Read More
No comments yet.
Add your comment