Archive - March 31, 2020

Okay now we have the pipeline running, we want to make sure that the state.json file is stored somewhere safe since we might opt to destroy a VM instance at some time, it is best to store the files in google cloud storage. Here are the steps we will take in this file:- a) Creating a Docker file b) Create an ENTRYPOINT script, that will run every time the docker image is run and will run the pipeline and store/load the state.json file from google cloud storage. Creating the Entry.sh file Here is the Bash file that will run our pipeline. Here is the explanation about the code:- a) We authenticate gcloud service account so that we can use gsutil to copy state.json file from google cloud storageb) If the file exists we set the state environment variable to point to the file otherwise it is kept emptyc) Run the[…]

When I was working on this problem, I found the documentation to be lacking in a lot of aspects. So, I decided to cover this in detail. Here are the steps we will follow:- a) Getting the catalog.json file ready for tap-adwords b) Getting data into target-bigquery c) Working with state.json file for tap-adwords Creating a Config.json file for tap-adwords Singer recommends different virtual environments for both the tap and the target. So, first step is to create a new virtual environment where we will install tap-adwords. virtualenv -p python3 env-tap source env-tap/bin/activate pip3 install tap-adwords The first this you need to run tap-adwords is a config file. So here is from where you get all these values from, start date and end date are pretty self-explanatory:- a) Developer token- You can get a developer token by logging in to your Google AdWords manager account and navigating to the AdWords API Center. b) OAuth Client[…]