Archives for category: digital marketing

I found another great write up that explains OAuth 2 authentication when accessing many of Google’s API’s.  Raffael Vogler has gone through the pain staking process of setting up a simple script to interact with the Google Analytics API and has documented the steps and what’s going on with specific lines of code.  It’s a good read for those of you still mustering around and trying to get your API calls to work.

Advertisements

Outh-20-LogoIf you are looking to leverage one of the several Google API’s from a server based application, using Service Accounts is the way to go.  In my last post, I had put together a getting started guide to interface with Google’s Content API for Shopping using Python. This was created using an “Installed Application” Client ID requiring a user to grant access to the application as part of the OAuth 2.0 authentication process.  In the case of Server applications such as cron/batch jobs for data based processing tasks, the users who have access to the underlying resource such as a Google Merchant Center or Google Analytics account are not present to grant such access.  This is where Service Accounts come into play as your server based application will call the respective Google API on behalf of the Service Account.

In the next series of posts, we’re going to take the previous program that retrieved a product from a Google Merchant Center account using the Content For Shopping API, and change it to use a Service Account.  Anyone writing applications for this API are most likely going to use Service Accounts to manage their own data feeds and product data.  After watching the Blackhawks trounce the Oilers the other night (sorry Edmonton), I thought it was going to be that easy to change the program up to use a Service Account.   In reality, this supposedly simple switch took countless hours of pain to figure out. I’m hoping this post saves you some time and aggravation.

My setup:

While calling Google API’s from an application using a Service Account can be done across several different languages and platforms, my setup was for Python running on Windows.  However, the concepts are the same and can be adjusted and used as a guide for your specific environment. My setup:

  • Developing on Windows 7
  • Python (version 2.7.8 )
  • Setuptools 7
  • Google API Client Library for Python (version 1.3.1)

Step1: Install Visual C++ Compiler for Python 2.7

The first error I received while when making a call using a Service Account was this:

CryptoUnavailableError: No crypto library available

Newbie mistake. I didn’t install a cryptography module which is needed since Google’s OAuth process uses a private/public key encryption method to verify that the application making the API call is indeed right entity to make that call.  The cryptography module is used to sign something called a JSON Web Tokens (JWT) using a private key supplied by Google.  This signed JWT is used to request an access token from Google’s servers which is then used on each subsequent call to the respective API.   I settled on installing PyCrypto 2.6.1 on my Windows machine but then got the following error when trying to run my test program:

 Unable to find vcvarsall.bat

Now I can’t be the only one who threw up their hands in frustration.  This error message has no meaning.  Fortunately there were some really smart people who posted the answer on Stack Overflow.   For those running on Windows 7, you will need to install the “Microsoft Visual C++ Compilter for Python 2.7“.  This is needed to compile PyCrypto properly.   If you aren’t running on a Windows platform you can skip this step.

Step 2: Installing PyCrypto 2.6.1

As I mentioned earlier, you need to install this Python module to sign JWT’s.  You can install this by typing in the following:

easy_install pycrypto 

Step 3: Installing OpenSSL for Windows

The next error I received when trying to run the program was this:

Error 3: PKCS12 format is not supported by the PyCrypto library. 

NotImplementedError: PKCS12 format is not supported by the PyCrypto library. Try converting to a “PEM” (openssl pkcs12 -in xxxxx.p12 -nodes -nocerts > privatekey.pem) or using PyOpenSSL if native code is an option.

Apparently the private key (PKCS12 format)  you get from Google when creating a Client ID within the Google Developer Console is not supported by the PyCrypto module.   You will need OpenSSL to convert this PKCS12 key to the PEM format which I will explain in the next post once you have downloaded your PKCS12 key from Google.  For now, if you are running on a Windows environment you will need to download OpenSSL:

In my case I downloaded “Win32 OpenSSL v1.0.1j”

openssl_win32

You now have the base underlying modules and tools needed to continue forward.  In my next post, I’ll walk though creating the Service Account and the respective code required to make the respective Google API call from an application using a Service Account.

I’d also highly recommend reading through Google’s documentation on OAuth2, specifically the part on Service Accounts.  This will give you a good overview of how it works: https://developers.google.com/accounts/docs/OAuth2

Managing Google Shopping Campaigns is a key part of any eCommerce retailer’s marketing mix, but is only as successful  as the underlying foundation.  Maintaining clean and accurate product data for Google Merchant Center is critical to ensuring your shopping ads get approved and listed.   Uploading clean product feeds into Google Merchant Center on a periodic basis is a reliable and simple form to ensure information such as promotions, pricing, and stock availability are up to date. Uploading daily feeds for retailers with small product catalogs is a reliable option, but things get more complex for those that have large product catalogs with products that move with high velocity, have constantly changing prices/promotions and stock availability.

shoppingcampaign

To illustrate the point, let’s take the example above of a retailer (let’s call it Superbikes) that has the above Shopping Campaign ad listed.  Assume that Superbikes carries a wide range of bike products that updates Google Merchant Center with a daily product feed.  A customer searching for a  “Shimano Dura-Ace” brake lever from retailer Superbikes may come across their ad between two feed uploads. In that time frame a merchandiser at Superbikes may adjust the price upwards and another customer may have come in and bought up the remaining inventory.  What is now advertised on Google is not in synch with Superbikes product page.  There are several problems with this situation:

  • First and foremost, this a bad experience for the customer. Superbikes’ customer (Miss Fast Bike Racer) ended up wasting time clicking on the ad to find out the price is not what was advertised.  On top of that, the product is currently out of stock which is causing her stress since she needs to get this replacement part for this weekend’s upcoming race.
  • Google wants to ensure the search experience is as best as it can be, and will spot check your product data for accuracy.  They will find problems like this and disapprove the ad.  With enough occurrences, you could risk your Google Merchant Center account being suspended.
  • Google Merchant Center fastest scheduler for data feed uploads is daily.  Products with periodic pricing and stock availability changes within a given day will increase the frequency of problems similar to the example above.

So what is a marketer to do in this situation?  Enter Google’s Content API for Shopping (v2).  Google has updated this API for eCommerce retailers that rely heavily on Shopping Campaigns to allow for tighter product data management.  However, the documentation Google has provided to get started is scattered in several places, making it difficult for a developer new to this API to piece together.  Instructions for installing an older Google Data library called “Google Data APIs Python Client Library” are still lurking around making setup really confusing.  On top of that many of the examples Google has on the respective API site are written for Java or PHP.  If you’re looking for Python, .Net, or Ruby examples you have to dig around and go to another site to download samples.  It took me a while to hook everything up, so hopefully this post will save you some time.

Although the inference of a marketer using an API sounds unrealistic, I’ve put this documentation together in the hopes marketers can learn the basics of how to update product information in near real time.  Your follow-on conversations with your engineering/development team should prove more fruitful with respect to soliciting their help.  I’m also hoping those developers out there will walk away with a better appreciation of the business problem their marketing team is trying to solve and lend a helping hand.   I don’t code as much these days and had to dust off a lot of dust, but I managed to get a small prototype up and running and documented my steps.

Step 1: Take your fellow developer out for coffee

Develop that relationship and take that coffee time to draw out the problem you’re trying to solve.  You need to develop that relationship, especially given how fast the marketing and technology disciplines are converging.  You’ll be surprised to find that your developer team mate will be more than willing to help the business out.  They will appreciate getting involved in the business early.  Plus, they don’t always talk about Star Wars… they have kids too, like to eat at restaurants, and build stuff.

Step 2: Picking a programming language (Python)

Google has taken the liberty to create frameworks around the Content API for Shopping in many programming languages such as .Net, Ruby, Java, PHP, and Python.  These are essentially “wrappers” that help the developer in their native language communicate with Google’s several API’s.  I was a former .Net and Java developer, but elected to try my hand at using Python for this example simply because I’ve heard many good things about Python and it’s an easy to understand syntax.  It also gave me an excuse to learn another language.   I’ve also read that Python has been used to teach programming in many high schools, and given that I haven’t typed a line of code in quite some time it seemed like a good fit (I joined marketing many years ago so my code isn’t going to be rock solid without repeated exposure)

Step 3: Install Python 2.7.x

The instructions outlined below are for installation of Python on a Windows 7 machine.  You can install Python on various operating systems, and can read through the instructions on the python.org web site.

The Content API for Shopping (v2) can only be run from the latest version of Python 2.   Python 3 is available but that’s not going to work for this client library.  Go to the downloads page and download the Windows MSI installer file:

pythondownloads

Startup the installer by opening up the MSI file.  After the installer finishes, add the path where you installed Python to your PATH environment variable.  This just makes it easy to access and run Python from anywhere.  To do that, right click on “Computer” to get access to “Properties” as follows:

path

Once you open up “Properties”, access “Advanced system settings”:

advanedsystemsettings

Once there, access “Environment Variables”:

environmentvars0Once there, scroll down in the “System Variables” window to find the “PATH” variable.  Edit this value to include the directory where Python was installed, specifically where the python.exe file lives.  In addition, add the subdirectory called “Scripts” to the “PATH” environment variable.  On my computer Python is installed in this directory:

C:\Python27

Adding the scripts path, my PATH environment variable now looks like this:

C:\Python27;C:\Python\Scripts

environmentvars

Step 4: Installing the Python package installation manager

You will need something called “setuptools” or “pip” to make it easy to install new Python packages like Google’s Content API for Shopping client.  Go to the setuptools download page and scroll all the way down to bottom. Download and unzip the contents to a temporary directory.

setuptools

Locate the directory where you unpacked the contents and type the following from the command line to install setuptools:

python ez-setup.py

A bunch of stuff will fly by the screen.  Just wait until it’s done.

Step 5: Installing the Google API Client Library for Python (v 1.3.1)

I mentioned earlier that Google released version 2 of their Content API for Shopping.  This service allows you to manage your Google Merchant Center data programmatically.  However your app, website, or script needs to make calls to this service. Google has already done some of the legwork and developed the code to access this service in several languages.  These code frameworks sit on the client side where your application resides and makes things a whole lot easier to interface with the host of Google’s services including the Content API for Shopping.   Since this example is for Python, we’ll be downloading the  “Google API Client Library for Python”.  At the time of this writing, this client side API for Python is on version 1.3.1. This diagram illustrates exactly how everything fits together:

apiclientandserviceflow

The easiest way to install the Google API Client Library for Python is type the following in the command line:

easy_install --upgrade google-api-python-client

I say easier, because it is supposed to be that easy, however I received a few error messages when trying this.  After running some connectivity tests to call the Google Content API for Shopping, things weren’t working. So I downloaded the Google API Client Library for Python and installed it manually.  If the easy_install script doesn’t work, go with the manual installation.  The client library can found on Python’s website.

googleapipythonclient

Alternatively you can grab it from GitHub (Do not to confuse this with installing gdata-python-client which is something different.  There is still documentation out there that points to this gdata client.  There is also documentation out there on Google’s developer websites pointing to an older 1.2 client library version.  Stay away from that as well)

You can find the latest Python client library on GitHub:

github

If the easy_install option mentioned above didn’t work for you, you can manually install this python client.  Unpack the .zip file contents which should contain the necessary “setup.py” file.  Type the following command:

python setup.py install

Step 6: Setting up the Google Developers Console

In order to access any of any of the API’s from Google, you need to login to the Google Developers Console to mark which APIs you are going to use, and set up credential information that will allow your application to connect and make calls to the respective API.  This allows Google to monitor your use against API quotas, some of which come with different pricing tiers based on usage.  The Google Content API for Shopping is free to use up to a certain limit.  We are going to create a project inside the developer console, activate the Content API for Shopping, and create the necessary authentication credentials for our Python application.

Login to to the Google Developer Console, and create a new project (I’ve called mine “My New Google Project”).  You’ll see in the examples below that I’ve blanked out some data for security reasons which I will explain later:

developerconsole1

developerconsole2

After creating project, navigate over to the “APIs” menu option underneath the “APIs & auth” heading:

apis

You’ll notice a number of APIs that Google has enabled for you automatically.  You won’t need any of these for this example, but you can leave them enabled to play with later.  The API to enable is called “Content API for Shopping”.  Scroll down the list of API’s and enable this by clicking on the “OFF” switch:

switch

Once enabled, click inside this API to get familiar with the information inside the “Usage” and “Quota” tabs :

enabledapis

As you test and develop your Shopping applications, these tabs contain useful information about how often you are using the service, the number of errors reported, etc.  The “Quota” tab will you how much of your quota is remaining at the current service level:

quotepage

The next step will be to enable the necessary credentials your Python application will need to request access for a given Google Merchant Center account. Click on the “Credentials” link under the “APIs & auth” heading, and click on the “Create new Client ID” underneath the “OAuth” heading:

consentscreen00

OAuth 2.0 access is required because our example application is looking retrieve and update information on a Google Merchant Center account which isn’t open to everyone.  The owner of the Google Merchant Center account will need to grant the Python application access.

The other type of API access is called “Public API Access” will only allow you to access data that is public and doesn’t require user consent. Public API Access will not allow your Python application to update anything inside Google Merchant Center.

After clicking “Create new Client ID”, you’ll be presented with the “Configure consent screen”

createclientID

Fill in an appropriate email address and “Product Name”, then click the “Save” button.  When the Python application runs and tries to access the Google Merchant Center account, a browser window will open asking the user (in our case, this is the owner of the Google Merchant Center account) to approve the Python Application using this Client ID to access and update data within the Google Merchant Center account.

consentscreen0

Once you click on the “Save” button you will be asked whether to specify an “Application Type” for this ClientID.  There are 3 types:

  • Web Application: If you are creating a web application that accesses the API.
  • Service Account: Good for batch jobs and scripts when a user is not present to grant consent/access to the underlying data.   A service account will call the API on behalf of the application.
  • Installed Application: If you are creating an application for a desktop or phone that will access the API.

For our example, select the “Installed Application” and specify “Other” for the “Application Type”

consentscreen

Once you’ve created your Client ID your Credentials screen should have both the “Client ID” and “Client Secret” displayed. Keep these values safe and secure.  These are needed to acquire “access” and “refresh” tokens when you make calls to the Content API for Shopping.  Anyone with these values will be able to access and make calls to your Google API’s messing with your quotas.  Just keep them safe.  You don’t want an unknown outside party messing around with your quotas or requesting access to stuff that belongs to you.

Click on the “Download JSON” button to download the credential information and save the file as “client_secrets.json”. This file is needed later to run the examples.

downloadjson

If you want to learn more about managing the Google Developer Console you can find that here.

Step 7: Understand OAuth2.0

Before you can access data requiring consent from any Google API, a clear understanding of how OAuth 2.0 authentication works is required. There is a good explanation that walks through the authentication process using Python that I highly recommend you read and understand before moving on.  Don’t rush through this as taking the time to understand this now will save you time later.

Essentially there are 3 parties involved in this authentication process:

  1. The Google Content API For Shopping service
  2. Your application
  3. The user who owns the Google Merchant Center account

The user (#3, the owner of the Google Merchant Center account) needs to grant your application that is trying to access it using a unique Client ID (#2) to access data in their Google Merchant Center account via the Content API (#1).   Someone at Google also created a very good presentation that clearly explains the entire authentication process using OAuth 2.0. (thank you, whoever you are!).

I’d also recommend reading this resource on how OAuth 2.0 works. It explains in detail the entire flow for a Web application, installed application, and a service account.

The next step is to test connectivity to make sure you have everything installed and configured properly.

Step 8: Testing connectivity to make sure everything was installed properly

There are a number of examples on accessing the Content API for Shopping on GitHub

Download the example called “product_list.py” and save it to the same location where you saved the “clients_secrets.json” file.  This “product_list.py” program will look for that file to get credential information and ill print out all the products in your Google Merchant Center account.

IMPORTANT NOTE: The first time I ran a Python client example I received a weird error message:

NotImplementedError: The gflags library must be installed to use tools.run(). Please install gflags or preferrably switch to using tools.run_flow()

I had no idea what this meant and figured I must have installed something incorrectly.  Luckily someone posted the answer on StackOverflow.  I would just go ahead install this “gflags” library.  You can do that by typing this in on the command prompt:

 easy_install –upgrade python-gflags

Back to the “product_list.py” example.  If you are logged into any Google account on your browser, now is the time to logout. The moment you kick off this program, it will open up a browser asking you to login and grant access to the Google Merchant Center account.    Kick off the program supplying a command line parameter with your Merchant Center ID:

python product_list.py [your Google Merchant Center ID]

The moment this program starts up, you will see this:

example1

Your browser will open up asking you to login.  Since you are trying to list products from your Google Merchant Center account, login with the Google account that has access.  The consent screen you configured from Step 6. will now show up as follows:

example2

Switch hats for a second and read this consent screen as if you were the owner of the Google Merchant Center account.  The entire OAuth process is in full swing and asking you as the owner, whether you grant access to the program/application in question.  Click “Accept”

After clicking accept, the command line window should output a message stating the Authentication was successful and proceed with printing out all the products in your Google Merchant Center Account.

Voila!  You just established connectivity to the Content API for Shopping using the Google API Client Library for Python:

example3

The entire “flow” process for authentication as Google calls it also writes out a file named “content.dat” using a “Storage” object.  You will see this new file created in the directory you ran the program from.  This “content.dat” file contains access tokens that grants your application certain access (called “scope”) to Google Merchant Center data.  The file also contains expiration data for these access tokens.

Step 9: Retrieving a product from Google Merchant Center

Now that you have all that busy work of configuration and setup out of the way, you can now focus on the real meat… how to best manage your your Google Merchant Center data and keep things fresh using this API.  In this first example we are simply going to retrieve information for a single product.  The GitHub site that I mentioned earlier has many examples, but doesn’t have one to retrieve a product.  We’re going to create a new Python program to do just that.  You can download the source code file “product_get.py” here .  The sample program takes two parameters as follows:

python product_get.py [Your Merchant Center ID] [product ID]

For example, to retrieve the product 888999 from Merchant Account 111222 you would enter the following:

python product_get.py 111222 888999

The program will print out the Product ID, Price, Title, and Stock Availability for that product.

Let’s walk through some of the important parts of the code to understand what’s going on.  In the code below we are retrieving a “Products” resource and setting things up for the “get” method by passing in the Merchant ID and the Product ID.  The Product ID you need to pass in, is not just a SKU number.  In my example you’ll notice the ProductID is prefixed with “online:en:US”.  To be honest, I don’t know exactly what this means, but I am presuming this is to uniquely identify the product to be listed online, for a Google Merchant Center account configured for English, and for the US market.  I just know the “get” method won’t work without this prefix:

request = service.products().get(merchantId=merchant_id, productId=”online:en:US:” + product_id)

Once you‘ve set things up, the next line of code actually makes the call and stores the product object represented in JSON inside “result”.

result = request.execute()

To access attributes of the product such as stock availability you can reference it through “result”:

print “Product Stock Availability:” + result[‘availability’]

Much of the data attributes you’re sending up via the data feed specification for Google Merchant Center can be found in the product object that gets returned.  You can find out more about what exactly is in this response from the following documentation resource.

Important Reminder:  You’ve already granted access to your client ID being used by this program from Step 8.  The access token is stored in the content.dat file that was created.  This new “product_get.py” program will look for this file and use this access token for each Content API for Shopping call.  You won’t have to go through the consent screen and granting access process again.  Do not share your client ID, client secret, and this content.dat file and ensure this information is secure.

Step 10: Updating stock availability and price for a single product.

There are a few ways to update product information.  This can be done using either the inventory.set method or the products.insert method.  We’re going to use the inventory.set method for this example.  I am choosing this for performance reasons, because the inventory.set method updates only those values that you want to change.  On the other hand, the products.insert method requires that you send in the full product object that may contain data attributes that you don’t necessarily need to update.

Create a new status value specifying the new price and product availability. In this example I am setting a price of $3.99 and stock availability status of “in stock”:

new_status = {

‘price’: {‘currency’: ‘USD’, ‘value’: 3.99},

‘availability’ : ‘in stock’}

Setup the new inventory resource providing the Merchant ID, storeCode, Product ID, and the new_status variable.  You might be asking what is storeCode?  If you remember the Product ID takes on the form of: “online:en:US:[your product identifier/sku]”.  The 3rd line in the code below splits that value into an array and grabs the first item which has the value of “online”.  I’m not sure what other possible values are for storeCode but this value is required.

request = service.inventory().set(

merchantId=merchant_id,

storeCode=product_id.split(‘:’)[0],

productId=product_id,

body=new_status)

Finally, make the call:

result = request.execute()

More information about the inventory.set method can be found here.

Other Learning Resources

That’s it.  I’m hoping this sets you on a journey to get more products listed, up to date, and converting into sales for your Shopping Campaigns.  Don’t let your product data get stale!   Your customers won’t like it, and Google won’t like it either.

Here are other resources that will help you along the way:

%d bloggers like this: