Integration testing Pub/Sub interactions in your application

On 18-02-2020
Category: Development

As part of the Google Cloud Platform (GCP), Google provides Pub/Sub as its queuing mechanism. After creating a Pub/Sub topic in GCP, you can create publishers to send messages to a topic and subscribers to receive messages from a topic. In order to send or receive Pub/Sub messages from GCP, you can choose to authenticate with GCP through user account or service account credentials.

This has one disadvantage; when running your build either locally or on a build server such as Gitlab, your application will attempt to communicate with GCP for its Pub/Sub interactions. Your build will either fail because there are no credentials available (usually the case on build servers), or it will use you personal user account credentials and publish messages to topics that may also be in use on other (test) environments. Both situations are not desirable.

In this article, I will first describe how to Dockerize Pub/Sub, followed by the changes required in your application. If you want to dive right into the code, go here. You can also find a prebuilt Docker image here.

Dockerizing the Pub/Sub server

In order to still allow for a build that includes tests with Pub/Sub interactions, we are going to Dockerize the Pub/Sub server. With this in place, we don’t need to authenticate with GCP anymore and can actually verify the messages that go through our topics and subscriptions without interfering on other environments. Google already provides us with an emulator that allows us to emulate a Pub/Sub server locally. Our Dockerfile is as follows:

As per the emulator’s installation instructions, we clone the Google repository and install the Pub/Sub requirements. Afterwards, we execute start-pubsub.sh:

This script indicates that we can provide the Docker container with two optional environment variables:

  • PUBSUB_PROJECT_ID: GCP project ID
  • PUBSUB_CONFIG: a JSON array that describes the topics and associated subscriptions you want to create.

With this script, we start the Pub/Sub server at http://localhost:8432 and execute a simple Python script (pubsub-configuration-parser.py) that interprets the JSON object.

Building and running the Dockerfile can be done as follows:

Making the application configurable

What remains is configuring the publisher and subscriber Java classes to connect with the locally running Pub/Sub server. Note that the code examples below use the configuration object PubSubConfig and that the examples are not 100% complete. Please refer to the code repository for the complete listing.

Given a topic with the name my-topic and a subscription with the name my-subscription, the minimal setup of a publisher and subscriber is as follows:

Both builder objects can take a CredentialsProvider instance that determines how we authenticate with GCP.

I’ve created my own CredentialsProviderFactory that returns either no credentials, user credentials or service account credentials based on the Spring property gcloud.authentication.method. If you are using service account credentials, you will also have to set the property gcloud.serviceaccount.credentials.file, which is a reference to the JSON file that contains the actual credentials. This JSON file can be retrieved from GCP in the IAM section.

Lastly, you will have to provide both the publisher and subscriber builder objects with a TransportChanneProvider. This provider will allow us to instruct Pub/Sub to use a plain text connection when we are running our application against the Dockerized Pub/Sub server. Furthermore, it also allows us to set the URL of the Pub/Sub server. This has been made configurable through the gcloud.pubsub.url property, which is set to localhost:8432 by default, but should be set to pubsub.googleapis.com when running in GCP.

Different scenarios, different settings

The table below shows the different scenarios that you can have when working with Pub/Sub and the appropriate property settings that go with each scenario:

Application running at Pub/Sub running at gcloud.pubsub.url gcloud.authentication.method gcloud.serviceaccount.credentials.file
localhost localhost localhost:8432 NONE N/A
localhost GCP pubsub.googleapis.com USER_ACCOUNT N/A
localhost GCP pubsub.googleapis.com SERVICE_ACCOUNT /path/to/credentials.json
GCP GCP pubsub.googleapis.com SERVICE_ACCOUNT /path/to/credentials.json

You can also run the Application.java and publish a message with the use of the Swagger UI or checkout this Pub/Sub integration test.

In conclusion

With this setup you can setup integration tests that run for every build in your pipeline, or you can use it to locally debug any issues you may have with Pub/Sub without having to rely on the cloud. If required, all it takes is changing a few properties to start listening to your actual GCP Pub/Sub queues. This allows for maximum flexibility when developing with Pub/Sub and will increase the trust you have in a correct functioning of your application.