Integration testing Pub/Sub interactions in your application

As part of the Google Cloud Platform (GCP), Google provides Pub/Sub as its queuing mechanism. After creating a Pub/Sub topic in GCP, you can create publishers to send messages to a topic and subscribers to receive messages from a topic. In order to send or receive Pub/Sub messages from GCP, you can choose to authenticate with GCP through user account or service account credentials.

This has one disadvantage; when running your build either locally or on a build server such as Gitlab, your application will attempt to communicate with GCP for its Pub/Sub interactions. Your build will either fail because there are no credentials available (usually the case on build servers), or it will use you personal user account credentials and publish messages to topics that may also be in use on other (test) environments. Both situations are not desirable.

In this article, I will first describe how to Dockerize Pub/Sub, followed by the changes required in your application. If you want to dive right into the code, go here. You can also find a prebuilt Docker image here.

Dockerizing the Pub/Sub server

In order to still allow for a build that includes tests with Pub/Sub interactions, we are going to Dockerize the Pub/Sub server. With this in place, we don't need to authenticate with GCP anymore and can actually verify the messages that go through our topics and subscriptions without interfering on other environments. Google already provides us with an emulator that allows us to emulate a Pub/Sub server locally. Our Dockerfile is as follows:

FROM google/cloud-sdk:272.0.0

RUN git clone /python-docs-samples && \
    cd /python-docs-samples/pubsub/cloud-client && \
    pip install -r requirements.txt

RUN mkdir -p /root/bin
COPY /root/bin/


CMD <"./root/bin/">

As per the emulator's installation instructions, we clone the Google repository and install the Pub/Sub requirements. Afterwards, we execute


if << -z "${PUBSUB_PROJECT_ID}" >>; then
  echo "No PUBSUB_PROJECT_ID supplied, setting default project name"
  export PUBSUB_PROJECT_ID=gcp-docker-project

export PUBSUB_EMULATOR_HOST=localhost:8432

# Start the emulator in the background so that we can continue the script to create topics and subscriptions.
gcloud beta emulators pubsub start --host-port= &

if << -z "${PUBSUB_CONFIG}" >>; then
  echo "No PUBSUB_CONFIG supplied, no additional topics or subscriptions will be created"
  python /root/bin/ ${PUBSUB_PROJECT_ID} "${PUBSUB_CONFIG}"

# After these actions we bring the process back to the foreground again and wait for it to complete.
# This restores Docker's expected behaviour of coupling the lifecycle of the Docker container to the primary process.
echo "Ready"
wait ${PUBSUB_PID}

This script indicates that we can provide the Docker container with two optional environment variables:

  • PUBSUB_CONFIG: a JSON array that describes the topics and associated subscriptions you want to create.

With this script, we start the Pub/Sub server at http://localhost:8432 and execute a simple Python script ( that interprets the JSON object.

Building and running the Dockerfile can be done as follows:

cd src/test/resources/docker

docker build . -t pubsub

docker run  --name pubsub \
            -p 8432:8432 \
            -e PUBSUB_PROJECT_ID=my-gcp-project \
            -e PUBSUB_CONFIG='<{"name": "my-topic", "subscriptions": <"my-subscription">}>' \
            -d pubsub

docker logs -f pubsub 

Making the application configurable

What remains is configuring the publisher and subscriber Java classes to connect with the locally running Pub/Sub server. Note that the code examples below use the configuration object PubSubConfig and that the examples are not 100% complete. Please refer to the code repository for the complete listing.

Given a topic with the name my-topic and a subscription with the name my-subscription, the minimal setup of a publisher and subscriber is as follows:

Publisher publisher = Publisher
Subscriber subscriber = Subscriber
    .newBuilder(subscriptionName, messageReceiver)

Both builder objects can take a CredentialsProvider instance that determines how we authenticate with GCP.

I've created my own CredentialsProviderFactory that returns either no credentials, user credentials or service account credentials based on the Spring property gcloud.authentication.method. If you are using service account credentials, you will also have to set the property gcloud.serviceaccount.credentials.file, which is a reference to the JSON file that contains the actual credentials. This JSON file can be retrieved from GCP in the IAM section.

    public CredentialsProvider create() {
        switch (pubSubConfig.getAuthenticationMethod()) {
            case NONE:
                return getNoCredentialsProvider();
            case USER_ACCOUNT:
                return getUserCredentialsProvider();
            case SERVICE_ACCOUNT:
                return getServiceAccountCredentials();
                throw new IllegalArgumentException("Unexpected authentication method " + pubSubConfig.getAuthenticationMethod());

    private CredentialsProvider getNoCredentialsProvider() {
        return NoCredentialsProvider.create();

    private CredentialsProvider getUserCredentialsProvider() {
        try {
            return FixedCredentialsProvider.create(GoogleCredentials.getApplicationDefault());
        } catch (IOException e) {
            throw new UncheckedIOException(e);

    private CredentialsProvider getServiceAccountCredentials() {
        try (InputStream stream = Files.newInputStream(Paths.get(pubSubConfig.getServiceAccountCredentialsFile()))) {
            ServiceAccountCredentials credentials = ServiceAccountCredentials.fromStream(stream);
            return FixedCredentialsProvider.create(credentials);
        } catch (IOException e) {
            throw new UncheckedIOException(e);

Lastly, you will have to provide both the publisher and subscriber builder objects with a TransportChanneProvider. This provider will allow us to instruct Pub/Sub to use a plain text connection when we are running our application against the Dockerized Pub/Sub server. Furthermore, it also allows us to set the URL of the Pub/Sub server. This has been made configurable through the gcloud.pubsub.url property, which is set to localhost:8432 by default, but should be set to when running in GCP.

    public TransportChannelProvider create() {
        ManagedChannelBuilder<?> channelBuilder = ManagedChannelBuilder.forTarget(pubSubConfig.getPubSubUrl());
        if (AuthenticationMethod.NONE.equals(pubSubConfig.getAuthenticationMethod())) {
        ManagedChannel channel =;
        return FixedTransportChannelProvider.create(GrpcTransportChannel.create(channel));

Different scenarios, different settings

The table below shows the different scenarios that you can have when working with Pub/Sub and the appropriate property settings that go with each scenario:

Application running at
Pub/Sub running at

You can also run the and publish a message with the use of the Swagger UI or checkout this Pub/Sub integration test.

In conclusion

With this setup you can setup integration tests that run for every build in your pipeline, or you can use it to locally debug any issues you may have with Pub/Sub without having to rely on the cloud. If required, all it takes is changing a few properties to start listening to your actual GCP Pub/Sub queues. This allows for maximum flexibility when developing with Pub/Sub and will increase the trust you have in a correct functioning of your application.

Mark Krijgsman

All articles by me