We often build server applications that talk to other backend services, mostly over HTTP. Such an application essentially receives requests, collects data from other services and transforms them into a response.
How to test communication with external, 3rd party services?
Christoph Dähne17.10.2024
How do you test such applications?
The tests should run offline and deterministic. Hence, calling the external services when running the tests is no option. This would make the tests non-deterministic and slow. We might even run into rate-limits and fraud-detection related issues. So, we have to emulate the external services in some manner.
Here are the most usual option, I am aware of. You can use several in one project of course.
Option 1: Start the external service locally
Sometimes you can start the service in a Docker container or similar. For example, you can start Mailpit to test outgoing emails, or Keycloak for OpenID authentication.
I prefer this approach when the external service is publicly available, quick to start, easy to configure and the API rather stable.
services:
postgres:
image: postgres
volumes:
- sso_postgres_data:/var/lib/postgresql/data
environment:
POSTGRES_DB: mydb
POSTGRES_USER: myuser
POSTGRES_PASSWORD: mypassword
ports:
- "5432:5432"
mailpit:
image: axllent/mailpit
ports:
- "8025:8025"
- "1025:1025"
Option 2: Hide the communication behind strategies
You can use the Strategy pattern to replace the communication with external services by local, predefined responses. Most of your own code is still executed during tests, the part sending the requests and receiving the responses is not. Also, your code becomes more complex, since you have to add the strategy pattern and provide mock implementations for testing.
I prefer this approach if the code already contains a strategy pattern, like storage in database, cloud or on disk. Then I like to tests the storage strategies individually and add an in-memory storage for other tests.
Option 3: Implement service mocks
You can implement a simplified version of each service which listens on a socket. They behave close enough to the real services (hopefully) for your tests. You would have to validate incoming requests and generate realistic responses.
If you have few services with small and simple interfaces I like this approach. Also, it allows to inject errors (500 internal server errors) and such to test error handling and recovery.
var handler http.HandlerFunc
handler = func(writer http.ResponseWriter, request *http.Request) {
writer.WriteHeader(http.StatusInternalServerError)
writer.Write([]byte("internal server error"))
}
server := httptest.NewServer(handler)
defer server.Close()
url := server.URL
Option 4: Request/Response recording and replay
This option is similar to option 3, but with more automation. With libraries like go-vcr you can automatically record and replay calls to external services. You have to replace the HTTP client during tests by a proxy.
While writing tests, the HTTP client proxy behaves like a production one: send the request and receive the response. Both are written to disk. While running tests, the HTTP client proxy uses the information on disk to find a response for each request. No network communication happens.
Additionally, you can review the external communication in the recording file.
One remark though: beware of secrets.
More often than not, the 3rd party service requires an API key, access tokens or similar.
Do not forget to remove them from the recording file before committing it to the repository.
Here is a small example how to do that in go-vcr:
package yourtests
import (
"crypto/sha256"
"fmt"
"gopkg.in/dnaeon/go-vcr.v3/cassette"
"gopkg.in/dnaeon/go-vcr.v3/recorder"
"regexp"
"strings"
)
// …
func CreateRecorder() {
// …
replaceSecretsHook := func(interaction *cassette.Interaction) error {
// in requests
secrets := []string{
"X-Api-Key",
"Authorization",
"client_secret",
"password",
"id_token",
"access_token",
"refresh_token",
"subject_token",
}
for _, secret := range secrets {
if interaction.Request.Headers.Get(secret) != "" {
value := interaction.Request.Headers.Get(secret)
if strings.HasPrefix(value, "Bearer") {
interaction.Request.Headers.Set(secret, "Bearer "+hashSecret(value[7:]))
} else {
interaction.Request.Headers.Set(secret, hashSecret(value))
}
}
if interaction.Request.Form.Get(secret) != "" {
interaction.Request.Form.Set(secret, hashSecret(interaction.Request.Form.Get(secret)))
}
patterns := []string{
fmt.Sprintf(`"%s":"([^"]*)"`, secret), // json
fmt.Sprintf(`%s=([^&]*)`, secret), // form
}
for _, pattern := range patterns {
interaction.Request.Body = replaceSecret(interaction.Request.Body, pattern)
interaction.Response.Body = replaceSecret(interaction.Response.Body, pattern)
}
if interaction.Response.Headers.Get(secret) != "" {
interaction.Response.Headers.Set(secret, hashSecret(interaction.Response.Headers.Get(secret)))
}
}
// we do not want to replay slowness during tests
// timeouts are tested separately
interaction.Response.Duration = 1 * time.Millisecond
return nil
}
r.AddHook(replaceSecretsHook, recorder.BeforeSaveHook)
}
// replaceSecret replaces secrets in a string with a deterministic hash
// Uses the given pattern to find and replace secrets in the target string.
// The pattern must contain a single capturing group that matches the secret.
func replaceSecret(target string, pattern string) string {
findRegexp := regexp.MustCompile(pattern)
matches := findRegexp.FindAllString(target, -1)
for _, match := range matches {
secret := findRegexp.FindStringSubmatch(match)[1]
target = strings.ReplaceAll(target, secret, hashSecret(secret))
}
return target
}
// hashSecret deterministic hash of a secret
// Obfuscates the secret and replaces it by a deterministic value.
// This way we can still match the request/response but do not expose the secret in the cassette.
// !!! Allows brute-force attacks and cannot protect week passwords !!!
func hashSecret(secret string) string {
longHash := sha256.Sum256([]byte(strings.ToLower(secret)))
shortHash := make([]byte, 4)
for i := 0; i < 4; i++ {
shortHash[i] = longHash[i] ^ longHash[4+i] ^ longHash[8+i] ^ longHash[16+i] ^ longHash[20+i] ^ longHash[24+i]
}
return fmt.Sprintf("%x", shortHash)
}
I prefer this approach, if the application under test communicates a lot with various external system.
As always, thanks for reading and feel free to share your opinion with us.