Search

gRPC Support

  1. Clients
  2. Servers
  3. Retrying gRPC Failures
  4. Propagating Priorities
  5. Fallbacks
  6. Context Cancellation

Failsafe-go makes it easy to use any policies with gRPC.

Clients

You can create a failsafe UnaryClientInterceptor for a policy composition to handle errors, limit load, or limit execution time:

interceptor := failsafegrpc.NewUnaryClientInterceptor[SomeResponse](retryPolicy, circuitBreaker)
client, err := grpc.NewClient(target, grpc.WithUnaryInterceptor(interceptor))

// Perform an RPC with retries and circuit breaking
service := somepkg.NewSomeServiceClient(client)
service.DoSomething(ctx, request)

Servers

On the server side, you can use load limiting or time limiting policies to create a failsafe ServerInHandle:

inTapHandle := failsafegrpc.NewServerInHandle[any](bulkhead, timeout)
server := grpc.NewServer(grpc.InTapHandle(inTapHandle))

You can also create a failsafe UnaryServerInterceptor:

interceptor := failsafegrpc.NewUnaryServerInterceptor[SomeResponse](retryPolicy, circuitBreaker)
server := grpc.NewServer(target, grpc.UnaryInterceptor(interceptor))

The difference between these two approaches is that a ServerInHandle handles a request before it has been decoded whereas a UnaryServerInterceptor allows your policies to handle the contents of a response.

For most load limiting use cases, prefer ServerInHandle since it does not create additional server side resources for requests that are rejected. For use cases where a policy needs to inspect the response, such as a Fallback or a CircuitBreaker, you can use a UnaryServerInterceptor.

Retrying gRPC Failures

The failsafegrpc package provides a NewRetryPolicyBuilder that can build retry policies with built-in detection of retryable gRPC errors, including Unavailable, DeadlineExceeded, and ResourceExhausted:

retryPolicy := failsafegrpc.NewRetryPolicyBuilder().
  WithBackoff(time.Second, 30*time.Second).
  WithMaxRetries(3).
  Build()

Propagating Priorities

When using policies that support execution prioritization, ideally priorities and levels should be propagated from gRPC clients to servers, and on to the server’s handler. On the client, we can propagate priority or level information from a context through an outgoing request by including an interceptor:

interceptor := failsafegrpc.NewUnaryClientInterceptorWithLevel()
client, err := grpc.NewClient(target, grpc.WithUnaryInterceptor(interceptor))

And on the server, we can decode priority or level information from an incoming request, optionally generate a level if one does not exist but a priority does, and propagate it to the handling context:

interceptor := failsafegrpc.NewUnaryServerInterceptorWithLevel(true)
server := grpc.NewServer(target, grpc.UnaryInterceptor(interceptor))

For distributed systems, you typically want to generate a level, if one does not exist, at the edge of your system, and then propagate the same level for all sub-requests.

Fallbacks

When using a Fallback with a UnaryClientInterceptor, it’s necessary to set your fallback value against the execution’s LastResult() since a UnaryClientInterceptor has no way to return an alternative result:

fb := fallback.WithFunc(func(exec failsafe.Execution[*SomeResponse]) (*SomeResponse, error) {
  exec.LastResult().Msg = "fallback"
  return nil, nil
})

A Fallback with a UnaryServerInterceptor can be used as usual since UnaryServerInterceptor does allow an alternative result to be returned:

fb := fallback.WithResult(&SomeResponse{Msg: "fallback"})

Context Cancellation

When using Failsafe-go’s gRPC support, Context cancellations are automatically propagated to the RPC context. When an execution is canceled for any reason, such as a Timeout, any outstanding RPC’s context is canceled. Similarly, when using a HedgePolicy, any outstanding hedge RPC contexts are canceled once the first successful response is received.