Batch Requests

The "N+1 problem" is a common challenge in GraphQL APIs that can lead to performance issues. You start with an initial request (1), that returns a list of objects. The size of the list determines how many follow-up requests (N) are necessary. The exact number of follow-up requests is not known until the first request is executed.

Consider a simple e-commerce API that provides a list of reviews, each with a rating and a reference to a product. The product is an entity that can be fetched separately.

1type Query {
2  reviews: [Review] # A list!
3    @connect(
4      source: "ecom"
5      http: { GET: "/reviews" }
6      selection: "id rating product: { id: productId }"
7    )
8}
9
10type Review {
11  id: ID!
12  rating: Int!
13  product: Product
14}
15
16type Product
17  @connect(
18    source: "ecom"
19    http: { GET: "/product/{$this.id}" }
20    selection: "id name"
21  ) @key(fields: "id") {
22  id: ID!
23  name: String
24}

In the Connectors Debugger, you can see that fetching the list of reviews results in a separate request for each product associated with the review.

To solve the N+1 problem, you can use Connectors batching. With batching, a single request is made for all N items in the list.

1type Product
2  @connect(
3    source: "ecom"
4    # Use the $batch variable to make a batch request for products
5    http: { GET: "/product", queryParams: "id: $batch.id" }
6    selection: "id name"
7  ) {
8  id: ID!
9  name: String
10}

This reduces the total number of requests to two: one for the list of reviews and one for the list of products.

Fetching entities in batches

Using Connectors batching requires an HTTP endpoint that accepts a list of keys and returns a list of items. This is often referred to as a batch endpoint. There are a few common patterns for batch endpoints and how to use the Connectors $batch variable to make batch requests:

Repeated query parameters

Using repeated query parameters, each ID is sent as a separate query parameter in the HTTP request, for example, ?id=1&id=2.

Use the http.queryParams mapping to specify the query parameter name and the value from $batch.id. The $batch.id variable is a list of IDs that will be sent in the request.

http: { GET: "/" queryParams: "id: $batch.id" }

Comma-separated list

Using a comma-separated list, all IDs are passed in a single query parameter, for example, ?ids=1,2. Commas will be percent-encoded: ?ids=1%2C2.

Use the joinNotNull mapping method to join the IDs into a single string, ignoring any null values. Interpolate the $batch.id variable in the http.GET mapping to create the query parameter.

http: { GET: "?ids={$batch.id->joinNotNull(',')}" }

JSON request body

Using a JSON request body, all IDs are passed in as a filter criteria in the JSON payload, for example, { "ids": [1, 2] }.

Use the http.body mapping to specify the request body. The $batch.id variable is used to access the list of IDs.

http: { POST: "/products" body: "ids: $batch.id" }

Composite keys

In this example, the Product type has a composite key that includes the id and the store.id. The request body needs to include both fields to uniquely identify each product.

1{
2  "items": [
3    { "id": 1, "store_id": 10 },
4    { "id": 2, "store_id": 20 },
5    { "id": 3, "store_id": 10 }
6  ]
7}

1type Product
2  @connect(
3    source: "ecom"
4    http: {
5      POST: "/products"
6      body: """
7      items: $batch { 
8        id 
9        store_id: store.id 
10      }
11      """
12    }
13    selection: """
14    id
15    name
16    store { id name }
17    """
18  ) {
19  id: ID!
20  name: String
21  store: Store
22}
23
24type Store {
25  id: ID!
26  name: String
27}

Rules for batch Connectors

• The mappings from the $batch variable must be scalar fields of the entity type.

  ✅ Valid example

  GraphQL

  copy

  1type Product
  2  @connect(
  3    source: "ecom"
  4    http: {
  5      GET: "/products"
  6      queryParams: "id: $batch.id"
  7    }
  8    selection: """
  9    id
  10    name
  11    """
  12  ) {
  13  id: ID! # scalar field
  14  name: String
  15}

  ❌ Invalid example

  GraphQL

  copy

  1type Product
  2  @connect(
  3    source: "ecom"
  4    http: {
  5      GET: "/products"
  6      body: "id: $batch.variation"
  7    }
  8    selection: """
  9    id
  10    name
  11    """
  12  ) {
  13  id: ID!
  14  name: String
  15  variation: Variation
  16}
  17
  18type Variation {
  19  id: ID!
  20  color: String
  21}

• The selection mapping for a batch request must evaluate to a list of objects.

  ✅ Valid example

  GraphQL

  copy

  1type Product
  2  @connect(
  3    source: "ecom"
  4    http: {
  5      GET: "/products"
  6      queryParams: "id: $batch.id"
  7    }
  8    selection: """
  9    # unwrap the { "results": [...] } wrapper
  10    $.results {
  11      id
  12      name
  13    }
  14    """
  15  ) {
  16  id: ID!
  17  name: String
  18}

  ❌ Invalid example

  GraphQL

  copy

  1type Product
  2  @connect(
  3    source: "ecom"
  4    http: {
  5      GET: "/products"
  6      queryParams: "id: $batch.id"
  7    }
  8    selection: """
  9    # evaluates to { "results": [{ "id": 1, "name": "Hat" }] }
  10    results {
  11      id
  12      name
  13    }
  14    """
  15  ) {
  16  id: ID!
  17  name: String
  18}

• The selection mapping for a batch request must contain the same fields as the $batch variable.

  ✅ Valid example

  GraphQL

  copy

  1type Product
  2  @connect(
  3    source: "ecom"
  4    http: {
  5      GET: "/products"
  6      queryParams: "id: $batch.id"
  7    }
  8    selection: """
  9    $.results {
  10      # maps the product_id field to the id field referenced from $batch
  11      id: product_id
  12      name
  13    }
  14    """
  15  ) {
  16  id: ID!
  17  name: String
  18}

  ❌ Invalid example

  Not selecting the id field means we can't associate the response items to the keys in the batch.

  GraphQL

  copy

  1type Product
  2  @connect(
  3    source: "ecom"
  4    http: {
  5      GET: "/products"
  6      queryParams: "id: $batch.id"
  7    }
  8    selection: """
  9    $.results {
  10      name
  11    }
  12    """
  13  )
  14{
  15  id: ID!
  16  name: String
  17}

Batch options

maxSize

You may want to limit the number of items included in each batch request. This is useful when the API has a limit on the number of items that can be fetched in a single request.

Use the batch.maxSize option to specify the maximum number of items included in each batch request. This results in (N / maxSize) + 1 requests to your API.

1type Product
2  @connect(
3    source: "ecom"
4    http: { GET: "/products", queryParams: "id: $batch.id" }
5    selection: "id name"
6    batch: { maxSize: 10 }
7  ) {
8  id: ID!
9  name: String
10}

A complex batching example

The Fast Healthcare Interoperability Resources (FHIR) standard defines a RESTful API for exchanging healthcare data. It has an unusual pattern for batch requests that takes a little more effort to implement with Connectors.

Batch requests in FHIR are made by sending a JSON payload with a list of requests. Each request has a method, URL, and optional resource.

1{
2  "resourceType": "Bundle",
3  "type": "batch",
4  "entry": [
5    {
6      "request": {
7        "method": "GET",
8        "url": "Patient/123"
9      }
10    },
11    {
12      "request": {
13        "method": "GET",
14        "url": "Patient/456"
15      }
16    },
17    {
18      "request": {
19        "method": "GET",
20        "url": "Patient/789"
21      }
22    }
23  ]
24}

You can accomplish this with Connectors by using the http.body mapping to create the JSON payload. The $batch variable is used to access the list of IDs and construct the requests.

1@connect(
2  http: {
3    POST: "https://fhir.example.com/"
4    body: """
5    resourceType: $("Bundle")
6    type: $("batch")
7    entry: $batch.id { # Iterate over the batch ids
8      request: {
9        method: $("GET")
10        url: $(["Patient", $])->joinNotNull("/")
11      }
12    }
13    """
14  }
15)

Lastly, you can map over resources in the response to return a list that the query planner can use to resolve the entities in the batch.

1@connect(
2  selection: """
3  $.entry.resource {
4    id
5    name {
6      family
7      given
8    }
9  }
10  """
11)

Limitations of "has-many" relationships

With a relationship like Product.reviews, it's usually not possible to batch requests. These relationships often involve pagination and providing the parameters in a batch request isn't possible.

In this example, fetching the reviews for a list of products will always result in a separate request for each product.

1type Product {
2  id: ID!
3  name: String
4  reviews(first: Int, after: String): [Review]
5    @connect(
6      source: "ecom"
7      http: {
8        GET: """
9        /products/{$this.id}/reviews
10        ?first={$args.first}
11        &after={$args.after}
12        """
13      }
14      selection: "id rating"
15    )
16}