# Distributed Tracing with Platformatic and Open Telemetry

In a microservices architecture, a request usually spans between multiple services deployed on different nodes. When this happens, it can be hard to trace issues precisely,  particularly when errors happen, or when bottlenecks need to be identified. Tracing can however be done using distributed tracing, which allows for the tracing of calls end-to-end across all the involved systems. 

[Platformatic](https://platformatic.dev/) uses [Open Telemetry](https://opentelemetry.io/) (OTEL) to implement distributed tracing. Open Telemetry is an observability framework which is vendor and tool-agnostic and can be used with a broad variety of observability backends, such as [Jaeger](https://www.jaegertracing.io/).

In this article, we will create an example of an API with Platformatic and see how the traces can be collected using Jaeger.

## Key Open Telemetry concepts

For our purposes, we will use Open Telemetry for traces (it can also be used for metrics and logs). Open Telemetry backends collect **spans**, which are generic “units of work”. A span can have a parent (a span with no parent is a “root” span), so the OTEL backends can relate all the spans  in a tree representing the  distributed trace.

## Platformatic Services

In Platformatic, we have different ways of creating and composing services:

* Platformatic DB: Services based on Databases
    
* Platformatic Service: Generic JavaScript/TypeScript services
    
* Platformatic Composer: Automatically aggregate multiple service APIs into a single API.
    
* Platformatic Runtime: Run multiple Platformatic microservices as a single monolithic deployment unit.
    

All Platformatic services are able to create span and propagate tracing, which we will explore below

## Setting up 

### Prerequisites

Ensure you have the following installed on your local machine;

* Node.js  >=  v18.8.0 or  >=  v19.0.0
    
* A code editor (e.g., Visual Studio Code or Sublime Text)
    
* Docker installed and working
    

### Run Jaeger

This can be done quickly with Docker:

```dockerfile
docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest
```

Check that the server is running by opening [http://localhost:16686/](http://localhost:16686/) with your browser.

### Platformatic Services Setup

We are going to create:

* A `movies` DB service exposing the OpenAPI to manage a movies database
    
* A `count` service which calls `movies` to get the full list of the movies, and then count them (while this wouldn’t be a very useful service in reality, the goal here is to show how distributed tracing works with  Platformatic) 
    
* A `composer` which aggregates the `movies` and `count` APIs into a single API. 
    

#### `movies` **DB service**

Create a `movies` folder and cd into it:

```plaintext
mkdir movies
cd movies
```

Then, create a db service in the folder using `npx create-platformatic@latest`. Be sure to set the `5042` port, select `sqlite` DB and create and apply the default migrations:

![](https://lh5.googleusercontent.com/2XwxWv4i9IlCuYKpBj_c6TgI6mBfAFQk1pofwqVPXSDhiYbc0_Dogokq2Jftt4vuuiGWt8jLER0K_ERLdc0bpzh4hQWlhBdim0aqB6rDcgdlNlYk_ul3ukAjANckjVJZVZfGSkhBDznJKgnkkHYWK6w align="left")

Now open `platformatic.db.json` and add the following:

```json
(...)
  "telemetry": {
    "serviceName": "movies",
    "exporter": {
      "type": "otlp",
      "options": {
        "url": "http://localhost:4318/v1/traces"
      }
    }
  }
```

You can now start the service with: `npx platformatic db start`:

![](https://lh6.googleusercontent.com/FJYIYyKEVwwWhKUCPnf7pASYTMAB-E4ZmQCpLY8Um92KQnlCFz_Qze-7vK-yLCR4lavYZ_RvFaiUYn8hynvV6GOEIGzEUF2MoNBS5dSlE9z2ZekT76ciocn2Tvvofcx1v4HP0PeK4obqsJVXWDjLtoY align="left")

#### `count` **service**

In a different folder, create the `count` directory:

```plaintext
mkdir count
cd count
```

Then, create a service in the folder using `npx create-platformatic@latest`. Be sure to set the `5043` port:

![](https://lh6.googleusercontent.com/cMSmCq54SBsc7gfphyMLl-IVEvYwnwPTbZboSjKijQidCoZoW9bzRnUbZbZHWRwrDMTB7HwpmplSNyldq49WIMGKTKlFt9qMHmgWAwJIrZ7QsNj9ZVtJOTMf6DLs7D7Yro4QejrIdVJPtcim4FqtvqU align="left")

Open the `platformatic.service.json` file and add the following telemetry configuration (this is  identical to DB, except for  a different `serviceName`):

```json
(...)
  "telemetry": {
    "serviceName": "count",
    "exporter": {
      "type": "otlp",
      "options": {
        "url": "http://localhost:4318/v1/traces"
      }
    }
  }
```

We want this service to invoke the `movies` DB service, so we need to add a client for it: 

```plaintext
npx platformatic client http://127.0.0.1:5042 js --name movies
```

This should be the output:

![](https://lh4.googleusercontent.com/0q3wPCuE0cdasAcd342WE5kYNZInyRmUhxLtwRmLA69S9RoDfJb5kMJYLKX2ydxBIKwVQ1ogynV-gdh49TE10HP2BOMT3PIp53itwHkF99yVfSw-m1cBBQFb40aZeyibFYkloh2uVQoRr1vTPy28_R4 align="left")

Check platformatic.service.json to ensure  that the client has been added (\`PLT\_MOVIES\_URL\` is defined in `.env`):

```json
(...)
    "clients": [
    {
      "schema": "movies/movies.openapi.json",
      "name": "movies",
      "type": "openapi",
      "url": "{PLT_MOVIES_URL}"
    }
  ]
}
```

Now open `routes/root.js` and change the content with the following:

```javascript
module.exports = async function (fastify, opts) {
  fastify.get('/count-movies', async (request, reply) => {
    const movies = await request.movies.getMovies()
    return { length: movies.length }
  })
}
```

This code calls movies to get all the movies and returns the length of the array.

Finally, start it with `npx platformatic service start`:

![](https://lh5.googleusercontent.com/EBFJJo8ZUAzTWbJ9xMJOI5zOfj19wO7iapum0pLrrdXKpPXz-EfCSqzz5SUT2dsPmzcr-mguQbiCvfIyP4Iod9GFXPj5HdGFiRK0cJYtnfsreWUeTtzhWjRTn8eich8Wx9L6fBa5xmr3gFJCCWcj2ow align="left")

#### `composer` **service**

Create at the same level of `movies` and `count` another folder for `composer` and cd into it:

```plaintext
mkdir composer
cd composer
```

Then create the composer on the `5044` port in the folder using `npx create-platformatic@latest`:

![](https://lh3.googleusercontent.com/dJqyTTMEzE8JPkRrE98y2ZAz31pDk3aMYApyG49H6JFmZuHg2VtDUsauHvECcvJ6JnyKyW5AzbHpBgnV_5iUly4fcTh56ie04VNnUVpC7tPrYEXrjGWo9nmMpaDEEo3dOkfgPef5YgYb7FDI3h4Xx7g align="left")

Open `platformatic.composer.js` and change it to the following:

```javascript
{
  "$schema": "https://platformatic.dev/schemas/v0.33.1/composer",
  "server": {
    "hostname": "{PLT_SERVER_HOSTNAME}",
    "port": "{PORT}",
    "logger": {
      "level": "{PLT_SERVER_LOGGER_LEVEL}"
    }
  },
  "composer": {
    "services": [
      {
        "id": "count",
        "origin": "http://127.0.0.1:5043",
        "openapi": {
          "url": "/documentation/json"
        }
      },
      {
        "id": "movies",
        "origin": "http://127.0.0.1:5042",
        "openapi": {
          "url": "/documentation/json"
        }
      }
    ],
    "refreshTimeout": 1000
  },
  "watch": true,
  "telemetry": {
    "serviceName": "composer",
    "exporter": {
      "type": "otlp",
      "options": {
        "url": "http://localhost:4318/v1/traces"
      }
    }
  }
}
```

Note that we just added `count` and `movies` as proxied services and added the usual telemetry configuration, with a different `serviceName`.

Finally, start it with `npx platformatic compose start`: 

![](https://lh6.googleusercontent.com/Ikad44osORzsuhqTgAVl6t8XwqiXsrgFbcdKa9SyjfHymKmUVso_sXn3w-P6qqxAY6OdULtFFa5UFd4c56ZmewzvMXvf032-rdFkWoMg823SspyrF6oytYvuj09AftLzlbuPjFNXTHhFQ_RQ4hMernc align="left")

Open the OpenAPI definition using this link: [http://127.0.0.1:5044/documentation/static/index.html](http://127.0.0.1:5044/documentation/static/index.html)

You should see something like this:

![](https://lh3.googleusercontent.com/ZKOxGkBopsvdsoTCaWlNf2Jphd-6eQjraWHTZnK72YR67oljP1o_OP3rG3fWZEugeOb5gLQTmL86Ip7SkY3sZRh-POfZursd8D1XRkuDwOCFEykC33BV_URUwNW50ngg6fQMf_pB5MB7u5oN91hXi9Y align="left")

Note that the `composer` acts as a facade for both the `count` and `movies` services 

Now, we can populate the `movies` DB running some `POST`, e.g.:

```pgsql
curl -X POST -H "Content-Type: application/json" -d '{"title":"The Matrix"}' http://127.0.0.1:5044/movies/
curl -X POST -H "Content-Type: application/json" -d '{"title":"The Matrix Reloaded"}' http://127.0.0.1:5044/movies/ 
```

![](https://lh5.googleusercontent.com/Ry-kNxuQExBWXD4N9f-inTC6Zxi4K7hS7PTV5l7B4k10OJ8QHL023TRknMhL-YKC4d3IEHBI09JV-s_qTmFYJogd6dhiaBbPz6_mF1xd5C_m0DJrWZbGnKklA4w9xFKorxIx8zw0rtfX26vWE25kK94 align="left")

### See the traces

Seeing as we want to obtain the “count” of the movies, we will need to call the composer service, which will proxy the `count` service which in turn will call `movies`:

![](https://lh4.googleusercontent.com/seF14sVKe1oHebcyiI4kkG0o9FJxgLZvFfJt8eg7uLzWaSqxGPk_cMSTpBY2g8I14-yM1nvIqZ-a9M71jDg-3hnsH8tylcaFi_qJApY2Mc82KZWwWTqY0_gB-RzacNxylSe-vG1JFzB0_bPP7WOf1eE align="left")

We can do the call with any HTTP client, e.g.:

```http
curl http://127.0.0.1:5044/count-movies
```

![](https://lh6.googleusercontent.com/NyWc3fcqQmfyBhUFdTIK3A0SSQLf85kc4_F8wJf3xZUS2-dPDrFz1Bx6ugJh4OHytxW4dodK2RiW4dYtQjQcKUjz5YED23vnay4QojEt20KR9ffQ4Mww8ffY7SqI_IlNID2amMMs6g9hdyEBJ0okmlg align="left")

Now, open Jaeger with [`http://localhost:16686/search`](http://localhost:16686/search`) and select in filter `composer` as Service and the `count-movies` operation, then click on “Find Traces”:

![](https://lh4.googleusercontent.com/HeU6OxLzZOCdcI7F9x3eZMMLhpXGcq3vP3Mti-uhxLX7nfmv8kkAxMnP2NX48d4W8G3Tr_xnFqjqRKh-Ng4V4FIWC6FOwwRhNAeT_7O3xgHgZ_QDbSbOn9j-gR7TP2lBJMqdg4mi3pGuvXbEOUMf1Hw align="left")

You can open the trace by clicking on it:

![](https://lh5.googleusercontent.com/yp369che2A3a8XQItV0CwzoJlx_PXp7xKOonPLrUJDo4OCJNBZyfGLrVctOXGjLhAs4coRlnC2qKANFxGMw-KVfeKV1CSflQdNIadArwV50UI7Q0Bvv9YThiPGuiz00p9rbHWShpBYHotYj5-Yy7nyQ align="left")

Note that everytime a request is received or client call is done, a new span is started. As such, we have: 

* One span for the request received by `composer`
    
* One span for the `composer` client call to `count` 
    
* One span for the request received by `count`
    
* One span for the `count` client call to `movies`
    
* One span for the request received `movies`
    

All these spans are linked together, so that you can see the whole trace.

Note that other useful views are available, such as  the flamegraph view:

![](https://lh3.googleusercontent.com/eXshLchdno4417tSguZ3ugUJkHtwhFaptAliwderiE9-q4kUn-anQAeKhgDKzWY0LnJ9Zgv4NBX0m9QRdL0-U4WYsPW85RTkpKauoL30Np_XfGAGCSvYYF6z0gqq7O5M9hg-MxZMzQRiv0RExCbP5Qs align="left")

### Wrapping up 

As we have seen, Platformatic now offers distributed tracing with Open Telemetry out–of-the-box!  

To use this feature, you will just need to set up the proper configurations, and then you’ll be able to see all the requests as they are propagated through the microservices.