Facing multiple Issues while integrating opentelemetry tracing

[bhupesh-sf]

Steps to reproduce

Install the following packages:

@opentelemetry/node
@opentelemetry/tracing
@opentelemetry/exporter-jaeger
@opentelemetry/plugin-http
@opentelemetry/plugin-https
@opentelemetry/plugin-express

AT the top of index.ts

import { JaegerExporter } from '@opentelemetry/exporter-jaeger';
import { NodeTracerProvider } from '@opentelemetry/node';
import { SimpleSpanProcessor, ConsoleSpanExporter } from '@opentelemetry/tracing';

Following at the top of the main function in index.ts file

  const option = {
    serviceName: 'basic-service',
    tags: [], // optional
    // You can use the default UDPSender
    host: 'localhost', // optional
    port: 6832, // optional
    // OR you can use the HTTPSender as follows
    // endpoint: 'http://localhost:14268/api/traces',
  }

  // Configure span processor to send spans to the exporter
  const exporter = new JaegerExporter(option);
  provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
  provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));
  provider.register();

Then import the following in the application.ts file:

import openTelemetry, { Span, Tracer } from '@opentelemetry/api';

Then I added this middleware in the application.ts

const middle: Middleware = async (ctx, next) => {
  const trace = openTelemetry.trace.getTracer('loopback:tracer');
  let span: Span;
  span = trace.startSpan(ctx.request.url);
  span.setAttribute('key', 'value');
  span.addEvent('invoking next');
  const res = await next();
  return res;
}

Then bind this middleware in the constructor in the application.ts:

this.middleware(middle);

Another thing if I use Axios to make an HTTP request then context propagation happens properly but if I use the recommended way of loopback by creating a rest connector then it doesn't happen. I have installed all the required modules in the app.

Current Behavior

Some modules (redis, express) were already required when their respective plugin was loaded, some plugins might not work. Make sure the SDK is setup before you require in other modules.

Expected Behavior

As per the auto instrument libraries from opentelemetry for HTTP, express, redis, psotgres all opentelemetry modules required to be loaded before any of such application module are loaded. So as per code sample above it should have worked all fine.

Another thing if I use Axios to make an HTTP request then context propagation happens properly but if I use the recommended way of loopback by creating a rest connector then it doesn't happen. I have installed all the required modules in the app.

Additional information

linux x64 12.18.4
├── UNMET PEER DEPENDENCY @loopback/[email protected]
├── UNMET PEER DEPENDENCY @loopback/[email protected]
├── UNMET PEER DEPENDENCY @loopback/[email protected]
├── @loopback/[email protected]
├── @loopback/[email protected]
├── UNMET PEER DEPENDENCY @loopback/[email protected]
├── @loopback/[email protected]
├── UNMET PEER DEPENDENCY @loopback/[email protected]
├── @loopback/[email protected]
├── @loopback/[email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├─┬ UNMET DEPENDENCY [email protected]
│ ├── @loopback/[email protected]
│ ├── @loopback/[email protected]
│ ├── @loopback/[email protected]
│ ├── @loopback/[email protected]
│ ├── @loopback/[email protected]
│ ├── [email protected] deduped
├── [email protected]

[dhmlau]

@raymondfeng, could you please help on this? Thanks.

[bhupesh-sf]

Any help here @strongloop/loopback-next ??

[raymondfeng]

@bhupesh-sf I will question the design of opentelemetry to rely on require hooks without providing explicit APIs. Meanwhile, maybe you should try to use -r flag for node itself? Such as node -r @opentelemetry/node -r @opentelemetry/plugin-http -r @opentelemetry/plugin-https -r @opentelemetry/plugin-express?

-r, --require=...           module to preload (option can be repeated)

[raymondfeng]

Propagation of tracing information over outbound http connections is even tricker at this point. It really depends on the connector modules which use their own http client packages. These modules do not support dependency injections.

[bhupesh-sf]

@raymondfeng opentelemetry does provide the API but they are giving auto instrumentation for some frameworks like express etc and they do this thing quite well. SO I was thinking if we can leverage the same for Loopback.

Otherwise, we have to manually do the same which will take a good amount of work for a framework.

[bhupesh-sf]

And the other option of node -r I tried but didn't work. @bajtos mentioned it earlier in of the conversations.

[raymondfeng]

Just want to correct one thing - we probably need to create a simple js file as follows:

opentelemetry-register.js

const { NodeTracerProvider } = require('@opentelemetry/node');

// Create and configure NodeTracerProvider
const provider = new NodeTracerProvider();

// Initialize the provider
provider.register()

Then use -r ./opentelemetry-register.js?

[raymondfeng]

@bhupesh-sf I suggest that you create a simple LB4 app with your code in index.ts to set up automatic instrumentation. We can then check it out for troubleshooting.

[mschipperheyn]

I have been getting nowhere with NodeTracerProvider. Nothing captured in a stack with redis, graphql, mysql, express. We use es6 code with babel. Could this be the reason?

[bhupesh-sf]

@raymondfeng I created a separate file with trace provider and loaded it using node -r opentelemetry-registry.js and everything worked out of the box.

Here is an example of my opentelemetry-registry.js file

import { JaegerExporter } from '@opentelemetry/exporter-jaeger'; import { NodeTracerProvider } from '@opentelemetry/node'; import { SimpleSpanProcessor } from '@opentelemetry/tracing'; const provider = new NodeTracerProvider(); const option = { serviceName: 'BPMN-service', tags: [], host: 'localhost', port: 6832, } const exporter = new JaegerExporter(option); provider.addSpanProcessor(new SimpleSpanProcessor(exporter)); provider.register();

And here is my npm start command
"start": "node -r './dist/opentelemetry-registry.js' .",

I used it inside a monorepo for microservice architecture and it's working all well. Below is the screenshot of the same.

Now the QUESTION here is can we do it using inside loopback so every user doesn't have to create this file and add to the project. If YES guide me, I can work on that PR.

If NOT then I believe we should document this for the ease of the community.

Let me know your thoughts about it.

[bhupesh-sf]

I have been getting nowhere with NodeTracerProvider. Nothing captured in a stack with redis, graphql, mysql, express. We use es6 code with babel. Could this be the reason?

@mschipperheyn Please try the above solution it shall work. Please note you have to install all the other auto instrumentation plugins like @opentelemetry/plugin-http, @opentelemetry/plugin-https, @opentelemetry/plugin-express, @opentelemetry/plugin-pg-pool, @opentelemetry/plugin-pg, @opentelemetry/plugin-ioredis OR any other of your requirement.

Let me know if still doesn't work.

[bhupesh-sf]

@raymondfeng Can you please suggest on above?

[bhupesh-sf]

@raymondfeng Any suggestion on the above?

[nflaig]

@bhupesh-sf did you check #3389 already?

[bhupesh-sf]

@nflaig I did it when I started working on opentelemetry not recently. The current implementation which Is available in a separate branch for tracing is based on opentracing not opentelemetry and it's completely broken. The solution I gave above completely works and more can be built upon this.

Right now I am short of bandwidth but if someone is interested can pick this up. The above solution works like a charm.