Beyond the Console: Engineering Production-Grade Observability

Let’s be honest: we’ve all been there. A high-priority production incident occurs, and your only clue is a fragmented [object Object] error in a log aggregator that hasn't seen a deployment in three weeks. In the world of universal apps—where the same logic might run on a browser, a mobile device via React Native, or a Node.js edge function—relying on console.log or isolated error reporting is like trying to map the subway system by looking through a straw.

Over the last year, I’ve been obsessed with moving past "logging" and into the realm of Observability (o11y) using OpenTelemetry (OTel). Here is how I’ve been engineering these systems to handle the complexities of universal applications.

The Fallacy of the Log

Logs are events. Traces are stories. When a user in London experiences a 500ms lag on a checkout button, a log tells you that it happened. A trace tells you that the frontend span waited for a gateway span, which was throttled by a database lock in a completely different microservice.

In universal apps, the challenge is Context Propagation. How do you ensure the traceId generated on an Android device survives the jump to your Go backend and eventually into your Postgres query?

The Architecture: The OTel Collector is Your Best Friend

One breakthrough I had was moving away from sending data directly from the client to a vendor (like Honeycomb or Datadog). Instead, I started using an OpenTelemetry Collector as a sidecar or a standalone service.

This allows the client apps to be "dumb." They push spans via OTLP (OpenTelemetry Protocol) to a managed collector, which then handles the heavy lifting of scrubbing PII, sampling, and exporting to multiple destinations.

Implementation: The Universal Tracer

Here’s a simplified version of how I initialize a global tracer that works across environments. The key is the BatchSpanProcessor—you never want to block the UI thread while sending telemetry data.

import { WebTracerProvider } from '@opentelemetry/sdk-trace-web';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const provider = new WebTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'universal-customer-app',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.2.0',
  }),
});

const exporter = new OTLPTraceExporter({
  url: 'https://otel-collector.internal.mycompany.com/v1/traces',
});

// Use BatchSpanProcessor for production performance
provider.addSpanProcessor(new BatchSpanProcessor(exporter, {
  maxQueueSize: 2048,
  scheduledDelayMillis: 5000,
}));

provider.register();

Solving the Mobile Problem: Offline Spans

Universal apps often include React Native. Unlike a browser, mobile devices lose connectivity constantly. If you try to send a span while the user is in a tunnel, it’s gone.

To solve this, I implemented a custom Persistent Span Store. Instead of immediately exporting, we buffer spans to the device’s local storage (like SQLite or MMKV) and flush them once the NetInfo API signals a stable connection. It turns out that observing "offline mode" behavior is often more valuable than observing the happy path.

Automatic vs. Manual Instrumentation

While auto-instrumentation libraries for fetch or XMLHttpRequest are great for quick wins, they often create too much noise. I've shifted toward a Hybrid Approach:

1. Auto-instrumentation for network requests and navigation changes.
2. Manual instrumentation for critical business logic (e.g., calculate_risk_score or process_payment).

async function processOrder(orderId: string) {
  const tracer = trace.getTracer('checkout-service');
  
  return await tracer.startActiveSpan('process_order_logic', async (span) => {
    try {
      span.setAttribute('order.id', orderId);
      // ... business logic
      span.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      span.recordException(error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw error;
    } finally {
      span.end();
    }
  });
}

Sampling: The Silent Budget Killer

In a production environment with millions of spans, sending everything will bankrupt your cloud budget. I recommend Tail-based Sampling. Instead of the client deciding what to send (Head-based), the OTel Collector looks at the entire trace. If the trace contains an error or an unusually high latency, it keeps the whole thing. If it’s a standard 200 OK, it drops 95% of them.

The Result: From Guessing to Knowing

Since moving to this architecture, our MTTR (Mean Time To Recovery) has dropped significantly. We no longer ask "What happened?"; we look at the waterfall chart and ask "Why did this specific database call take 4 seconds?"

Engineering observability isn't just about adding tools; it's about changing your mindset from "Does it work?" to "How is it performing for the user in real-time?"

If you haven't explored OpenTelemetry for your universal apps yet, start small. Instrument your most critical API call. You'll be surprised what's actually happening under the hood.