Context

One of the easiest ways to build a bad third-party integration is to treat every request as if it were free.

In this air-tracking workflow, users started from an internal shipment identifier, not the carrier-facing AWB the upstream provider actually needed. That meant the system had to resolve internal identity, validate the result, create or reuse a paid external subscription, and then return something trustworthy to the user. If any of those steps were sloppy, we would burn vendor credits, create duplicate subscriptions, or surface tracking states that looked available when they were not.

This was not just an API problem. It was a product, cost, and operations problem wrapped inside one onboarding flow.

Problem

The original shape of the workflow invited expensive mistakes:

  • Internal shipment IDs could point to stale or incomplete carrier identifiers.
  • Invalid AWBs could travel surprisingly far before failing.
  • Repeated requests could trigger repeated paid subscription attempts.
  • Concurrent requests created duplicate-creation risk.
  • Upstream credit exhaustion was a business incident, not just a technical error.

The hard part was not “call carrier API X.” The hard part was making the expensive step intentional and safe.

Constraints

Several constraints made the implementation more interesting:

  • Upstream tracking subscriptions had real cost.
  • Some upstream behaviors were permissive enough to create false confidence.
  • Existing internal workflows already depended on the system boundary.
  • The same shipment might be requested repeatedly by users, jobs, or downstream tools.
  • The user-facing tracking read path needed to stay fast even when the subscription path was expensive.

That ruled out any design that mixed subscription creation and normal reads together.

What I Built

I split the workflow into two very different operations with different guarantees.

First, I separated onboarding from read access. The system no longer treated “show me tracking” and “create the paid upstream subscription if needed” as the same action. A dedicated creation path handled the expensive side effect. A read path returned normalized data only after the onboarding step had happened or after reusable state already existed. That one design choice removed a huge amount of accidental spend risk.

Second, I tightened identifier validation early. I added local AWB validation, including prefix and check-digit rules, so obviously bad identifiers failed before the system spent money or emitted misleading downstream errors. I also added a one-time stale-mapping refresh path so the system could recover when internal GUID-to-AWB mappings were outdated without turning every request into a broad resync.

Third, I made creation concurrency-safe. This was the kind of path where “rare race condition” really means “support incident waiting to happen.” I used transactional boundaries, row locking, and duplicate-key recovery patterns so two concurrent requests for the same shipment could not quietly create duplicate upstream work.

Fourth, I treated upstream credit exhaustion as an operations signal. When the paid provider started refusing new subscriptions, the system did not just log an error and move on. I added cooldown-based alerting so the team got a usable operational signal instead of a flood of identical failures.

Finally, I kept the normalized tracking read model separate from the subscription step. That preserved the user experience. Once a shipment was onboarded, normal reads could stay cheap, predictable, and cache-friendly.

Validation

I validated this flow against the specific failure modes that had been hurting trust and cost control:

  • invalid or malformed carrier identifiers
  • stale internal-to-external mappings
  • repeated create attempts against the same shipment
  • concurrent onboarding requests
  • simulated upstream refusal conditions

The goal was not only “does the happy path work?” It was “can this flow fail without burning money, duplicating state, or confusing operators?”

That meant scenario-based testing, transaction review, and operational sanity checks around alert behavior. I also reviewed the boundary between onboarding and reads to confirm that normal tracking requests no longer carried hidden paid side effects.

Outcome

The resulting flow was much more intentional.

  • Paid subscription creation became explicit instead of accidental.
  • Invalid identifiers were rejected earlier and more clearly.
  • Duplicate create behavior became much harder to trigger.
  • Support had a better mental model for whether a problem came from identity resolution, onboarding, or normal tracking reads.
  • Vendor credit exhaustion became visible fast enough to act on.

What I like about this story is that it reads well outside logistics too. The core pattern is broadly useful: if a third-party side effect is expensive, rate-limited, or contract-sensitive, separate it from normal reads and design the boundary like it matters.

Lessons

Cost control is part of interface design.

Teams often talk about retries, backoff, and fallback. Those matter. But in paid integrations, you also need to ask whether a request should exist at all, whether the side effect belongs on the user-facing path, and what happens when two callers ask for the same expensive thing at once.

That is where the real engineering leverage lived here.

If you need to make third-party APIs safer without turning every user request into a paid gamble, this is exactly the kind of boundary design I enjoy working on. Let’s talk .