Friday — 5:42 PM
Six months of work had finally reached production.
This wasn’t just another feature.
The new payment integration touched the very heart of our ERP: the point of sale.
The rollout plan was intentionally conservative.
One branch first.
If everything behaved as expected, we would gradually enable it across the remaining stores.
The pilot looked successful.
Transactions were flowing, the logs looked clean and everything behaved exactly as we expected from a brand-new production deployment.
I closed my laptop convinced the hardest part was finally behind us.
On Monday morning, my phone started ringing.
A store manager wasn’t sure whether a customer had actually been charged.
The bank showed the transaction.
NetPay showed the transaction.
The ERP didn’t.
There was no trace of a final response.
All we knew was that Metrify had sent the payment request, a new sales record existed, and the transaction had never reached its final state.
At first, I didn’t think it was a bug.
After months of testing, my instinct was to treat it as an isolated incident.
Uncomfortable?
Yes.
But isolated.
The customer was still standing at the counter.
Their banking app already showed the payment.
The cashier needed an answer.
The question was no longer technical.
It had become operational.
Could the customer leave with the merchandise?
Friday · 5:42 PM
Production deployment
The payment integration reached production after months of development, testing and validation.
Monday morning
First call
A branch reported a payment confirmed by both the bank and NetPay, but still open inside the ERP.
Tuesday · 5:00 PM
Second incident
With more branches online, another customer was waiting while the bank showed a successful payment but the ERP still considered the sale pending.
Rolling out gradually
flowchart LR A["Pilot"] --> B["1 Branch"] B --> C["3 Branches"] C --> D["7 Branches"] D --> E["14 Branches"] B:::active classDef active fill:#1f2937,stroke:#facc15,color:#ffffff
During the first week we weren’t trying to validate an idea in theory.
We wanted to observe how the payment flow behaved in a real store, with real cashiers, real customers and real transactions.
The problem was that the incidents never followed a predictable pattern.
It wasn’t always the same terminal.
It wasn’t always the same branch.
It wasn’t always the same step in the payment flow.
Out of a thousand transactions, perhaps one or two behaved differently.
But when they did, they didn’t leave enough evidence behind.
When it stopped looking isolated
On Tuesday we enabled the integration in more branches.
Around five in the afternoon, another call arrived.
A customer was waiting at the counter.
Their banking application already showed the payment.
The ERP still considered the sale pending.
The same questions came back.
Should the customer leave with the merchandise?
Should the cashier charge again?
Or should everyone simply wait?
Nobody had enough evidence.
Only assumptions.
We were no longer looking for a bug.
We were trying to understand the exact moment a transaction stopped leaving evidence behind.
After the second incident we started reviewing everything.
Apache logs.
PHP logs.
Fatal errors.
Exceptions.
Sockets.
Database records.
Nothing explained the behavior.
The system wasn’t crashing.
That would have been easier.
Instead, everything appeared normal until, at some point, the transaction simply disappeared from our perspective.
Following a payment through the system
sequenceDiagram autonumber participant ERP participant Netpay participant Terminal participant Bank ERP->>Netpay: POST Sale Netpay-->>ERP: 200 OK Netpay->>Terminal: Send payment Terminal->>Bank: Authorization Bank-->>Terminal: Approved Terminal-->>Netpay: Result Netpay-->>ERP: Webhook ERP->>ERP: Complete sale
At first glance, the architecture looked straightforward.
One request.
One gateway.
One payment terminal.
One bank.
Reality was very different.
A single payment travelled through multiple independent systems, each with its own clocks, retries, logs, failure modes and, ultimately, its own version of the truth.
The problem wasn’t simply that some payments failed.
The real problem was that nobody could explain where the transaction stopped existing from the ERP’s point of view.
That realization completely changed our objective.
We stopped trying to reproduce a bug.
We started building a way to explain what had actually happened.
Seven branches later
By the time seven branches were running the new payment flow, the problem had changed completely.
Each branch could have multiple payment terminals.
Transactions were no longer happening one after another.
While one sale was sending a payment request, another was waiting for authorization and a third one could already be receiving the final callback.
The log kept growing over time.
Not by transaction.
JSONL stream · 7 sucursales · 2 terminales por sucursal
08:35:10.102 S10-T1 pre-request oDMfPMBz...
08:35:10.448 S03-T2 pre-request C7ai-tOW0...
08:35:11.006 S10-T1 post-response oDMfPMBz...
08:35:11.391 S07-T1 webhook-received L9kaP02m...
08:35:11.884 S02-T2 pre-request aQm91ZxK...
08:35:12.120 S03-T2 post-response C7ai-tOW0...
08:35:12.612 S06-T1 pre-request R81mKQpl...
08:35:13.048 S14-T2 post-response N8xw0Paa...
08:35:13.441 S01-T1 webhook-received kP91aQx2...
08:35:13.982 S10-T1 webhook-missing oDMfPMBz...That meant a single sale was buried among hundreds of events generated by completely unrelated transactions.
Searching by amount still worked sometimes.
Searching by terminal sometimes helped.
Looking up the sales folio occasionally narrowed the search.
But none of those approaches scaled.
More than once we ended up opening database tables and reconstructing a sale manually while the customer was still standing at the counter.
That was the moment we realized the real problem.
We didn’t lack logs.
We lacked context.
We needed a different way to search
We didn’t start by building dashboards.
We didn’t start by adding alerts.
We didn’t build an observability platform overnight.
We started with something much smaller.
Every transaction needed an identity.
That’s how the correlation_id was born.
It wasn’t the hero of the story.
It was simply the first tool that allowed us to follow a payment instead of chasing timestamps.
The real idea was much bigger.
We needed observability designed around the actual payment flow.
Not around servers.
Not around applications.
Around the transaction itself.
The first piece of evidence
Raw transaction evidence
post-response · payment request accepted
{
"stage": "post-response",
"method": "POST",
"endpoint": "/gateway/integration-service/transactions/sale",
"http_status": 200,
"timestamp_iso": "2025-10-06T08:39:30-06:00",
"correlation_id": "rTAOeW0rc80QdKWaaHMdk",
"response": {
"code": "00",
"message": "Request accepted successfully"
},
"latency_ms": 1639.24
}For the first time, every payment could be identified individually.
The process was still completely manual.
Download the JSONL file.
Open it.
Press Cmd + F.
Search for the correlation_id.
Primitive?
Absolutely.
Useful?
More than anything we had before.
For the first time, every investigation had a starting point.
Correlation ID
rTAOeW0rc80QdKWaaHMdk
Stage
post-response
HTTP Status
200 OK
Latency
1639.24 ms
Observed result
The gateway accepted the request, but that alone didn't prove the ERP had received the final payment confirmation.
The important difference
NetPay accepted the payment request and returned 200 OK.
The ERP still had no record of the final payment confirmation.
Nothing had technically failed.
Yet the transaction had entered an unknown state.
Failure Mode
Ambiguous payment
The gateway successfully accepted the payment request.
The customer completed the payment.
But the ERP never received the final state required to complete the sale.
Operationally, this was worse than a clean failure.
A failed request can be retried.
An ambiguous payment has to be investigated.
Following the transaction
Correlation Journey
rTAOeW0rc80QdKWaaHMdk
POST Request
The ERP submitted the payment request.
Gateway Response
The gateway returned HTTP 200.
Terminal
The customer completed the payment.
Webhook
The callback never arrived.
ERP Update
The sale remained pending.
The correlation_id never existed inside every external system.
It didn’t have to.
It existed where we needed it most.
Inside our own evidence.
Instead of searching for timestamps or matching amounts, we could finally reconstruct a payment from beginning to end and determine the exact stage where, from the ERP’s perspective, the transaction stopped moving forward.
Building the first analyzer
By the time fourteen branches were using the platform, manual investigation had become impossible.
Events from every branch were interleaved.
Payment flows happened in parallel.
The JSONL log kept growing every minute.
We needed a tool capable of receiving a single correlation_id, finding every related event and rebuilding the complete journey automatically.
That was the beginning of our first analyzer.
python analyze.py logs.jsonl —cid rTAOeW0rc80QdKWaaHMdk
Searching…
Found 8 events
Last Stage:
POST Response
Webhook:
NOT FOUND
Diagnosis:
Gateway accepted request.
No callback was received.
Metric
Average investigation time
Before
30 min
After
30 sec
Thirty minutes wasn’t the real improvement.
The real improvement was confidence.
For the first time, we could answer a branch using evidence instead of assumptions.
We no longer started by asking whether the network had failed.
Or whether the payment terminal was offline.
Or whether the ERP had crashed.
Or whether the payment provider had a problem.
We started by following the transaction.
Everything else came afterwards.
What we actually built
For weeks I believed we were stabilizing a payment integration.
We weren’t.
We were building observability.
The payment gateway was simply the problem that forced us to do it.
Today, when someone calls asking what happened to a payment, we don’t start guessing.
We start tracing.
And when you’re dealing with money, that changes everything.
The tool
The analyzer eventually became an Open Source project after the original incident stopped being urgent.
It wasn’t created for GitHub.
It was created for production.
For a transaction to be reconstructed, every event needed to include, at minimum:
correlation_idstagetimestamp_isoendpointhttp_statusrequestresponselatency_ms- branch, terminal, user and sales reference
With that information, a complete payment journey could be rebuilt in seconds.
The complete implementation is available as an Open Source project.
GitHub
WatchPost
Open Source analyzer for JSONL logs capable of reconstructing complete transactions through Correlation IDs and generating automatic diagnostics.
This recommendation is not specific to NetPay.
The payment provider doesn’t matter.
The ERP doesn’t matter.
Even the programming language doesn’t matter.
Any distributed system that exchanges information between multiple services will eventually face the same problem.
Not because software is poorly written.
Because distributed systems naturally produce uncertainty.
What we actually learned
In systems that move money, the goal isn't to eliminate every error.
The goal is making sure no error ever becomes a mystery.
That incident changed the way I think about production software.
We didn’t just improve a payment integration.
We changed the way we investigate failures.
Since then, every critical feature we build starts with the same question.
If this fails in production, will we have enough evidence to explain what happened?
If the answer is no, the feature isn’t finished.
Lessons learned
This incident became the foundation for several engineering principles that are now part of how we build Metrify.
Applied Observability
Understanding production behavior through evidence instead of assumptions.
Correlation IDs
Following a single transaction across independent systems and events.
Traceability
Reconstructing the complete lifecycle of a critical business operation.
Structured JSONL Logging
Capturing chronological events in a format that is easy to inspect and analyze.
Production Debugging
Investigating under pressure using evidence rather than intuition.
Distributed Systems
Accepting that no single system owns the complete truth about a transaction.
Every one of these topics deserves its own article.
But they all started with the same production incident.
A payment that nobody could explain.
And a question that completely changed how I build software:
What evidence will I have when this fails in production?