← All articles

When a customer taps their Visa card at a supermarket terminal, the perception is instantaneous. The reality is a choreographed exchange involving at minimum six distinct systems, two cryptographic operations, and a message protocol designed in 1987 that still processes the majority of the world's card transactions today.

The ISO 8583 Message

ISO 8583 is not an API—it is a binary wire protocol that encodes financial transactions as fixed and variable-length fields. Every POS message carries a Message Type Indicator (MTI), a bitmap declaring which data elements are present, and the data elements themselves.

The MTI is a four-digit BCD (Binary Coded Decimal) value. The first digit encodes the ISO version (0 = 1987, 1 = 1993, 2 = 2003). The second digit is the message class: 1xxx = Authorization, 2xxx = Financial, 4xxx = Reversal. The third digit encodes function (Request / Response / Advice), and the fourth describes the transaction originator.

MTIMeaningDirection
0100Authorization RequestAcquirer → Issuer
0110Authorization ResponseIssuer → Acquirer
0200Financial Transaction RequestAcquirer → Issuer
0210Financial Transaction ResponseIssuer → Acquirer
0400Reversal RequestAcquirer → Issuer
0420Reversal AdviceIssuer → Acquirer

Following the MTI is the primary bitmap—64 bits indicating which of Data Elements 1–64 are present. If bit 1 is set, a secondary bitmap extends coverage to DE 65–128. A tertiary bitmap (private use, DE 129–192) exists in some proprietary implementations.

Python — ISO 8583 bitmap parser
def parse_bitmap(raw_hex: str) -> list[int]:
    """Return list of present DE numbers from hex bitmap string."""
    bitmap_bytes = bytes.fromhex(raw_hex)
    present = []
    for byte_idx, byte in enumerate(bitmap_bytes):
        for bit_pos in range(8):
            if byte & (0x80 >> bit_pos):
                de_number = byte_idx * 8 + bit_pos + 1
                present.append(de_number)
    return present

# Example: F220000012800000 (primary bitmap only)
# DE 1,2,3,6,10,11,12,13,22,26,32,38,39,41,42 are present
print(parse_bitmap("F220000012800000"))
# → [1, 2, 3, 6, 10, 11, 12, 13, 22, 26, 32, 38, 39, 41, 42]

The Authorization Flow

A contactless payment triggers the following sequence. Timing figures below are representative of a European four-party scheme (Visa/Mastercard) with an acquiring bank in the same country as the issuer.

Authorization Message Latency by Pipeline Stage
Median (p50) and 99th percentile (p99) latency in milliseconds. Production data representative of a mid-size European acquirer, 2020–2021.

The most variable stage is Core Banking Query, which must perform an account lookup, balance check, velocity checks, and—for chip transactions—a dynamic CVV2 (iCVV) verification. Banks running legacy COBOL cores on mainframes typically exhibit p99 latencies in the 200–350ms range here; modern cloud-native issuers can achieve under 80ms at p99.

"The p99 latency of your core banking is the ceiling of your customer experience. You cannot build a 1-second payment on a 600ms mainframe call."

Key Data Elements in an Authorization

The critical data elements in a 0100 Authorization Request are:

DENameFormatPurpose
DE-2Primary Account NumberLLVAR n..19The PAN (card number). Never stored in plaintext post-auth.
DE-3Processing Coden6Transaction type. 00xxxx = Purchase, 01xxxx = Withdrawal, 20xxxx = Credit.
DE-4Amount, Transactionn12In minor currency units (cents). €10.50 = 000000001050.
DE-22Point of Service Entry Moden3051 = chip+PIN, 071 = contactless NFC, 010 = manual.
DE-35Track 2 Equivalent DataLLVAR Z..37PAN + Expiry + Service Code. Present in mag-stripe; absent in chip.
DE-55ICC Data (EMV Tags)LLLVAR b..510Chip data block: ATC, ARQC, TVR, CVR, amount authorised.

Decline Reason Distribution

Not all transactions are approved. Understanding the decline split is essential for fraud and risk teams—and for product teams measuring checkout abandonment caused by technical failures.

Authorization Outcome Distribution
Aggregate across consumer debit and credit cards, Western European issuer, Q1 2021. Response codes per ISO 8583 DE-39.

Clearing vs. Settlement

Authorization is a hold—it does not move money. The actual fund movement happens in two further stages:

Clearing (T+0 to T+1): The acquirer submits a clearing file to the card scheme, containing all captured transactions from the authorization day. This is typically a batch process running at the end-of-business cut-off, though some schemes now support near-real-time clearing.

Settlement (T+1 to T+2): The card scheme performs multilateral netting across all acquirers and issuers, then initiates net settlement payments through central bank systems (TARGET2 in Europe, Fedwire in the US). Issuers debit cardholder accounts; acquirers credit merchant accounts.

Merchant Net Settlement Amount
Netmerchant = Σ(Gross Sales) − Σ(Refunds) − Interchange Fees − Scheme Fees − Acquirer Margin

Reconciliation and the Break Problem

Every cleared transaction must be reconciled against the original authorization. Three types of breaks are common:

Amount mismatches arise when the final cleared amount differs from the authorized amount—common in hospitality (tips, fuel pre-auth). The scheme permits partial clearance or over-clearance within defined tolerances.

Orphan clears occur when a clearing record has no matching authorization—caused by voice authorization, system failures, or mis-routed transactions.

Expired authorizations happen when the clearing arrives after the authorization has lapsed (typically 5–7 days). The issuer may not honour the clearing, creating a liability dispute.

Daily Settlement Reconciliation Rate (12-Month Trend)
Straight-Through Processing (STP) rate — percentage of cleared transactions automatically matched to authorizations without manual intervention.

The EMV Chip: Cryptographic Guarantee

Modern chip cards perform a challenge-response authentication with the terminal and issuer. The card generates an Application Cryptogram (ARQC) using a session key derived from the card's master key and a diversification factor. The issuer verifies this cryptogram, providing a cryptographic guarantee that the physical card was present—not just the card number.

EMV ARQC Verification — simplified
from Crypto.Cipher import DES3
from Crypto.Hash import CMAC

def derive_session_key(icc_master_key: bytes, atc: int) -> bytes:
    """Derive per-transaction key using ATC diversification (EMV 4.3 A1.3)."""
    diversification = atc.to_bytes(2, 'big') + b'\xf0' + b'\x00' * 5
    diversification += atc.to_bytes(2, 'big') + b'\x0f' + b'\x00' * 5
    cipher = DES3.new(icc_master_key, DES3.MODE_ECB)
    left  = cipher.encrypt(diversification[:8])
    right = cipher.encrypt(diversification[8:])
    return left + right

def verify_arqc(session_key: bytes, pdol_data: bytes, arqc: bytes) -> bool:
    """Verify the Application Request Cryptogram using 3DES-MAC."""
    mac = CMAC.new(session_key, ciphermod=DES3)
    mac.update(pdol_data)
    expected = mac.digest()[:8]
    return expected == arqc

Implications for Data Engineering

For anyone building data pipelines over payment data, several architectural considerations arise directly from this flow:

Event duality: Authorization and settlement are separate events with potentially different amounts, timestamps, and system identifiers. A naive schema that models "transactions" as a single entity will produce incorrect balance calculations.

PAN handling: PCI DSS requires that full PANs are never stored in analytical systems. Pipelines must tokenize or truncate (first 6 + last 4) at the point of ingestion. The BIN (first 6–8 digits) retains analytical value for card scheme, product type, and issuing bank identification.

Latency asymmetry: Authorization events arrive in real-time (sub-second); settlement arrives as batch files hours or days later. Any analytical system that joins these must handle late-arriving data gracefully—ideally with a watermark strategy rather than a fixed batch join.