Retrospective on a Payments Workflow

A story told in the infrastructural tense

I want to tell a story about a piece of infrastructure we built back in 2019, when NUMI was an on demand grocery app in Nairobi, Kenya.

In part, I’m recording this to celebrate the work we did in Kenya. But much more so I want to capture a snapshot of a scene in metamorphosis. I’m certain that in a few years it will be hard to believe, let alone remember, all the arcane things we had to do in order to make online commerce work back then.

The Challenge

Ecommerce players in emerging markets almost all are forced to accept cash. In part this is because of the dominance of cash as a payment method. But even more so because customers won’t part with hard-earned money until they can verify the merchant has made good on their half of the trade.

In Euro-merica, customers know they can dispute a charge if the merchant doesn’t deliver—both literally and figuratively. But in emerging markets, card adoption is still low. So customers there don’t have the same default-trusting posture as their American and European counterparts.

While our cash management challenge was far from unique, it was certainly revelatory. To solve it we had to reassemble the world around us, venturing into territory where there was little or no prior art. Through the process we learned how commerce, payments, trust, transport, and urbanism all interlocked in Nairobi in 2019.

And our specific experience of the problem, as a pre-seed grocery delivery startup, uncovered additional layers that few ever get to see.

Non-Deterministic Checkouts

As far as trust went, it was already bad enough that nobody had heard of us. Even worse, on-demand grocery suffered from the curse of non-deterministic checkouts. On grocery apps, the basket and price on the “Confirm Order” screen does not necessarily reflect what you’ll receive or what you’ll owe.

The non-deterministic checkout curse is especially strong in the beginning because most on demand grocery apps tend to start as guerrilla operations: they pick from established grocery stores rather than holding their own inventory. This model, often called hyperlocal, is a great hack for instantly getting the same breadth of product assortment as a legacy supermarket.

But hyperlocal is terrible for predictability because the on-demand app has no idea what’s in stock. So a lot of the time, on demand grocery checkout screens don’t really know for certain how much you’ll owe when you place your order. They have to add form fields like “What should we do if items are out of stock?”, which would seem absurd in any other ecommerce experience.

Here’s what ours looked like. Always tryhards, we even had an offer where we’d scour the supermarkets of Nairobi until we completed your basket.

Even as an on-demand platform gets bigger, the non-deterministic checkout problem never fully goes away. It isn’t something that on demand grocery apps scale out of, but more like a force of nature that they negotiate with all the way up.

Consider Instacart. It’s now a publicly traded company teeming with world-class engineers and operations researchers. It has tens of millions of deliveries under its belt. It still can’t tell you for certain how much a tomato will cost:

Nature has the last laugh, as it refuses to produce tomatoes at a consistent mass.

So not only were we operating in an environment with justifiably low trust in ecommerce, we were asking customers to entrust us with their most intimate purchase—their food. We were also pitching them on a product that didn’t know how much you owed until you we had completed shopping for you and you were on the hook to pay.

It made total sense then that customers wouldn’t pay until they saw the goods in person and on their doorstep. We didn’t take any of this personally. Quite the opposite: we were a team of excellence seekers who loved the challenge of winning over customers with such exacting standards.

But this added a complication to our entire business. Low baseline trust in ecommerce—worsened by the unique non-determinacy of grocery—and the state of the payments infrastructure we sat atop, all pointed towards one uncomfortable conclusion:

Our whole business had to build itself around payment upon delivery.

Kenyan Payments circa 2019

In the US when you check out on Instacart and they don’t know how much a tomato will cost, they place a pre-authorization on your credit card. For many reasons, this option was off the table for us.

First off, in 2019 Kenya, cards were still very new and penetration was very low. And the people who did have cards used them exclusively for making international purchases like Netflix or Amazon.

Plus, card numbers are seen as a much more sensitive piece of information than they are in the United States. In part it’s that cards in Africa are usually directly connected to your bank account, and consumer credit—at least as we know it in the US—is quite rare. So card charges in Africa represent a much more direct claim on your personal cash balance than they do in America.

Plus, from the consumer’s perspective, it’s not clear that the resolution process for disputing a charge will run as swiftly and favorably as we expect it to in the United States.

When we consulted our credit card processor, a leading player in the space, they advised against pre-authorization for all of these reasons.

Enter M-Pesa.

M-Pesa is a mammoth payment network run by Kenyan telco Safaricom. Its customer base constitutes the majority of the adult population of Kenya. It is, without exaggeration, one of the most staggering accomplishments of human ingenuity in the last 50 years.

Years before the arrival of credit cards or even broadband internet, Kenya was the first country at its income band to crack cashless payments. In the years since M-Pesa’s launch in 2007, telcos around the world have created dozens of national mobile money networks by copying M-Pesa’s playbook. For those uninitiated to the magic of M-Pesa, let me give a very lossy compressed summary of its provenance and how it works:

SIM cards in Kenya (and markets like it) have historically been provisioned on pay as-you-go plans. Customers top up on minutes on their SIMs by purchasing scratchable airtime cards.

Example of an airtime card from 2004, on sale.

These cards are available at Kenya’s informal convenience stores, locally known as “dukas”. They’re Africa’s answer to New York’s bodegas or Japan’s konbini: small-scale community stores that sell fast moving consumer goods to clientele situated within a few hundred feet of their shops.

And like bodegas and konbini, dukas in the last couple decades have expanded into telephony services as mobile phones have become ubiquitous among their consumers.

A duka in Nairobi, offering mobile money services for at least 3 telcos.

What sets apart dukas from bodegas or konbini is the sheer density of their coverage. Because a duka can set up shop on a roadside without a permanent physical structure, there are more than 10x more dukas per capita in Kenya than there are konbini in Japan. These dukas became the primary channel through which they distributed airtime cards to consumers.¹

For the telcos, building out a distribution network to reach these kiosks was a massive undertaking. But it was necessary to meet the customer where they were. They also added the ability to send minutes to other phone numbers, something that family members would often to do to gift each other airtime.

One Kenyan telco, Safaricom, realized that they they could repurpose the infrastructure they had built for distributing airtime into a giant payments network.

Here’s an extremely simplified walkthrough of how:

Imagine if a SIM could carry two separate balances: one for minutes, and one for money.

The same flows used to change cash into minutes could change cash into mobile money. The same accounting systems built to maintain a massive ledger of pay-go plans could now maintain a ledger of mobile money wallets. And the ground game they had developed to distribute top-up cards could be used to create a network of mobile money agents.²

You can think of these agents as human ATMs, or micro bank branches.³ You can hand them cash, and you’ll receive an SMS from Safaricom notifying you of your new M-Pesa balance. Or you request a withdrawal, and you’ll receive a debit notification SMS, after which they’ll hand you cash. Like ATMs, agents earn a small convenience fee whenever a customer withdraws.

The brilliance of M-Pesa is that it was an infrastructural breakthrough that required almost no net-new ingredients: just the existing human will and national-scale logistical competence. M-Pesa doesn’t even require a smartphone. It runs entirely through the SIM using an esoteric legacy protocol called USSD.

M-Pesa has been an astounding success. By the time we arrived in Kenya, its annual transaction volume was equal to half of Kenya’s GDP. And it had lifted a million Kenyans out of poverty.

M-Pesa’s APIs

M-Pesa was amazing, but its initial product left a gap that prevented its use in true internet-native commerce.

Namely, while you could send money to another mobile wallet (and later a merchant wallet), there was no way to programmatically attach metadata to the transaction. This prevented a merchant from conclusively reconciling M-Pesa Payment 123 with Order ABC.

After M-Pesa allowed customers to send money to merchant wallets, they added the ability to specify an “account number” for that merchant. The utility companies made great use of this to design bill payment workflows completed entirely over SMS and USSD. A customer would request to pay a certain amount to a utility via SMS or USD; the the utility would send back a token corresponding to the request; the customer would pay the merchant wallet for the utility and include the token as the account number. Still there was no way for the merchant to initiate a payment request from a customer, with the right amount, account, and metadata. So placing an order and paying for it were still totally disjointed.

In 2018, M-Pesa’s developer team opened beta access for a new API to issue payment requests to customers programmatically. Called STK Push, it allowed merchants to initiate a request to pop open the M-Pesa USSD confirmation screen on a customer’s phone. All the customer had to do was enter their PIN and the merchant would receive funds to their wallet instantly, to an account number they could specify. It was Kenya’s answer to the 3DS flow that European credit card users know so well.

When the payment succeeded, you’d receive a webhook with the payment metadata including transaction id, customer phone number, and amount. STK Push was nothing short of revolutionary. For the first time, Kenyans could complete a purchase entirely on the web, using the dominant cashless payment method.

Here was the workflow, in a nutshell:

We were proudly one of the first production deployments of STK Push.

As an early adopter, we had a front row seat to watch the proverbial cement dry.

Cement Drying Problems

My friends and I invoke “cement drying” as a reference to the opening of Disneyland, where Disney was building so maniacally to the last minute that it had to bring in helicopters to dry the cement, still wet, as the first guests streamed into the park.

Many of STK push’s cement drying problems were the kind you’d expect from new API infrastructure rolled out by a legacy telco. For example, developers trying to access the M-Pesa API credentials were subject to an agonizing yak shave, especially if they were developing on Macbooks.

Here’s a snippet from our internal documentation, meant to onboard our developers to getting set up on the M-Pesa APIs, written by Ishuah Kariuiki, our founding engineer:

In order to access the M-PESA Organization Portal you have to install an M-PESA certificate on the specific computer that you intend to use. The certificate is valid for two years and can only be installed on computers running Windows 7.

…

If the computer you intend to use does not run Windows 7, you can create a VM on VirtualBox and install Windows 7.

1. Using the Admin email (in NUMI’s case, *********@numi.tech), send a blank email to M-PESABusiness@safaricom.co.ke. That same email should respond with the password. Alternatively you can call Safaricom Business on 0722002222 with a whitelisted number. (This password is updated every week)

2. On the Windows 7 computer, run Internet Explorer and open the url: https://vmtke.ca.vodafone.com/certsrv. There should be a login prompt. Enter the username and password from the email/call in step 1.

3. Fill in the name, email and company field with valid entries that you used to register your till number and submit.

4. If the request is successful you will be redirected to a page with the title ’Certificate Pending’. You’ll wait 24-48 hours. Please call after 24 hours to ensure your request is on the queue.

Note that at this time, Windows 7 was a decade old, and the last version of IE had been released 6 years prior. Once you got the credentials, and got working in production, you were confronted with another challenge: STK Push didn’t always send webhooks upon payment completion.

The first M-Pesa app developers, us among them, discovered this through awkward production incidents. Typically a customer would find themselves gated from accessing value on an app because M-Pesa’s webhooks had not fired for the transaction that they just completed.

Eventually developers began to realize they needed to reconcile payments multiple ways. This included the proactively query about a specific payment via the M-Pesa transaction status API. And if that failed, they would need to wait for the data dump cron job on the VPN-gated FTP server that they could set up with Safaricom’s developer team.

To the Safaricom API team’s great credit, they were pushing the limits of their own infrastructure trying to deliver a Stripe-like experience. And through the resourcefulness of Silicon Savannah engineers, working both their command lines and their rolodexes, online-native payments began to emerge in Nairobi.

Upgrades

It wasn’t just the M-Pesa developers who had to scrape the rust off their payment rails. We discovered that many customers’ SIMs were on older versions of firmware that were backwards incompatible with STK Push.

And neither the merchant nor the customer would receive an error message if an STK push request was issued to an old SIM. Instead the customer would just be waiting, as the driver awkwardly tried to debug what the problem might be.

We eventually found out from Safaricom customer support that subscribers with old SIMware needed to punch in a series of USSD commands to receive an over the air upgrade to their SIMs.

So we created the first payment loading screen that I’ve ever seen, which provided the USSD sequence to install an OTA upgrade on their SIMs.

Proof of work: our STK push loading screen included SIM upgrade instructions

We took pride in shipping error copy so actionable that it drove our customers to the frontiers of the latest payments infrastructure. Error screens like this are relatively rare in the history of user interfaces, because they only make sense in settings that are changing rapidly.

Here’s what our M-Pesa flow looked like once it considered the cement drying challenges, and of course the inevitable insufficient funds errors:

The nature of the problem made it so that our flow of goods and money looked very different from any ecommerce that we, two internet-native US-raised founders recently transplanted from San Francisco, had ever seen.

While STK Push was a big improvement, we still saw about 20% of our orders paid in cash. In a game where we wanted to maximize our growth, increase word of mouth and create local network effects, we couldn’t ignore this 20%. The lore has it that Flipkart became ascendent in India because they accepted cash on delivery.

Like Safaricom did with M-Pesa, we had to meet our customers where they were.

The Cash Management Status Quo

Our competitors were on-demand platforms live in multiple countries, with billions in funding. We naturally assumed they had figured out the best practices in managing cash. Our riders, who delivered for all the platforms, told us how they did it.

Every platform let riders hold the cash that they picked up over the course of a week, and then required them to come back into the head office. There they’d drop it off and reconcile it with someone in finance.

Contractors on on-demand platforms everywhere have to worry about timeliness of payouts. At its core this is a working capital problem. Typically it’s the case that the contractors are waiting on the platforms to pay them out.

That’s because the platforms are getting paid by the customers via card. After a few days, the funds settle from the platform’s payment processor into the platform’s bank account, where they will leave at the end of the pay period to be transferred into the contractor’s bank account.

The story gets messier in markets where the platforms accept cash. The money flows in two directions: from platform to contractor for cashless payments, or from contractor back to platform for cash payments.

Who carries working capital for whom becomes a bit of a question of political economy. Riders liked getting paid in cash because it allowed them to get their wages up front and use the float to cover their fuel and maintenance costs. The platforms prefer to get paid up front, and ideally to skip handling cash entirely.

For the platforms, handling cash introduces so much more headache surface area. It comes with a unique set of operational challenges:

1. Finance managers and riders disagreements on exactly how much cash is owed, how much cash traded hands, or whether certain bills are too damaged to be accepted at a bank

2. A bank teller and finance manager disagree on the same

Each problem has a solution. But they add process to your flow of funds. And as you add process, you increase the number of potential failure points.

One way to sidestep these thorny questions of institutional design is to deflate the stakes. So we tried that first.

We required all riders to drop off their cash with us at the end of each shift. This reduced the magnitude of each cash headache and allowed us to defer implementing a more robust balance tracking system.

Marchetti’s Motorcycles Strike Back

While our daily cash remittance policy made cash management problems smaller, it clashed with the urban layout of Nairobi.

In Marchetti’s Motorcycles, my friend Osarumen Osamuyi excellently breaks down how the speed of transport shapes the spatial development of Africa’s megacities. As the mileage that a person can travel in an hour grows, the city will naturally expand outward from the center to reach that limit.

We began to feel these forces we reached order volumes that 1) allowed us to batch multiple orders in the same trip and 2) made deliveries to Nairobi’s peripheral exurbs more and more frequent. For most our time, we picked from a single grocery store and dispatched across all of Nairobi.

Here’s what how our order density played out geospatially:

A heat map of our orders as of October 2019 (n=3352).

Much of our order volume clustered around a few affluent districts. And then the rest was spread across the city, in smaller clumps of neighborhoods where Nairobi’s young professional families flocked.

An aside: our team came up with some novel storage solutions as a result of this batching which—like the payment workflow created—we were surprised to see others hadn’t tried before.⁴

One of our delivery riders showing off their self-branded storage container.

The riders began batching their orders so that their routes would go in order from the Carrefour at Junction Mall (our primary shopping site), all the way out to last stops in the exurbs. At the end of their shift, they’d try to plan their last deliveries to be along their way home.

This was the natural way to sequence deliveries. Having to come back to our office to drop off cash created a huge pain for the riders.

Consider this illustrative delivery route. The rider departs from Carrefour Junction (1), making multiple drop offs until the last customer at Stop 2. Rather than go home directly to Stop 4, they’d have to backtrack to our headquarters at Stop 3 to drop off their cash.

Hypothetical example of a delivery made by a rider who gets paid in cash.

In Nairobi’s ever-present traffic, this could easily add 45 minutes or more of driving time to a rider’s day. And these were 45 physically and cognitively expensive minutes, as the rider spent them either weaving between cars or enduring the micro-vibrations of their motorcycle engine’s as they drove on a freeway.

Our riders and my cofounder Harrison (who himself became a boda rider), devised a solution that worked beautifully.

Our Final Cash Management Workflow

If the customer informed the rider that they’d be paying in cash, the rider would open their rider app and record a cash payment for that order. We’d have it marked in our system that the rider owed the value of that order in cash. That much put us at parity with the other platforms.

But rather than having the riders come back to the office, they would remit that cash back to us via the same STK Push rails we had built for our customers.

After their last delivery, they’d ride just a couple hundred feet to the nearest M-Pesa agent. They’d load up their M-Pesa wallet with the cash they’d collected on that shift. And then they’d use our rider app to initiate an STK Push request to our till number. Using the metadata on the transaction, we’d know this was a rider remitting cash back to us. After verifying the payment, we’d deduct the value of the STK push transfer from the balance of the rider’s cash account.

This meant that from the rider’s perspective, every single M-Pesa agent was effectively a NUMI cash remittance agent as well. Suddenly NUMI went from 1 cash drop-off site to hundreds of thousands of them.

Here’s the final workflow visualized:

When Geoff Charles was running his Fintech World Tour before he started at Ramp, we got dinner one night in Nairobi. I told him about our policy of accepting cash, and he said “that must be an operational nightmare.” I was surprised that in his travels he hadn’t come across a better solution than the ones we had encountered from the other platforms in Nairobi. I checked with friends in Latin America and Southeast Asia, and they similarly hadn’t heard of any examples.

I’ve wondered why we were, to our knowledge, the first people to arrive at this cash management workflow. We were a creative and determined team, but that was far from unique. Simply due to their size, the platforms in other markets had many more creative and determined people building them.

The key ingredient in our solution was the magic of Nairobi. Nairobi has always been an early adopter city, and we happened to be there when new infrastructure unlocked. It’s not clear to me that there was another payment ecosystem like ours in 2019, where there was a ubiquitous mobile money service that had unrolled a payments request API for merchants.

And what about our local competitors? Why didn’t they come up with this solution? My guess is their scale and maturity focused them on running their existing processes well rather than inventing new ones. Our solution turned out to be very technically involved. It’s unlikely that the local on-demand platforms, themselves branches of companies based in Europe and SF, had the engineering resources available to build something similar.

When we launched, most our competitors hadn’t even implemented STK push yet in their native checkout flows. Their wisdom and restraint—holding off on adopting unproven infrastructure—was the opposite of our boundless, earnest naivety.

Ultimately, our Kenyan on-demand dreams didn’t pan out. But it was a joy to build at the frontier of the possible, and in our small way, help inch that frontier forward.

Made it this far? Let’s be friends on X (nee: Twitter)

1. This is an imprecise lowball estimation. There are on the order of 50,000 konbini in Japan, and around 250,000 dukas in Kenya. So about 1 konbini for every 2,540 people in Japan, and 1 duka for every 228 people in Kenya. The vast majority of mobile money agents are dukas.

2. There is the much harder problem of float management which we can’t get into here, but it is the infrastructure built on top of the infrastructure, so to speak.

3. The irony of “human ATM”, once the acronym is expanded, is not lost on me.

4. The standard boda boda storage is a fiberglass glass box mounted to the back of the bike. It stores about 20 liters or 5 gallons and costs about 3 weeks of a boda rider’s wages. Installation can take up to 1 week for a mechanic with a custom rig for each bike. We figured out that we could buy rubber containers from a local factory with about 3x the storage capacity, and about 1/10th the total cost. We’d strap them to a bike with bungee cords, and could stack up to 3 at a time.