This is a post analysis of an issue that arised with a former client. It wasn't a difficult bug to diagnose as much as it was comical.

It all started one morning when we received a ticket from the client that some orders were not totalling correctly.  Upon further investigation, we realized that some products were not being charged at all.  They were receiving the products completely free. What made it bad is that these were items could range from $400 to well over $1500.  Naturally, we tried recreating the issue by adding the same products to our cart, but no matter what we did, we could not reproduce the error. 

In the old version of the website, to get any prices or information of the store, they would make API calls to the pricing software called Siriusware.  This software is quite literally some ancient legacy software that comes to a crawl during peak hours of business. In order to remedy this, we create a synchronization job that runs once a day during off peak hours.  This would synchronize everything including products, prices, and availabilities from Sirius into our own database.  What I have come to realize is that a lot of business logic is simply synchronizing data from one system to another.

So this meant that the prices on our store were stored in our database, even though the source of truth came from Siriusware.  This meant that the next step was to verify the prices stored in the database.  Were there any prices that were incorrect or missing?  To properly understand how the prices are saved in the DB, I should preface that the prices are not stored based on product, but based on variants. The same product could technically have different prices.  For example the same pair of headphones can have different colours and each colour could have a different price.  What was silly was that in rare cases, a variant could also be broken into variant items, and each variant item would have their own price.

For the most part, prices stay the same for each day. But there are times where a product will be on sale for a short period of time.  In order to account for this, we end up storing each product, variant, and variant item, by day.  Now that might not seem so bad at first, but if you have 100 products, and each product has 5 variants and each product has another 3 variant items.  If you save a whole years worth of products, we are looking at a cool 1000 x 5 x 3 x 365 = 5.5 MM entries in the db. While 5.5 million entries isn't the worst, it is not the only table that needs to be accounted for.  So in order to reduce the size of the prices table, we would periodically prune prices from older days.  This made sense because we don't need to check the prices of products from 3 weeks ago if we were making a purchase today right?....RIGHT?

Well that is where we were wrong.  When you add an item to the cart, it saves the date as well.  This means that if you save an item in your cart, when we do the price lookup, it will request the price of the product/variant/variant item using the date that the product was added to your cart.  This was by design because the site administrators wanted users to be able to add products to their cart and have the price locked in until they make the payment.  But since we were pruning older prices, what ended up happening is that the getting the price would fail. 

Instead of throwing an exception, if the price lookup fails, it simply replaces the price with a null value, which conveniently gets converted to a BIG FAT 0.  This meant that clients that left items in their cart for 3 weeks or more would automatically be able to check out their items for FREE.  And of course, Siriusware being the great software it is, has no guards in place to actually check the prices we are sending is in sync with the prices in their system and simply blindly approves whatever we send them.

Ultimately, the issue was fixed by simply getting the product's price for today in the event that the price lookup fails for that particular day.  I recall needing to update the pricing information in the cart logic.  Unsurprisingly, although the fix is seemingly simple, the checkout and cart flow was quite difficult to follow.  Regardless, we got the job done.

Perhaps I am wiser now, but at the time, I do not recall being intimidated or scared even though I was updating the checkout logic, which could technically adversely affect all purchases done on the site.  While this is technically a small change, in hindsight, the issue could have easily become quite large if done incorrectly. In addition, I think I was always a little trigger happy when it came to deploying to production.  You could say that I was younger and less experienced.  There is nothing quite like the boldness of a junior developer.