Ecommerce Nightmare(s) Before Christmas December 7, 2020 by Paul Byrne Business Environment COVID-19 seasonal christmas mistakes Is Christmas a Nightmare or a Blessing for Digital Merchants? As soon as I became involved in digital commerce, I became aware of the many difficulties the industry faces around this time of year. Retail, in general, faces many challenges when it comes to the holidays because many merchants are so dependent on holiday sales that any slight hiccup can ruin an otherwise promising year. 2011 ProStores Outage When I attended conferences (those were the days!) with colleagues who were involved in ProStores, inevitably, the great ProStores Outage of 2011 came up. ProStores was a SaaS (Software as a Service) ecommerce platform owned by Ebay and one of the first platforms Razoyo worked on. At the time of the outage, Razoyo worked with hundreds of merchants doing business on the platform. We all woke up on Thanksgiving morning with buzzing phones and inboxes teaming with angry, often impolite, emails about stores being completely offline and customer support at ProStores was completely MIA. So what happened? The myth goes something like this: a substitute janitor in the data center that hosted ProStores was cleaning the plexiglass cover over the panic switch. Because of the amount of current required to run the servers, data centers have kill switches to shut off all power to the facility in case of fire so as to not electrocute first responders. As you may have guessed by now, the employee had accidentally pushed the kill switch at around 4:30am. Getting the power back on was an arduous process that first required getting an engineer who had the keys to restore power onto the site on Thanksgiving morning. The power coming back on is done in phases and then there were numerous issues with corrupt databases and fried networking equipment that had to be dealt with. Fortunately for most merchants, everything was back online in time for Black Friday and Cyber Monday. Nonetheless, I can tell you that our service migrating stores from ProStores to Magento really picked up the next week! How to protect yourself from this type of nightmare Most SaaS platforms today run redundant data centers for this very reason. Make sure you are using one of them. If you are on open source (Magento, OroCommerce, Spree, etc.), talk to your developer about your options well before your peak sales period to put a plan in place. Prepare a copy of your site on a different data center or on a different platform if you are on SaaS. Make sure your DNS is hosted somewhere you will have access to if your platform goes down so that you can switch your traffic. There will still be issues, but, at least you won’t lose your sales. Bad Company Sometimes it is not the platform itself, but the services you are connected to that can cause the most problems. Razoyo had a client who used a homegrown order management system for about 5 years to send orders from Magento to their warehouse. We had no control over that system and the client failed to invest in our recommendations for allowing Magento to queue orders separately. As they became more and more successful with each Black Friday sale, we inevitably had a developer on call to monitor the system and make adjustments on the fly when the system inevitably started wheezing and moaning under the tremendous traffic load. How to protect yourself from this type of nightmare Stress-test your systems before the holidays and/or spread out your sales over multiple days to keep the server load down. If you are on a SaaS platform, check with your provider to ensure that handline spikes in traffic are included in your plan. If not, maybe reconsider your holiday sales strategy. More Bad Company On other occasions either the UPS, USPS or Fedex APIs have gone down at peak sales periods. Many stores are crippled at checkout when they cannot retrieve shipping quotes from couriers resulting in the absence of shipping options at checkout for customers. How to protect yourself from this type of nightmare Even if you only use one courier, configure a second one and leave it disabled. Make sure you know how to switch if an API goes down. There are also API monitoring sites available that will send you notifications if your couriers API does crap out. Have your developer simulate an API outage on a testing site so that your team is sure your solution is solid and that you know how to quickly switch to your failsafe. Why Are All The Products Gone? On two occasions, we have had clients delete their entire product database during off hours just before a holiday sale. One poorly-formatted full-catalog upload sheet is all that’s needed to induce this particular nightmare. How to protect yourself from this type of nightmare Test any product uploads you are doing on a staging site or sandbox before uploading to production. If running your promotion requires that kind of upload and you don’t have anywhere to test it, break the file up into small chunks. Upload a single line first and make sure it didn’t kill products. Before uploading, download your entire product database or have a developer take a backup of your product tables. 2020 COVID Style With the recent explosion of activity in ecommerce due to COVID-19, many Razoyo clients have been experiencing their best sales growth ever. The acceleration of the trend toward shopping online has continued, so you would think we’d be in store for a phenomenal holiday season. However, almost every merchant whose catalog exceeds 100 products has some items or parts sourced from China or elsewhere in Asia: supply line interruptions are a dark cloud on the horizon. How to protect yourself from this type of nightmare If you are in this situation as of today (early December) it may be too late to rectify. Many of our merchants are already taking a look at their product portfolios strategically. Locating and nurturing manufacturers in their own hemisphere as primary and back up suppliers will be a lesson learned from this crisis. Learning from your mistakes Even if you don’t have a major disaster this holiday season, you will learn things from your holiday rush. Keep notes as issues arise. Set up a post-holiday learning session with your customer service, operations, and development teams. As you walk through each of the items in your notes, discuss what you can do to avoid the issues next year. Many organizations have turnover, switch platforms, or change developers in the new year, however, if you have a catalog of issues prepared now, you can put together solutions to improve your future holiday sales year after year. Happy Holidays!