Even Bigger Troubles With Big Data

Even Bigger Troubles With Big Data


Vivek Sood




June 12, 2019

Troubles in big data are starting to emerge.

I have written about these earlier and warned the clients to first make sure their small data is working as intended before jumping into big data.

If you cannot control your small data, your ERP system, your EIS or your cloud then forget about BIG data. All you will have is a big mess instead of a small mess.

Despite companies spending hundreds of millions of dollars in their systems renewal efforts, user satisfaction remains stubbornly low. As I say in my book ‘The 5-STAR Business Network’:

Not too many years ago, a very large corporation operating worldwide, made news with the downgrading of their earnings expectations due to supply chain system’s implementation setbacks. The expectation was that the new system would reduce the new production cycle from 1 month to 1 week.

Furthermore, it would better match the demand and supply of its products to place the correct products in the right locations and quantities, all at the right time – a very lofty goal. The company spent an enormous amount of money, exceeding US $400 million in order to achieve its aim.

However, the software system ‘never worked right’. It caused the factories to crack out too many unpopular products and not enough of the trendier ones in high demand. While making the earning downgrade, the CEO asked the rhetorical question, ‘is this what we get for $400 million?’

The market analysts were not surprised. One respected market analyst [AMR] commented, ‘fiascos like this occur all the time but are usually kept quiet unless they seriously hurt the bottom line.’

Another respected market analyst commented that while the CEO made it sound like it was a surprise for him, if he did not have checkpoints for the projects, he does not have control over his company. A third analyst commented that companies are confused by escalating market hype and too often underestimate the complexity and risks.

Another [Forrester Research] commented ‘when the software projects go bad companies are more likely going to scurry up and cover it up because they fear that they are the only ones having trouble. But far from it; our conversation and research reveals this company was not unique or the only one having this kind of trouble‘.

Despite their lofty goals, many of the large information technology deployment projects derail. It takes time for the word to filter out because, in most cases, the executives involved in the process are far too embarrassed to talk about what happened.

They do mutter among themselves; after several similar instances the mutterings become more vocal and a trend emerges where a number of people start talking about the shortcomings of the system itself or the implementation process, or of the time taken for implementation.

Because the cost of this failure is so high – greater than $400 Million in the above case – it is instructive to understand the real root causes of this failure.

All the above problems with small data are only multiplied big time when they apply to big data. However, this blog post is not about these small problems. Most companies survive these small problems by stumbling through them.

Now even BIGGER problems are emerging with Big Data. Target was always one of the poster children of big data. Highlighted in Charles Duhigg’s book and several newspaper articles were its capabilities of predictive behavioural scoring in order to maximise the revenues.

Kashmir Hill, writing in Forbes magazine online in February 2012, cited New York Times in an instance of How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did. [1]

Target assigns every customer a Guest ID number, tied to their credit card, name, or email address that becomes a bucket that stores a history of everything they’ve bought and any demographic information Target has collected from them or bought from other sources.

[They] ran test after test, analyzing the data, and before long some useful patterns emerged. … Take a fictional Target shopper named Jenny Ward, who is 23, lives in Atlanta and in March bought cocoa-butter lotion, a purse large enough to double as a diaper bag, zinc and magnesium supplements and a bright blue rug. There’s, say, an 87 percent chance that she’s pregnant and that her delivery date is sometime in late August.

The anecdote quoted in the article by Kashmir Hill where father storms angrily into target demanding an apology for encouraging his teenage daughter to get pregnant by mailing her coupons of baby stuff, only to retract the demand later on when he discovers that she was indeed already pregnant, demonstrated the power of predictive business intelligence.

This article as well as the New York Times article [2] by Charles Duhigg and the book it is based on The Power of Habit: Why We Do What We Do in Life and Business also by Charles Duhigg. I quoted this example in my book as well and cited Target’s ability cautiously. Back of my mind were the concerns about data integrity and security – which have now come true. This holiday season, Target was one of the two large retailers who felt the brunt of the hackers.

As per this news report in NBC: Target said Wednesday that the cyber criminals who breached its system used credentials they stole from one of the retailer’s vendors. “The ongoing forensic investigation has indicated that the intruder stole a vendor’s credentials, which were used to access our system,” Target spokeswoman Molly Snyder said in a statement. She declined to elaborate on what type of credentials were taken from the vendor.

Meanwhile, the Justice Department is investigating the hacking, Attorney General Eric Holder said Wednesday. While Target is not the only one to have suffered such lapses – it is one of the most serious.

Reminds me of the joke where a bank robber was asked why did he always rob banks, and he replied because that is where the money is.

The news report quoted above shows the magnitude of the theft. Target has said a breach of its networks during the busy holiday shopping period resulted in the theft of about 40 million credit and debit card records and 70 million other records with customer information such as addresses and telephone numbers.

Target has not yet specified which vendor was responsible for the breach, and whether it was an IT vendor or a supply chain vendor. Target was not the only one though.

Nieman Marcus was another high profile retailer in a similar situation, albeit on a smaller scale.

In fact, there were more; in the news report above Reuters reported Jan. 23 that the FBI has warned U.S. retailers to prepare for more cyber attacks after discovering about 20 hacking cases in the past year that involved the same kind of malicious software used against Target.

Final point this episode highlights is the axiom that you will always pay for your vendors’ sins. I use the example of BP’s oil rig in my book to illustrate that point. Will write on this aspect of the episode in a later blog.

Copyright - These concepts, frameworks and ideas are copyright of GLOBAL SUPPLY CHAIN GROUP from the time of their creation. Do NOT copy these without permission and proper attribution.


  1. These ideas and concepts will be usually expressed by our thought leaders in multiple forums - conferences, speeches, books, reports, workshops, webinars, videos and training. You may have heard us say the same thing before.
  2. The date shown above the article refers to the day when this article was updated. This blog post or article may have been written anytime prior to that date. 
  3. All anecdotes are based on true stories to highlight the key points of the article - some details are changed to protect identification of the parties involved. 
  4. You are encouraged to comment below - your real identity and email will not be revealed when your comment is displayed.  Insightful comments will be  featured, and will win a copy of one of our books. Please keep the comments relevant, decorous and respectful of everyone. All comments represent opinions of the commentators.

Vivek Sood

Our Quick Notes On Five Flows Of Supply Chain Management

Part of our new “Quick Notes” series – this report answers your most pertinent questions of the topic.

  • What are the five flows of SCM?
  • Why are they important TO YOU?
  • How can you map, track, and optimise these flows to serve YOU?
  • What is the importance of difference between "Supply Chain" and "Value Chain"?
  • What are the stellar case studies of each of the five flows?


USD 20


What Else Exists Besides The Product Flow In The Supply Chain?
Experimental Models In Supply Chain Management
Finance, Sales, HR, Supply Chain – Where Do Most Effective Business Transformation Leaders Come From?
  • With the advancement of technology and the internet, a large amount of data started getting accumulated. Companies started facing difficulty to store this data. Even they realized that they are not utilizing the whole amount of data they are storing.

    Big Data is growing continuously and there is a need to store and manage this data. Big Data comes with big issues to deal with. Consider a small amount of data, it is easy to store, manage and analyze. With data getting bigger it becomes difficult to store it. This is where problems arise and these problems can affect businesses in various ways.

    • Big data is often characterized by 3Vs:

      The extreme volume of data, the wide variety of data types and the velocity at which the data must be processed. Although big data doesn’t equate to any specific volume of data, the term is often used to describe terabytes, petabytes and even exabytes of data captured over time.

  • Agreed! my point of view, First work to manage and create small data and get verify it’s working or not? Then we should move forward to convert it into big data.

    • Data science projects offer you a promising way to kick-start your career in this field. Not only do you get to learn data science by applying it, you also get projects to showcase on your CV! Nowadays, recruiters evaluate a candidate’s potential by his/her work and don’t put a lot of emphasis on certifications. It wouldn’t matter if you just tell them how much you know if you have nothing to show them! That’s where most people struggle and miss out.

  • While Big Data offers a ton of benefits, it comes with its own set of issues. This is a new set of complex technologies, while still in the nascent stages of development and evolution. Some of the commonly faced issues include inadequate knowledge about the technologies involved, data privacy, and inadequate analytical capabilities of organizations. A lot of enterprises also face the issue of a lack of skills for dealing with Big Data technologies. Not many people are actually trained to work with Big Data, which then becomes an even bigger problem.

  • Big data is known as the information that arrives from numerous sources. Another striking feature of this information is it is prone to continuous change. Today, most companies hold immense volumes of classified data. Analyzing this data can help businesses with actionable insights that help improve their decision-making. Upon analysis, data offers varying insights on market trends, competitor moves, and customer sentiments. By using these insights, authorities can decide what they can do to stay ahead of the competition.

  • Big data presents a number of challenges relating to its complexity One challenge is how we can understand and use big data when it comes in an unstructured format, such as text or video. Another challenge is how we can capture the most important data as it happens and deliver that to the right people in real-time. A third challenge is how we can store the data, and how we can analyze and understand it given its size and our computational capacity. And there are numerous other challenges, from privacy and security to access and deployment.

  • {"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}