Data Analytics is about understanding your data and using that knowledge to drive actions. It reveals the trends and outliers within the data which might be otherwise difficult to note. It is a scientific way to convert raw data into information that helps guide difficult decisions. A number of statistical tools and softwares are available to perform data analytics. The nature of data and the problem which needs to be solved using the insights from data guides the choice of statistical tools and techniques. Domain knowledge and expertise are also very important to interpret and apply the results obtained from analytics. Lastly, in our experience, the best data analysts are those who have the ability to dig into the data but can also layer common sense and domain knowledge into their recommendations.
Businesses are using analytics to make more informed decisions and to plan ahead. It helps businesses to uncover opportunities which are visible only through an analytical lens. Analytics helps companies to decipher trends, patterns and relationships within data to explain, predict and react to a market phenomenon. It helps answer the following questions:
What is happening and what will happen?
Why is it happening?
What is the best strategy to address it?
Collecting large amounts of data about multiple business functions from internal and external sources is simple and easy using today’s advanced technologies. The real challenge begins, when companies struggle to infer useful insights from this data to plan for future. Using analytics businesses can improve their processes, increase profitability, reduce operating expenses and sustain the competitive edge for the longer run.
Building analytics function requires long term commitment and extensive resources. An organization has an option to seek analytical help from in-house resources or from outside analytical vendors or use both in parallel. Any organization needs to spend considerable time and money to recruit and train in-house analytical help. At times they may not possess the required know-how to recruit such specialized staff or decide on the technologies that would be best suitable for carrying out analysis. In these circumstances they rely on analytical vendors like IQR Consulting. Such vendors can closely work with the management team to help the organization to adopt analytics. The organization has to trust and co-operate with the vendors while sharing their data and researching it to make the analytics engagement a success. Organizations can follow another model in which they build an internal team to manage their relationships with an external analytical vendor. Many analytically mature companies resort to this to supplement their internal efforts
A typical analytics project or engagement is generally divided into the following four stages:
Stage 1 - ‘Research’ where the analyst helps to identify and understand the problems and issues that the business is facing or would be encountering in the future. At this step there is significant interaction between the management team and analysts.
Stage 2 - ‘Plan’ where the analyst helps decide what type of data is required, sources from which the data is to be procured, how the data needs to be prepared for use and what methods to be used for analysis.
Stage 3 - ‘Execute’ where the analyst explores and analyzes data from various angles. The analysis paves way to interesting results that are shared with the management. Based on these results, strategies are formulated to tackle the problems identified in stage 1.
Stage 4 - ‘Evaluate’ where the analyst measures the results of the strategies formulated and executed. This stage helps learn and revise future strategies and processes.
A strategy built using analytics is a set of simple implementable recommendations that efficiently uses the information drawn from the data. An effective and efficient strategy suggests best use of the available business resources. It helps to find solutions for some of the biggest problems faced by the company. The process followed to formulate the strategy might be complex, but the final result is actionable and useful for management.
Analytics is not for one time or special event, yet it is a continuous process. The businesses should not take their attention off analytics and plan to adopt it as a regular business function. The business has to make collecting, cleaning and analyzing data a routine and a support role to functions that do not have the capability to do so. Most businesses look towards analytics when they face a problem and think the solution lies within their data. Once businesses start appreciating the potential analytics has to solve problems, they begin to use it to take all kinds of strategic and regular business decisions.
The resources and time required for an analytics project is dependent on a number of factors. The major factors being the scope and scale of the project, readiness and availability of required data, understanding of the analysis tools, skills and knowledge of the analytical team and most importantly, acceptance and approval from the management team to carry on the analytics project. The analytics team generally defines a project timeline dependent on the factors listed above. Intermediary findings and analysis difficulties might alter the goals and objectives of the project. This might require the team to re-work the time and resources required for completing the project. Deemsoft would be happy to provide you an estimate of the resources required to complete the analytics project and goals that you have in mind for your organization. Please contact us with details of your project.
Data is the most important resource for any analytics project hence the business should make sure that it captures its business and customer data in a structured manner. This will ensure that company has all the relevant data in the most usable form and can help the project move along quickly.
Delays in analytics projects generally take place when the data rendered to the analytical team is not usable in its current form. The data needs to be structured, cleaned and mined to make it usable. This step can take from hours to days to months depending upon the size and form of data.
Deemsoft would be happy to talk to you more about the state of your data and more specifically how 'ready' it is for analysis projects. Please contact us with details of your project.
For analytical needs, an organization can decide to use data analysis softwares like SAS or SPSS, seek help from custom consulting companies like Deemsoft or even build data analytic capabilities in-house. Today companies are even using a combination of the above.
Each of the above options comes with their own pros and cons. An organization has to find which option would suit their analytical needs best depending upon the nature of their business and existing resources. The costs associated with these options are rarely same for any two organizations. Deemsoft provides free consultation to evaluate the solutions needed.
There are two types of models, predictive and descriptive. Descriptive models are good to explain what has happened and what is happening. Predictive models explain what would be happening and why. These models are increasingly being utilized to solve problems across finance, marketing, human resource, operations and other business functions. At IQR, we have seen these models being used in financial services, casinos, airlines, retail, telecom, insurance, healthcare and even manufacturing industries.
Increased competition has expanded the scope, the need and the use of predictive modeling. Businesses need to be more proactive than before to build or sustain a competitive advantage. They need to get answers for tomorrow even before it arrives.
Predictive models are created using past and present data to foresee happenings in future. These models are being built to find answers to some of the most challenging businesses questions. It helps to manage portfolio returns, retain customers, undertake cross-selling activities, organize direct marketing campaigns, assess employee attrition and absenteeism, manage risks and formulate underwriting criteria, predict inactive customer accounts, cope with customer service requests, plan inventory and much more.
“Big data” is an all-inclusive term used to describe vast amounts of information. In contrast to traditional structured data which is typically stored in a relational database, big data varies in terms of volume, velocity, and variety. Big data is characteristically generated in large volumes – on the order of terabytes or exabytes of data (starts with 1 and has 18 zeros after it, or 1 million terabytes) per individual data set. Big data is also generated with high velocity – it is collected at frequent intervals – which makes it difficult to analyze (though analyzing it rapidly makes it more valuable). Or in simple words we can say “Big Data includes data sets whose size is beyond the ability of traditional software tools to capture, manage, and process the data in a reasonable time.”
This question cannot be easily answered absolutely. Based on the infrastructure on the market the lower threshold is at about 1 to 3 terabytes. But using Big Data technologies can be sensible for smaller databases as well, for example if complex mathematiccal or statistical analyses are run against a database. Netezza offers about 200 built in functions and computer languages like Revolution R or Phyton which can be used in such cases.
Contrary to what some people believe, intuition is as important as ever. When looking at massive, unprecedented datasets, you need someplace to start. In Too Big to Ignore, I argue that intuition is more important than ever precisely because there’s so much data now. We are entering an era in which more and more things can be tested. Big data has not replaced intuition — at least not yet; the latter merely complements the former. The relationship between the two is a continuum, not a binary.
Roughly 80% of the information generated today is of an unstructured variety. Small data is still very important — e.g., lists of customers, sales, employees and the like. Think Excel spreadsheets and database tables. However, tweets, blog posts, Facebook likes, YouTube videos, pictures and other forms of unstructured data have become too big to ignore. Again, big data here serves as a complement to — not a substitute for — small data. When used right, big data can reduce uncertainty, not eliminate it. We can know more about previously unknowable things. We can solve previously vexing problems. And finally, there’s the Holy Grail: Big data is helping organizations make better predictions and better business decisions.
Not exactly. Though there is a lot of buzz around the topic, big data has been around a long time. Think back to when you first heard of scientific researchers using supercomputers to analyze massive amounts of data. The difference now is that big data is accessible to regular BI users and is applicable to the enterprise. The reason it is gaining traction is because there are more public use cases about companies getting real value from big data (like Walmart analyzing real-time social media data for trends, then using that information to guide online ad purchases). Though big data adoption is limited right now, IDC determined that the big data technology and services market was worth $3.2B USD in 2010 and is going to skyrocket to $16.9B by 2015.
Big data is often boiled down to a few varieties including social data, machine data, and transactional data. Social media data is providing remarkable insights to companies on consumer behavior and sentiment that can be integrated with CRM data for analysis, with 230 million tweets posted on Twitter per day, 2.7 billion Likes and comments added to Facebook every day, and 60 hours of video uploaded to YouTube every minute (this is what we mean by velocity of data). Machine data consists of information generated from industrial equipment, real-time data from sensors that track parts and monitor machinery (often also called the Internet of Things), and even web logs that track user behavior online. Major retailers like Amazon.com, which posted $10B in sales in Q3 2011, and restaurants like US pizza chain Domino’s, which serves over 1 million customers per day, are generating petabytes of transactional big data. The thing to note is that big data can resemble traditional structured data or unstructured, high frequency information.
Eventually the big data hype will wear off, but studies show that big data adoption will continue to grow. With a projected $16.9B market by 2015 (Wikibon goes even further to say $50B by 2017), it is clear that big data is here to stay. However, the big data talent pool is lagging behind and will need to catch up to the pace of the market. McKinsey & Company estimated in May 2011 that by 2018, the US alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions. The emergence of big data analytics has permanently altered many businesses’ way of looking at data. Big data can take companies down a long road of staff, technology, and data storage augmentation, but the payoff – rapid insight into never-before-examined data – can be huge. As more use cases come to light over the coming years and technologies mature, big data will undoubtedly reach critical mass and will no longer be labeled a trend. Soon it will simply be another mechanism in the BI ecosystem.
From cloud companies like Amazon to healthcare companies to financial firms, it seems as if everyone is developing a strategy to use big data. For example, every mobile phone user has a monthly bill which catalogs every call and every text; processing the sheer volume of that data can be challenging. Software logs, remote sensing technologies, information-sensing mobile devices all pose a challenge in terms of the volumes of data created. The size of Big Data can be relative to the size of the enterprise. For some, it may be hundreds of gigabytes, for others, tens or hundreds of terabytes to cause consideration.
In my opinion, it is absolutely essential for organizations to embrace interactive data visualization tools. Blame or thank big data for that and these tools are amazing. They are helping employees make sense of the never-ending stream of data hitting them faster than ever. Our brains respond much better to visuals than rows on a spreadsheet. Companies like Amazon, Apple, Facebook, Google, Twitter, Netflix and many others understand the cardinal need to visualize data. And this goes way beyond Excel charts, graphs or even pivot tables. Companies like Tableau Software have allowed non-technical users to create very interactive and imaginative ways to visually represent information.
The data scientist is one of the hottest jobs in the world right now. In a recent report, McKinsey estimated that the U.S. will soon face a shortage of approximately 175,000 data scientists. Demand far exceeds supply, especially given the hype around big data. However, to become a data scientist one does not necessarily follow a linear path. There are many myths surrounding data scientists. True data scientists possess a wide variety of skills. Most come from backgrounds in statistics, data modeling, computer science and general business. Above all, however, they are a curious lot. They are never really satisfied. They enjoy looking at data and running experiments.
It’s an interesting point, and I discuss it in Chapter 4 of Too Big to Ignore. If we look at the relational databases that organizations have historically used to store and retrieve enterprise information, then you are absolutely right. However, new tools like MapReduce, Hadoop, NoSQL, NewSQL, Amazon Web Services (AWS) and others allow organizations to store much larger data sets. The old boss is not the same as the new boss.
A few relatively small organizations that have taken advantage of big data. Quantcast is one of them. There’s no shortage ofmyths around big data, and one of the most pernicious is that an organization needs thousands of employees and billions in revenue to take advantage of it. Simply not true. I don’t know in the near future if my electrician or my barber will embrace big data. However, we are living in an era of ubiquitous and democratized technology.
It’s already happening. Big data is affecting our lives in more ways than we can possibly fathom. The recent NSA Prism scandal shed light on the fact that governments are tracking what we’re doing. Companies like Amazon, Apple, Facebook, Google, Twitter and others would not be nearly as effective without big data. As you know, most people don’t work in data centers. Rather, it’s better for people to know about the companies whose services they use. Are those companies using big data? These days, the answer is probably yes. By extension, then, big data is affecting you whether you know it or not. In addition, as more and more companies embrace big data, there will be major disruption in the workforce.