Talend:
We are living in an era where data has become a critical competitive factor. In the words of Talend CEO Mike Tuchen: “Data is the new battleground. For companies, the situation is clear – their future depends on how quickly and efficiently they can turn data into accurate insights.”
True to the motto “Insight without action is useless” I should add that it is just as important, of course, to follow insights with appropriate action. This action will, in turn, have a measurable effect, and subsequently, lead to new insights. The resulting cycle is becoming increasingly significant for organizations seeking a competitive advantage.
Before data can be analyzed it must be collected and stored systematically. Whereas traditional business intelligence has only very selectively stored data from operative systems in a central data warehouse, things have changed enormously in the era of big data. Alongside a plethora of related open source projects, Apache’s Hadoop project, which celebrated its tenth anniversary in 2016, has changed the world. It has become possible to store and process large quantities of raw data inexpensively (e.g. from sensors, log files or cameras). When combined with advanced analytics techniques, this data provides the primary source for new, systematically gained insights.
Self-service tools such as Tableau, Qlik and TIBCO Spotfire have democratized the analysis of data. Simple methods now allow more people than ever, particularly in the user departments, to visualize data in order to gain insights. Through the use of dashboards, either by admin staff or by other departments, organizations as a whole are able to glean valuable insights from their data more efficiently. These systems generally form part of an organization’s business intelligence strategy. They use data stored in a data warehouse – the structured storage of relevant information from the operative systems for this specific purpose.
The area of data science has simultaneously evolved at a spectacular rate, with the Harvard Business Review naming the occupation of data scientist as “the sexiest job of the 21st century”. The term generally refers to the extraction of knowledge from data. Starting off from a specific issue, data is consulted and interpreted in order to build models using statistical techniques which can range from comparatively linear models to artificial neural networks – currently the subject of much discussion, especially in conjunction with deep learning.
It is worth noting that the cycle of insight to action is largely closed automatically in the data science environment, whereas it tends to take place via humans in the area of business intelligence.
Data scientists are able to build models that can then be deployed in live operating situations. Such models have multiple uses: for example, to predict malfunctions and plan maintenance work accordingly (predictive maintenance), or to identify cart abandoners in online shops and take appropriate steps to convince them to buy. As the world changes, these models are continually adapted (drift of concept / change of concept), or even learn continuously. To this effect the cycle is closed.
Conversely, in the field of business intelligence knowledge tends to be acquired by repeatedly checking reports or manually monitoring dashboards with specific issues in mind. The resulting decisions based on insights help to shape the world, which in turn is reflected in the data. However, this data largely lends itself to human decisions only.
Experts like Dr. Carsten Bange, CEO of the Business Application Research Center (BARC), sees automation in this area as imminent: More and more decisions can be made entirely automatically. We are moving from decision support via automatic decision-making checked by humans and autonomous decision-making by machines through to self-learning systems, which constantly improve the quality of their decisions as they learn from experience.
Of course, we still have quite a long way to go. However, in the foreseeable future, data and algorithms will play a critical role in the market competition. The winners will be those organizations that are able to gain insights from data quickly and effectively, and to derive appropriate courses of action using a high level of automation, whose consequences, in turn, can also be measured.
Quoting Mike Tuchen (“Data is the new battleground…”) companies everywhere are equipping their troops. The better the tools, the more knowledgeable the staff; the more central the subject is anchored in the organization the more successful it will be. In some sectors the battle has already begun, in the others, it has yet to begin.