Big Data or Big Mess?
Big Data, machine learning, AI – the hype words that will pull you into the magic circle of modern technology. Everyone wants it, everyone wants the possibilities, the growth, the kickstart that can be given to you by a good amount of data and a smartly designed software built around it.
But then why 87% of the data analytics, big data and AI projects fails? Aside from organizational and cooperation-relatedproblems (more info here), there is more behind this amazingly high failure rate.
Is your data good data?
You could ask is good data and bad data actually a thing? Well maybe not in itself – but data without context is worthless (read more in our earlier blogpost here). Building the wrong relations, the wrong connections, having the wrong approach and conclusions can make your data bad data.
Like when in The Big Bang Theory Leonard, Howard and Raj found some notebooks of a late physicist, Professor Abbot,filled with hundreds of pages of seemingly random numbers without any notes or explanation. Seems like something important, maybe it is his life’s work, maybe it contains something important, exciting isn’t it? Realizing that it is in fact his daily calorie intake diary such a bummer...
You can’t know if your data is valuable if you don’t have the context. You have to know what this data is about, otherwise you might as well end up building a model to predict some long dead professor’s calory intake….
Seemed like a good idea...
There are a lot of projects that fail because they seemed like a good idea, but in reality were completely worthless.
A company that had a relatively high employee turnover decided to up their „HR game”. They started researching the signs of people resigning, trying to predict which employee will leave the company soon. They tracked measurable data, like the number of years they worked for the company, their commuting distance, their salary, overtime, sick leaves and some hardly quantifiable data like their engagement to the company. They were looking for detectable connections, that will help them make their employees next move predictable.
While in some cases, collecting data and finding connections simply turns out to be not viable at all in production, in our case, that was not the problem. The issue was the wrong understanding of what data is and what data science and statistics can do for you.
The company actually was planning to use the knowledge they earned: they were planning on using it in their evaluation and hiring process. The idea was to evaluate new candidates on their likelihood of leaving, and only hire those applicants whose likelihood of leaving the company in the first 3 year is below a certain rate.
...but it simply does not work that way!
Now this sounds great, isn’t it? Just add in a couple of variables about your new candidates, and a software will spit out the possibility of them becoming a loyal, long-term employee.
Is it doable? From a programming point of view, with quantifiable variables, it is no magic to create a program like that. But will it work? The answer is a straight and obvious no.
Besides the fact that collecting data like this is questionable at best, with regards to the privacy of employees, it does not make sense from a business perspective. If we really give some deep thoughts of why anyone would want to quit their job, the factors are multiple, and are neither predictable, nor measurable for start. How someone will respond to workplace dynamics, or personal issues they might encounter, are not quantifiable and cannot be predicted even with the utmost caution.
And beware of the self-fulfilling prophecy! A simple notification to a manager that a certain employee is most likely to leave the company in the near future based on these parameters will itself affect the workplace dynamics – and might bring upon what one wanted to prevent.
The idea completely ignores the single most important thing of why some people stay at a workplace even when underpaid or working extreme hours, and why others with a generous salary or personalized benefits will still leave. And that is the human factor (read more here).
Causes or consequences – did you make the right conclusions?
But let’s assume, that you are able to quantify and take into consideration every human factor – that is probably a mission impossible, but sticking to the above example let’s just assume it for a second, and try and dig deeper of why the idea is inherently wrong.
The software would calculate a vast number of variables, that are not decidedly causes, but most likely consequences of an employee leaving the company. Having less overtime, going on a sick leave, besides of other factors can actually be consequences of an already made decision: our employee already decided to leave, and preparing for the change, is looking for another job and so on.
The algorithm does not predict, it just tells us what is happening. We are too late to do anything about this employee leaving. That is how wrong connections, or even only seemingly existing relations can make your data bad data – and lead to wrong conclusions and wrong business strategy.
Machine learning and its limits
Last but not least, even if you get every single part of your use case right, although we mentioned it in our previous blog post, but it can never be emphasized enough: machine learning can be a goal, but it is very rare, that it can be your first step of data analytics. The reason for that is simple: you need huge, and I mean really huge amount of data for your software to be able to actually find working relations, rule out the insignificant factors, be aware of unique cases and extremes and create predictions that will actually give you useful and trusted information.
Having a machine learn relations, connections, create predictions is not so much different than having economists and researchers conducting statistical analysis. You need an unbiased representative sample to get to a conclusion, and to avoid biased sample you need to know your population. In our example case even if we ignore any other issues, simply reviewing 500 employees actions are just not enough to be able to get clean data that can be used as a base of any sort of prediction.
For a kickstart, most companies don’t need machine learning right away: synchronizing, centralizing your data, and automating just some of your daily tasks can already be a huge step towards the right direction.
This is how sometimes even small but fast companies on the market are able to steal market share from well-known giants: they already know that there is no need to reinvent wheel to gain momentum.
You can share this blog post
More blog posts
I grew up in a small village in the Dutch province of Zeeland. During my childhood I spent a lot of my free time outdoors playing with my friends, climbing trees, building tree houses, playing football and swimming in the sea. My father was (and is) a watchmaker and as a small boy I was always intrigued seeing him repair watches and clocks. Looking at all those small parts ultimately forming one single system that tells time. I guess that during those days my passion for technology started, although at that time I excelled taking clocks apart, not repairing them…
Later I got intrigued by the possibilities of computers as we got at home in the late 1980’s: a Commodore Amiga 500. Initially we used it to play games, but soon after that I tried coding new games myself.
At relatively young age I already knew I wanted to be an engineer and I ultimately decided to study Mechanical Engineering at the Delft University of Technology. During the Master phase I specialized in the field of human machine interaction. This is all about making sure you provide people with the right information at the right time to make sure they will be able to make the best possible decisions. This is the red line in my career so far: utilizing (software) technology to help people make timely and well-founded decisions.
Gaëtan and I started Cuurios in 2018 because we’ve noticed that many industrial big data analytics projects face difficulties to make it into daily reality. In our vision you need to add business value in the early stage of digitization and transform data into prioritized (and assigned) actions for it to be taken seriously. The underlying technology to transform the data does not need to be complex (i.e. AI or learning algorithms), the trick is to select the right mechanisms for your particular use case! Most importantly, make sure you well manage the actions resulting from these data analytics: your average dashboard does not do any of this!
In my current role I get a lot of energy bridging the gap between the customer’s daily reality, new technology and software development. I’m enthusiastic, curious and people oriented. We typically like to listen very carefully to our clients ideas and requirements and ask the right questions in order for us to be able to deliver software solutions with a WOW factor: always go one step further!
Both professionally and privately I like complex puzzles, varying from large data sets, challenging customer use cases all the way to large LEGO Technic sets. I am confident that this mindset helps me to keep pushing boundaries in search for that missing piece!
My very first program was written in Turbo Pascal. It was a very simple little application, where you could give your birth date, and the app was telling you your astrological sign and all the main characteristics. That was over 20 years ago, when computer science classes for the students in my country meant learning how to turn on and off a computer and change wallpapers.
I was always intrigued by the possibilities of this technology, and my primary school teacher was really happy that someone showed interest, so we started learning the real things, hardware and software as well. After primary school, I never again had the same opportunities or means to keep on learning.
Fast forward almost two decades, in my late twenties, still trying to find my passion, becoming tired and burnt out from marketing and copywriting that I was doing for over a decade, and not finding joy in being in law school either – mainly because I saw my future as lawyer, and suffice to say that it wasn’t as „romantic” as I imagined it.
From customer relations officer to HR assistant, from journalist to head of marketing I tried myself in many 9 to 5 jobs, until one day I decided, I wanted to do something completely different. At age 27 I decided it was time for a real change – and I started learning again. In just 3 years I became a web designer, and after rediscovering my love for coding, kept on learning to become a full stack developer.
Today, for the very first time in my life, I can finally combine my need for creativity and challenge, I can connect the missing dots. Before coding, when I had an idea, I always thought: „how cool it would be”. But today, I know how to make it happen, I have control over what happens and how, and I can bring all those ideas to life.
Cuurios is my very first workplace as a full stack developer, but I believe I could not have been any luckier. I am not only a „coder” who translates an idea into code – my ideas are appreciated, and I can work in my own rhythm, try and solve problems alone, but always get support if needed. This is an environment where I believe an entry level programmer can really thrive, learn, and grow as fast as possible, facing challenges, a huge variety of tasks, being responsible but still getting all the support necessary.
Feels like it was yesterday when I created my very first hangman in python as a student. Back then I did not even dream of creating enterprise applications in just a couple of months after school, but with all the tools and support provided at Cuurios, I never felt uncomfortable facing that challenge.
Being a full stack developer helped me grow as a person as well. Learning how to make mistakes a thousand times and keep on trying until you get it right, knowing when you reached your limit and being able to ask for help without feeling uncomfortable about it – it is all part of a developers life as much as life itself.
Being a teenager in the 90s, with the internet, mobile phones and all sort of exciting new technologies coming up, it was hard not being enthusiastic about Computer Sciences!
After finishing my degree (Computer Science and Networks), I took a detour, working as a consultant, busy with the soft side of IT (writing design documents, requirements, sketching business and IT processes, these sort of things). After a while I realized that I wanted to go back to the core of technology and changed focus towards more technical and code related work.
The real joy in coding for me is creating something out of your brain, and see it come to life - something most programmers would recognize. In addition to the creative thinking and joy of creation, computers follow straightforward logical patterns, what you code is what you get. Refreshing compared to working with humans (Not that I have misanthropic tendencies, but hey, everyone will agree, people can be difficult ;) ).
I am result-driven, inquisitive, and professional, always curious, and ready to learn new technologies as well as dive deeper in what I already know. I love challenges and new discoveries, which makes my field as a professional really broad. I do not like to fit in a box and am eager to always keep learning! So, java bytecode, deep learning, graph databases, SQL, high level architecture ontologies, web technologies, security, networking, bring it on! :) .
I am one of the owners, Cuurios is my baby. As a child of the first internet bubble, of the Amazon and the Google era, being an entrepreneur has always been one of my deepest dreams. You can see it as an extension of the coder’s creative drive. Thinking out, designing, and developing a visionary product and bring it to market, full control, and total responsibility. Daunting, yet exhilarating!
As owner and lead developer Cuurios gives me the possibility to completely give direction to the work, according to my own ideas, and to where I want to go. It also gives me the freedom of setting my own agenda and being able to work as I wish – probably every thriving programmer dreams about that at some point.
In addition to that, I believe Cuurios does not only hold value for the owners or the employees. We aspire to deliver real values to our customers, always helping them to get the most of their experience with us – not just with our products, but also every level of our communication, every step of us working together. We are very proactive, and we tell it how it is, we can expect no bs from us!
After a first month as part of the YES!Delft investor readiness program, it is time to share some thoughts on the process, what we’ve learned so far and how it impacted us.
DISCLAIMER: This light-hearted description is only very loosely based on reality and should definitely not be taken at face value :-)
Like most, we started this journey as enthusiastic entrepreneurs, completely sold onour ideas, focused on how great it is and all the cool stuff we have in petto, the myriad of functionality it will provide and how it will revolutionize the market. We were pretty confident, sure of our game.
Then we had 1 minute to pitch.
- Everybody understood we had something great, nobody could quite get what it was all about.
Investors are from Mars
That would be the first thing we learned. Pitching to an investor (or a prospective client) is not about YOU, it’s about THEM.
Investors speak the language of TAM, SAM, SEM, ROI, Churn, Business Models, Valuation, EBIDTA, Cash Flows... Learning the investor’s lingo was a damning task, like learning a second language full of acronyms and numbers. Frustrating at the beginning, very enlightening in the end. It really helped us get our story straight and forced us to do the math. Cause you’ve got to do the math. You can’t stay forever in the fluffy stage.
The first hurdle passed, it was not the end of it. We had some numbers, great, but what was it again that we sold? And, most importantly, sell to whom?
- Ok, you’ve got 30 seconds.
Hmm, yes, you know, we make software that does stuff with data, and it’s super cool.
At this point we knew we had to do something about that elevator pitch.
Here comes the WOW method in place. A simple method to get your ideas straight and write an elevator pitch with a true WOW factor (yes, it is in the name).
Suddenly, it all made sense. This is the story we want to tell, these are our customers, and this is what their pain is.
Elevator pitch much improved, still some way to go, but at least most people now understand what we are trying to achieve.
No free lunch
Now that we’ve got a good enough elevator pitch, and some numbers, time for what we’ve all come for: Show me the money!
Again, a good learning experience: there is nothing like a free lunch in this world. Investors are not charities, they’re in for the big bucks! And how to convince them that you are the next Google when you are a fledging start-up?
- Get some customers on board, build traction, show you’ve got it!
But wait a sec, to build any sort of real traction we first need to finish up our product! And wasn’t raising money supposed to help get us there? It seems a bit like a chicken and egg story, isn’t it?
Welcome into the real-world people!
There are some options fortunately, government grants and loans, to help you get started and get you through this first stage in your development, without of course forgetting the most common investment of all, some good old-fashioned hard work!
So far, a very valuable program, it has helped us to sharpen our message, build up our case, and get us ready for the next stage!
Part of my final year of international business at The Hague University of Applied sciences is to do an internship. During this engrossing year I was lucky enough to do my internship here at Cuurios. I have followed almost every business related subject during the last 3 years at my university, ranging from human resources to finance to operations management. I always missed a really tech related subject which is quite important in the modern days. I’m really happy I can fulfil that missing piece here at Cuurios.
I applied to do my internship at Cuurios because it is a relatively young, still small sized but growing company. I like this size during this internship because even though I am an intern at Cuurios I am also part of the team and I am really responsible for the tasks assigned to me. Although the pandemic has made us work from home most of the time, I am still able to get proper guidance and plenty of sparring opportunities.
Whilst I’m doing my internship here, Cuurios participates in the Investor readiness program by YES!Delft. It is a really interesting program where I get the opportunity to translate gained knowledge from my study into reality. I help Cuurios to prepare for an acceleration of their business, this ranges from a good company value proposition and pitch to a financial planning and business plan. The tasks are diverse and I’m able to work however I feel best suited while still being coached and questioned which I find is working perfectly.