In 1989, I took my first decision sciences course and started coding in SAS at the age of 20. I greatly enjoyed pulling discoveries buried within mounds of data, although and even small datasets had many discoveries back then.
At the root of every model I’ve built, even the simplest, was a solid understanding and foundational rigor of statistical theory. When computing simple statistics or developing descriptive models, I thought through the math behind the model and how this would impact the formation, application, and interpretation.
It was about 30 years ago when I started my decision sciences journey, and I’m still applying techniques and building models to empirically solve problems, answer questions, overcome challenges that improve, reduce error, or otherwise benefit a situation.
Over the past three decades, I’ve noticed trends and shifts, an evolution of sorts, in the foundational underpinnings of development and application within this interesting profession. I’ve come to the following conclusions that illustrate the evolution of the data science function over the past few decades:
- Data science is more applied than ever. What can be built and fit over a real-life scenario has The dreadful requirement of mattering. Modeling for modeling sake is no longer thing, and best fit diagnostics are less important than the best fit for the situation. If a model goes unused, it serves no purpose. We no longer can tolerate or afford the luxury of building models purely for R&D purposes without consideration of utilization.
- The skill of computer use seems to have taken over the knowledge of applied statistics. Understanding the interior workings of the black box has become less important unless you are the creator of the black box. Fewer data scientists with truly deep knowledge of statistical methods are kept in the lab creating the black boxes that hopefully get integrated within tools. This is somewhat frustrating for long-time data professionals with rigorous statistical background and understanding, but this path may be necessary to truly scale modeling efforts with the volume of data, business questions, and complexities we now must answer.
- Data scientists are not weird anymore. We’re seen as strategic inputs to the decision-making process, and our craft is becoming much more understood. This trend is evident by C-level positions at large companies, vertical alignment, and paths for data scientists, and inclusion at the highest levels, as well as the many academic programs and emphasis now available globally. This appreciation and positioning can sometimes make the field appealing for what seasoned data scientists might call the “wrong reasons” such as corporate fame and value. My colleagues and probably myself would argue that we really want professionals in the field with a thirst for the truth – the science should be about empirically answering questions, and powered by truth-seekers at their heart.
- Data Science is becoming more widely recognized as both art and science. Understanding the importance of the human-machine integration and complementary decision-making skills from each appears to have made its way more squarely into our field of understanding.
Today’s excellent data scientists, especially on the applied side, tend to get things done more quickly, and with improved computer science skills. The need for understanding the inner-workings of the black box seem to balance with the need to know how to run programs, operate tools, and write code. I’m excited to see where the next several years will take us, and look forward to seeing further evolution of the data scientist.
