The data science and AI market may be ready for recalibration

The data scientist should be “the sexiest job of the 21ste century “. The question of whether the famous Harvard Business Review aphorism of 2012 is holding water. However, data about data scientists, and the related roles of data engineering and data analysts, are beginning to sound alarm bell.

The personal part of the Harvard Business Review aphorism is whether you really enjoy finding and cleaning up data, building and debugging data pipelines and integration code, and building and improving machine learning models. It is on this list of tasks, in this order, that data scientists spend most of their time.

Some people are actually attracted to data -based careers through job descriptions, while growing demand and salaries attract others. Although the dark parts of the job description itself have not been heard of, the growth and salary side has been less challenged. However, the situation may change: demand for data scientist positions is still high, but not immune to market turmoil.

Mixed signal

At the beginning of 2022, the first signs that there may be a change appeared. As the IEEE Spectrum analysis of data published by online recruitment firm Dice showed in 2021, salaries in the field of artificial intelligence and machine learning have fallen even as, on average, wages in the US technology sector have risen. of about 7%. However, as the IEEE Spectrum notes, competition for machine learning, natural language processing, and AI experts has softened, with average salaries dropping 2.1%, 7.8%, and 8.9%. , respectively.

This is the first time this has happened in recent years, as the average salary in the United States for software engineers with machine learning expertise, for example, jumped 22% in 2019 compared to 2018, then rose even further by 3.1% in 2020. At the same time, the demand for data scientist roles is showing no signs of slowing down, on the contrary.

Developer recruitment platforms are reporting seeing increasing demand for IT skills in data science. The latest report from developer screening and interview platform DevSkiller recorded a 295% increase in the number of recruiters of data science -related tasks scheduled for candidates in the interview process in 2021.

The CodinGame and CoderPad survey in 2022 also defined “Tech Hiring” data science as a profession which demands far more than supply, along with DevOps and machine learning specialists. Therefore, employers will need to re-evaluate the salaries and benefits they offer their employees if they want to remain competitive.

Delay flow

In addition, 2021 is marked by the so -called “Great Resignation” phenomenon, a time when everyone is rethinking everything, including their careers. In theory, the fact that part of the workforce will redefine its trajectory and its goals and/or cessation should increase demand and wages – studying why data scientists quit and what data scientists can do. employers to handle them began to spread.

Then came the layoffs, including data scientists, data engineers and data analysts. As LinkedIn reviews the latest phase of layoffs, the tech industry’s turbulent year has been marked by daily announcements of layoffs, hiring freezes and job opening cancellations.

About 17,000 workers at more than 70 tech start-ups around the world lost their jobs in May, a 350% jump from April. This is the highest number of job losses in the sector since May 2020, at the height of the pandemic. Additionally, tech giants like Netflix and PayPal are also cutting jobs, while Uber, Lyft, Snap and Meta are slowing down hiring. According to data shared by tech layoff tracking site, layoffs affect between 7% and 33% of the workforce of the companies being tracked. If we look at the data specific to each company, it can be seen that these are also data -driven positions.

Analysis of layoff data from fintech Klarna and insurance startup PolicyGenius, for example, shows that the roles of data scientist, engineer and data analyst are affected at the junior and senior levels. At the same company, these roles cost approximately 4% of layoffs.

Measure the consequences of automation

What should we think of these conflicting signals? Demand for data science -related jobs seems to remain strong, but salaries are declining and these jobs are also not immune to layoffs. Each of these signals has its own context and implications.

As Michelle Marian, Director of Marketing for Dice, told IEEE Spectrum, a variety of factors are likely to be contributing to the decline in machine learning and AI salaries, one of which is the more technologists learn and apply these skills. “As the number of talent increases over time, employers may have to pay at least slightly less, because skills are easier to find. We’ve seen this happen with a series of certifications and other highly specialized technology skills, ”he said.

This seems like a reasonable conclusion. However, for data science and machine learning, something else may be at play. Data scientists and machine learning experts are not only competing with each other, but also increasingly competing in automation. As Hong Kong -based quantitative portfolio manager Peter Yuen points out, quantitative analysts have seen it all before.

After learning that top AI researchers earned salaries in the $ 1 million range, Peter Yuen wrote that it “should be interpreted more accurately as a continuation of a long trend of“ high-tech ” coolie to code themselves to lose their jobs in the middle of a global world. ” too much skilled labor. “

“Judging by the experience of three generations of quants in automating financial markets,” Peter Yuen wrote, “the automation of AI-based practitioners in many industries will probably not be just for ten years. ” After that, he added, “a small, select group of AI practitioners will reach executive or owner status, while the rest will remain in mid-paying jobs assigned to monitoring and caring for their creations.

We may be in the early stages of this cycle, proving that developments such as AutoML and libraries of machine learning models are ready to use. If history is anything to go by, what Peter Yuen describes is likely to happen as well, which will inevitably lead to questions about how displaced workers can “climb the heap”.

The explosion of the AI ​​bubble

However, one can assume that data scientists don’t have to worry too much about it in the immediate future. After all, another frequently mentioned fact about data science projects is that almost 80% of them always fail for many reasons.

The Zillow case illustrates this particular failure of data science. Zillow’s business relies heavily on the data science team to develop accurate predictive models for its home buying service. It turns out that the models are not that accurate. As a result, the company’s stock fell more than 30% in five days, the CEO blamed the data science team, and 25% of staff were laid off.

Whether or not the data science team is to blame for Zillow is up for debate. As for the recent layoffs, they are likely to be seen as part of a larger economic change rather than a failure of data science teams with each other. As Data Science Central community editor Kurt Cagle wrote, it’s talking about an impending AI winter, reminiscent of the 1970s period when funding for AI projects completely dried up.

Kurt Cagle believes that although an AI winter is fuzzy, one can expect an AI fall with excessive cooling of venture capital in this space. The winter of AI in the 1970s was largely due to technology not getting the job done and not having enough digitized data for mine.

Today, the available computing power is greater and the amount of data is increasing. According to Kurt Cagle, the problem may be that we are approaching the limitations of the neural network architectures currently in use.

Like many others, Kurt Cagle pointed out the shortcomings of the system of thinking that consist in saying “deep learning can do everything”. This criticism seems valid, and the integration of currently neglected techniques could advance the field. However, let us not forget that the technological aspect is not the only thing that matters here.

Recent history may enlighten us: what can we learn from the history of software development and the Internet? In some ways, where we are today is reminiscent of the days of the dotcom bubble: increased availability of capital, excessive speculation, unrealistic expectations and massive costs. Now we can head towards the burst of the AI ​​bubble. That doesn’t mean that data science roles will lose their appeal overnight or that what they do is worthless. After all, software engineers are still in demand for all the advances and automation that software engineering has seen over the past few decades. But that likely means re -calibration is needed and expectations should be handled accordingly.


Leave a Comment