ML Engineering
Developing and releasing quality software is hard. If there is no automated process, it is downright expensive and increases risk to business due to unseen failures.
For a software engineer, being able to write, test, and quickly deploy and perform tests (integration, performance, vulnerability) are very important to have a successful non-eventful release. An Organization must develop an engineering cultural practices that recognizes continuous – building, testing, and deployment practices. To achieve such culture, engineers must adhere to writing test codes that covers most test scenarios as well as design for automation.
Data Science Software development is even harder if there is no engineering culture of continuous – build, test, and deploy practices. Following flow shows a typical ML process where there is Data Pipeline (ETL) and Training Model is part of process. A data scientist is dependent on Data Engineer to implement ETL and operationalize it before a model can be operationalized. Similarly, once the training is complete, the data scientist is dependent of ML Engineer to operationalize his model. Even before operationalizing it, the Data Scientist is dependent on Product team to integrate with the current or future service offering.
Those intricate interaction among various team members if they are in the same team otherwise, the interaction will be with various teams which bring much more complexity not only in software engineering but also team and inter personal interaction.
This where DevOps plays an integral role in bringing engineering culture of continuous build, test, and deploy practices. Automating model training and deployment will hugely save time to market for operationalizing model. Thus, allowing Data Scientists to focus on their core competencies rather than coordinating efforts to deploy their model into production.
