Life cycle of DS, ML and Data Visualization

Photo by Scott Graham on Unsplash

You have a business problem that you need to be solve.Depending on the nature of this problem,explore a mariad of solutions either a traditional solution or a more complex solution . Follow the steps below.

Framing the problem

Depending on the size of the organization, this part will be done by Business Analyst and Product manager as follows:
1.Defination of the business problem: The problem to be solved should be clear and the business objectives made as clear possible.
Finding Solutions:
1. Are there any current available solutions.What are its pros and cons.

2. Are coming up with new solution?
If so, What is the solution?
How will be implemented(Are there tools and resource that can be resused?
Do we have the capacity and expertise to implement this solution) and what are the success measure?.

Are this success measure alligned to the objectives meantioned above?

Are there any assumptions? and an they be verified?

Getting the Data

Considering the size and the structure of the company this part of the project will be handled by a data engineer.

Note: automate as much as possible so you can easily get fresh data.

1. From the curated solution, the data aengineer with the help of other stakeholders should be able to list the data needs, type and the size.
2. Find and document where you can get that data
3. Consider possible storage type and capacity.
4. Consider legal obligations (and acquire/adhere to them) needed to obtain and use the said data.
5. Set up the environment that will be used to collect the data.This is a tech workspace i.e servers, databases , IDEs e.t.c.
6. Get the data, Ideally this set should be automated.
7. Create a datawarehouse, to separate the transactional data A. from analytical data.This will enable other stakeholders to manipulate the data without tampering with the original data.
8. Make sure your data follows all the legal obligations identified above, e.g ensuring personal sensitive information is deleted or protected

Data Preprocessing and feature engineering, Model training,Model Evaluation

This is where the data scientist comes in, in the folowing steps

A. Data Preparation and feature engineering :This refers to algorithmic and creative process of cleaning , selecting and transforming raw data into features that can be used in training models.

Why Data Prep and feature engineering?

1. ML algorithms take numbers and sometimes ave conditions and requirements.
2. Data prep help uncover underlying pattern/relationships which helps in training a better model.
3. Good features give more flexibility in hyperparameter tuning and model selection.
4. Good feature enable training of a ssimpler model, which is explainable.easy to debug, effecient in terms of time and processing power.

Steps to be taken.

1. Discovering insights from the data by use of visualization, Aka Explanatoty Data Analysis(EDA)
2. Data cleaning.From the EDA above you will be able to find out more about the shortcomings of the data and what need to be corrected in order to make the data high quality and complete inorder to gain maximum,accurate insights and well performing models.
3. Feature engineering:The process of extracting features from raw data.
4. Select and train a model based on your needs.
5. Model evaluation and fine-tuning.

Model Deployment,Model Serving and Model Monitoring

This part is done by software engineer or Applied scientist( A person with both Data science and software engineering expertise)

First step is deployment into production.This can be done using APIS.The API(s) is then intergrated into your system .

Write monitoring code to check your system’s live performance at regular intervals and trigger alerts when it drops. This is important to catch not only sudden breakage, but also performance degradation.

This is quite common because models tend to “rot” as data evolves over time, unless the models are regularly trained on fresh data.

Happy Learning!