Mada za sehemu hiiCleaning and organising dataMada 4
- Performing basic data cleaning
- Performing basic data transformations
- Performing basic data normalisation and standardisation
- Using specialised libraries for data cleaning
When working with real-world data, different features often have very different ranges. For example, a dataset about small businesses in Tanzania might have "monthly profit" measured in tens of thousands of shillings and "years in operation" measured in single digits. Many machine learning algorithms work much better when all features are on a similar scale. This process of adjusting data to a common range is called scaling, and the two most common approaches are normalization and standardization.
Normalization scales data to a specific range, usually between 0 and 1. The most common method is min-max scaling, which uses the minimum and maximum values of the data.
The formula for min-max normalization is:
xnormalized=xmax−xminx−xmin
Where:
- x is the original value
- xmin is the smallest value in the feature
- xmax is the largest value in the feature
After this transformation, the smallest value becomes 0 and the largest becomes 1. All other values fall between 0 and 1.
Worked Example
Suppose a Tanzanian farmer recorded tomato sales over five days:
| Day | Tomatoes Sold |
|---|---|
| 1 | 20 |
| 2 | 35 |
| 3 | 15 |
| 4 | 50 |
| 5 | 40 |
Using min-max normalization:
- xmin=15
- xmax=50
For Day 1 (20 tomatoes): xnormalized=50−1520−15=355=0.14
For Day 3 (15 tomatoes): xnormalized=50−1515−15=0
For Day 4 (50 tomatoes): xnormalized=50−1550−15=1
The normalized values are: 0.14, 0.57, 0, 1, 0.71
Standardization transforms data so it has a mean of 0 and a standard deviation of 1. This is also called z-score normalization. It measures how many standard deviations a value is from the mean.
The formula is:
z=σx−μ
Where:
- x is the original value
- μ (mu) is the mean (average) of all values
- σ (sigma) is the standard deviation
Worked Example
Using the same tomato sales data:
Values: 20, 35, 15, 50, 40
First, calculate the mean: μ=520+35+15+50+40=5160=32
Next, calculate the standard deviation. First find the differences from the mean:
- 20 - 32 = -12
- 35 - 32 = 3
- 15 - 32 = -17
- 50 - 32 = 18
- 40 - 32 = 8
Square each difference: 144, 9, 289, 324, 64 Sum of squares = 830
Variance = 830÷5=166 Standard deviation σ=166≈12.88
Now standardize Day 1 (20 tomatoes): z=12.8820−32=12.88−12≈−0.93
The z-score of -0.93 means 20 is about 0.93 standard deviations below the mean.
The sklearn library (Scikit-learn) in Python provides simple functions to perform both normalization and standardization.
Normalization with MinMaxScaler
from sklearn.preprocessing import MinMaxScaler
import numpy as np
# Sample data: monthly profits in TSH for a small shop
profits = np.array([150000, 280000, 95000, 320000, 210000]).reshape(-1, 1)
scaler = MinMaxScaler()
normalized_profits = scaler.fit_transform(profits)
print(normalized_profits)
Standardization with StandardScaler
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
standardized_profits = scaler.fit_transform(profits)
print(standardized_profits)
Both scalers learn the parameters from the data using fit_transform(), which first learns the parameters (min, max, mean, std) and then transforms the data.
Scaling improves model performance in several ways:
- Faster convergence: Algorithms like gradient descent reach solutions more quickly when features are on similar scales
- Fairer feature comparison: No single feature dominates others simply because it has larger numbers
- Distance-based algorithms: Methods like k-nearest neighbors and support vector machines give equal importance to all features
- Regularization: Techniques like L1 and L2 regularization work properly only when features are scaled
-
Normalization (min-max) works well when:
- You know the exact minimum and maximum values
- The data has a bounded range
- You need values between 0 and 1 (for neural networks, for example)
-
Standardization (z-score) works well when:
- The data has outliers (because it uses mean and standard deviation)
- The algorithm assumes normally distributed data
- You don't know the exact range of values
In Tanzania, a market vendor tracking daily sales of different items (kilograms of rice, dozens of eggs, liters of cooking oil) would find each item has vastly different value ranges. A mobile phone seller in Arusha might have prices ranging from 50,000 TSH for basic phones to over 1,500,000 TSH for smartphones. By applying normalization or standardization before using a sales prediction model, the vendor ensures that the cheap phone prices do not get ignored simply because their numbers are smaller than smartphone prices. This scaling helps the model treat all products fairly, leading to more accurate sales forecasts and better inventory planning.
Jinsi utakavyotathminiwaTathmini ya mchakato · Tathmini ya matokeo · Tathmini ya maarifa
Tathmini ya mchakato
- Scale numerical data effectively.
- Apply normalization or standardization based on context.
Tathmini ya matokeo
A preprocessed dataset with numerical features normalized or standardized.
Tathmini ya maarifa
Detailed knowledge of:
Method used: The student should explain how to:
- normalize data using Python
- standardize data using Python.
Principles: The student should explain the principles of
- Reducing feature bias in data analysis.
Theories: The student should explain:
- Normalization versus standardization (e.g., min-max scaling vs. z-score).
Circumstantial knowledge:
Detailed knowledge about:
- Deciding when scaling is necessary (e.g., for algorithms sensitive to data ranges).
Swali
What is the output range of min-max normalization?
Ingia ili kuwasilisha jibu lako na lihesabiwe katika umahiri wako.
Ingia ili kufanya mazoeziMwalimu
Umekwama? Niulize chochote kuhusu mada hii.
Ingia ili kumuuliza Mwalimu wa AI wa Sonza kuhusu swali hili.
Ingia ili kuulizaMajadiliano
Hakuna maswali bado