Mada za sehemu hiiCleaning and organising dataMada 4
Data cleaning is the process of fixing errors, inconsistencies, and inaccuracies in datasets so that they become reliable for analysis. When data is collected from different sources — such as surveys, transactions, or sensors — it often contains missing entries, duplicate records, or wrong data types. Cleaning this data ensures that programs can process it correctly and produce accurate results.
Raw data is rarely ready for use immediately after collection. Common problems include:
- Missing values: Some cells are empty or have "NaN" (Not a Number)
- Duplicate rows: The same record appears more than once
- Wrong data types: A column that should contain numbers contains text
- Inconsistent formatting: Dates written in different styles, or text with extra spaces
Data cleaning fixes these issues so that analysis and visualization can proceed smoothly.
Missing values occur when no data is recorded for a particular field. In Python's pandas library, missing values appear as NaN (Not a Number) or None.
How to Identify Missing Values
import pandas as pd
# Load a CSV file
df = pd.read_csv("students.csv")
# Find missing values in each column
print(df.isnull().sum())
Techniques for Handling Missing Values
-
Drop rows with missing values: Use this when the missing data is few.
df_clean = df.dropna() -
Fill missing values with a specific value: Useful when you want to keep the data.
# Fill missing ages with 0 df["age"] = df["age"].fillna(0) # Fill missing names with "Unknown" df["name"] = df["name"].fillna("Unknown") -
Fill with mean or median: Common for numerical data.
df["score"] = df["score"].fillna(df["score"].mean())
Duplicate records distort analysis. Pandas makes it easy to find and remove them.
# Check for duplicate rows
print(df.duplicated().sum())
# Remove duplicate rows
df_clean = df.drop_duplicates()
Data must be in the correct type for calculations. For example, a column of prices should be numeric, not text.
# Convert a column to numeric (coercing errors to NaN)
df["price"] = pd.to_numeric(df["price"], errors="coerce")
# Convert a column to integer
df["quantity"] = df["quantity"].astype(int)
# Convert a column to string
df["item_code"] = df["item_code"].astype(str)
Imagine a small shop in Dar es Salaam has a CSV file called sales.csv with the following messy data:
| item | quantity | price (TZS) | date |
|---|---|---|---|
| Rice | 5 | 15000 | 2024-01-01 |
| Rice | 5 | 15000 | 2024-01-01 |
| Sugar | 2 | 2024-01-02 | |
| Sugar | 2 | 8000 | 2024-01-02 |
| Tea | 2000 | 2024-01-03 |
This data has three problems: a duplicate row, a missing price, and a missing quantity.
Step 1: Load the Data
import pandas as pd
df = pd.read_csv("sales.csv")
print(df)
Step 2: Remove Duplicates
df = df.drop_duplicates()
print(df)
Step 3: Fill Missing Values
# Fill missing price with the mean price
mean_price = df["price (TZS)"].mean()
df["price (TZS)"] = df["price (TZS)"].fillna(mean_price)
# Fill missing quantity with 0 (assuming no quantity recorded means 0 sold)
df["quantity"] = df["quantity"].fillna(0)
print(df)
Step 4: Ensure Correct Data Types
df["quantity"] = df["quantity"].astype(int)
df["price (TZS)"] = df["price (TZS)"].astype(float)
print(df.dtypes)
After cleaning, the data is ready for calculating total sales, profits, or other analysis.
When we clean data, we are changing the original records. This raises ethical questions:
- Transparency: We must document what changes were made and why.
- Accuracy: Filling in missing values with estimates can lead to false conclusions.
- Integrity: We should keep a copy of the original raw data unchanged.
- Bias: If we fill missing values based on assumptions, we may introduce bias.
In professional settings, data cleaning decisions should be recorded so that others can verify the process.
Data cleaning involves three main tasks:
- Handle missing values — either remove rows or fill them with appropriate values
- Remove duplicates — delete repeated records
- Correct data types — ensure each column has the right data type for analysis
Using pandas, these tasks can be performed quickly on datasets of any size, making data ready for science, business decisions, or research.
A market vendor in Arusha can use data cleaning to fix errors in their daily sales record — for example, removing duplicate entries of the same sale, filling in missing prices for items sold without recorded prices, and converting text entries to numbers so they can calculate total profit per day or identify which items sell best.
Jinsi utakavyotathminiwaTathmini ya mchakato · Tathmini ya matokeo · Tathmini ya maarifa
Tathmini ya mchakato
- Identify and handle missing data (e.g., imputation or removal).
- Handle duplicaterecords or invalid entries.
Tathmini ya matokeo
A cleaned dataset free from missing values, duplicates, or data type issues.
Tathmini ya maarifa
Detailed knowledge of:
Method used: The student should explain how to:
- handle common data cleaning issues.
Principles: The student should explain the principles of
- consistency, reliability, and accuracy.
Theories: The student should explain:
- Data quality dimensions (e.g., completeness, validity).
Circumstantial knowledge:
Detailed knowledge about:
- Data quality issues
Swali
What is the primary purpose of data cleaning?
Ingia ili kuwasilisha jibu lako na lihesabiwe katika umahiri wako.
Ingia ili kufanya mazoeziMwalimu
Umekwama? Niulize chochote kuhusu mada hii.
Ingia ili kumuuliza Mwalimu wa AI wa Sonza kuhusu swali hili.
Ingia ili kuulizaMajadiliano
Hakuna maswali bado