Mada za sehemu hiiCleaning and organising dataMada 4
- Performing basic data cleaning
- Performing basic data transformations
- Performing basic data normalisation and standardisation
- Using specialised libraries for data cleaning
Data transformation is the process of changing the structure, format, or values of data to make it more suitable for analysis. In data science and programming, we often need to clean, organize, and reshape our data before we can use it to answer questions or make decisions. This topic covers the basic techniques that every data analyst must know: filtering rows, sorting rows, renaming columns, and creating new calculated columns from existing data.
Filtering means selecting only the rows that meet certain conditions. When working with datasets, we often need to work with a subset of data that satisfies specific criteria.
To filter data in Python using pandas, we use conditions inside square brackets. For example, to select only rows where a column called price is greater than a certain value, we write:
filtered_data = data[data['price'] > 5000]
This reads as: "Give me rows from data where the price column is greater than 5000."
We can combine multiple conditions using:
&for AND (both conditions must be true)|for OR (at least one condition must be true)
Example: selecting students who scored above 60 in both Mathematics and English:
passed = data[(data['math'] > 60) & (data['english'] > 60)]
Sorting arranges rows in order based on one or more columns. We can sort in ascending order (smallest to largest) or descending order (largest to smallest).
The pandas function for sorting is sort_values(). Basic syntax:
sorted_data = data.sort_values(by='column_name')
To sort in descending order, add ascending=False:
sorted_data = data.sort_values(by='price', ascending=False)
To sort by multiple columns, pass a list of column names:
sorted_data = data.sort_values(by=['district', 'sales'], ascending=[True, False])
This sorts first by district in alphabetical order, then by sales in descending order.
Sometimes column names are unclear, too long, or use inconsistent naming. Renaming columns makes data easier to work with and understand.
In pandas, we use the rename() function with a dictionary that maps old names to new names:
data = data.rename(columns={'old_name': 'new_name'})
Example:
data = data.rename(columns={'tot_sales_2023': 'total_sales', 'cust_name': 'customer_name'})
A derived feature is a new column whose values are calculated from existing columns. This is also called feature engineering. Common examples include:
- Total or sum columns: Adding multiple values together
- Average columns: Calculating means
- Categories: Converting continuous values into groups
- Boolean flags: Creating True/False columns based on conditions
To create a new column in pandas:
data['new_column'] = data['column1'] + data['column2']
Example: calculating total marks from three subjects:
data['total_marks'] = data['math'] + data['english'] + data['science']
To create a column based on a condition, we use the np.where() function from NumPy:
import numpy as np
data['status'] = np.where(data['attendance'] >= 75, 'Present', 'Absent')
This creates a 'status' column that shows 'Present' if attendance is 75 or more, otherwise 'Absent'.
When preparing data for reports or further analysis, we may need to change the order of columns. We can select columns in a new order:
data = data[['name', 'district', 'sales', 'target', 'achievement']]
This rearranges the dataset so columns appear in the specified order.
Consider a small shop in Arusha that tracks daily sales of maize flour. The dataset has columns: date, product, quantity_sold, price_per_unit, and district.
import pandas as pd
import numpy as np
# Sample data
sales_data = pd.DataFrame({
'date': ['2024-01-01', '2024-01-01', '2024-01-02', '2024-01-02'],
'product': ['maize_flour', 'rice', 'maize_flour', 'sugar'],
'quantity_sold': [50, 20, 30, 15],
'price_per_unit': [2500, 2200, 2500, 3000],
'district': ['Arusha', 'Meru', 'Arusha', 'Arusha']
})
# 1. Calculate total revenue for each row
sales_data['total_revenue'] = sales_data['quantity_sold'] * sales_data['price_per_unit']
# 2. Filter only maize flour sales
maize_sales = sales_data[sales_data['product'] == 'maize_flour']
# 3. Sort by total revenue (highest first)
maize_sales = maize_sales.sort_values(by='total_revenue', ascending=False)
# 4. Rename column for clarity
maize_sales = maize_sales.rename(columns={'quantity_sold': ' packets_sold'})
print(maize_sales)
After running this code, the output shows only maize flour sales, sorted from highest to lowest revenue, with a clearer column name. The shop owner can quickly see which days generated the most income from maize flour.
| Operation | Pandas Function |
|---|---|
| Filter rows | data[data['column'] > value] |
| Sort data | data.sort_values(by='column') |
| Rename columns | data.rename(columns={'old': 'new'}) |
| Create new column | data['new'] = expression |
| Conditional column | np.where(condition, true_val, false_val) |
| Reorder columns | data[['col1', 'col2', 'col3']] |
In Tanzania, market vendors and small business owners can use basic data transformation to analyse their daily sales. For example, a trader at Kariakoo Market in Dar es Salaam can filter their sales data to see only rice transactions, calculate total earnings per day by multiplying quantity sold by price, sort to find the best-selling days, and identify which products need promotion. This helps them make better decisions about which items to stock more of and when to offer discounts.
Jinsi utakavyotathminiwaTathmini ya mchakato · Tathmini ya matokeo · Tathmini ya maarifa
Tathmini ya mchakato
- Manipulate data structures and apply transformations like filtering and sorting.
- Create meaningful derived variables.
Tathmini ya matokeo
A transformed dataset that meets specified requirements (e.g., sorted, filtered, or with additional features).
Tathmini ya maarifa
Detailed knowledge of:
Method used: The student should explain how to:
- apply transformations on data
Principles: The student should explain the principles of
- Reproducibility in data transformations.
Theories: The student should explain:
- Feature engineering and its role in improving data utility.
Circumstantial knowledge:
Detailed knowledge about:
- Handling errors when performing complex transformations (e.g., applying functions to entire columns).
Swali
Which pandas expression selects only rows where the age column is greater than 18?
Ingia ili kuwasilisha jibu lako na lihesabiwe katika umahiri wako.
Ingia ili kufanya mazoeziMwalimu
Umekwama? Niulize chochote kuhusu mada hii.
Ingia ili kumuuliza Mwalimu wa AI wa Sonza kuhusu swali hili.
Ingia ili kuulizaMajadiliano
Hakuna maswali bado