Mada za sehemu hiiCollecting and storing dataMada 4
- Collecting data from multiple sources
- Implementing retrieval of data from files
- Implementing local data storage
- Performing connection to remote databases
Reading Data from Files
When you write a program, you often need to work with data that has been saved previously. Instead of entering data manually every time, you can read it from files stored on your computer. This makes your programs faster, more efficient, and able to handle large amounts of information.
In this topic, you will learn how to retrieve data from three common file types: CSV, Excel, and JSON files. These are formats frequently used in data collection and data science.
CSV stands for Comma Separated Values. Each value in the file is separated by a comma, and each new line represents a new row. The first row is usually the header row, which contains column names.
Using the csv Module
Python's built-in csv module allows you to read CSV files easily:
import csv
# Open the file in read mode
with open('students.csv', 'r') as file:
reader = csv.reader(file)
# Skip the header row
header = next(reader)
print("Columns:", header)
# Print each data row
for row in reader:
print(row)
Using the pandas Library
The pandas library provides a more powerful way to read data. It stores data in a DataFrame, which is like a table:
import pandas as pd
# Read CSV file into a DataFrame
df = pd.read_csv('students.csv')
# Display the data
print(df)
Excel files (.xlsx) can contain multiple sheets. The pandas library can read Excel files directly:
import pandas as pd
# Read Excel file (default reads the first sheet)
df = pd.read_excel('sales.xlsx')
# Read a specific sheet by name
df = pd.read_excel('sales.xlsx', sheet_name='January')
# Display the data
print(df)
JSON (JavaScript Object Notation) stores data in a structured text format. It is commonly used for web applications and APIs:
import pandas as pd
# Read JSON file
df = pd.read_json('products.json')
print(df)
You can also use Python's built-in json module:
import json
with open('products.json', 'r') as file:
data = json.load(file)
print(data)
When reading real data, you will often encounter problems. Two common issues are:
1. Missing Values
Data may be empty in some cells. Pandas represents these as NaN (Not a Number). You can check for missing values:
import pandas as pd
df = pd.read_csv('data.csv')
# Check for missing values in each column
print(df.isnull())
# Fill missing values with a default
df.fillna(0, inplace=True)
2. Incorrect Formats
Data may be in the wrong format, such as text where a number is expected:
# Convert a column to numeric, forcing errors to become NaN
df['price'] = pd.to_numeric(df['price'], errors='coerce')
# Fill any resulting NaN values
df['price'].fillna(0, inplace=True)
A small shop in Arusha maintains a CSV file called items.csv with the following content:
item_name,quantity,price
Rice,50,25000
Sugar,30,15000
Salt,100,5000
Cooking Oil,20,35000
The owner wants to read this data and calculate the total value of all items in stock.
import pandas as pd
# Read the CSV file
df = pd.read_csv('items.csv')
print("Stock Report:")
print(df)
# Calculate total value for each item
df['total_value'] = df['quantity'] * df['price']
# Calculate overall stock value
total_stock = df['total_value'].sum()
print("\nItem-wise totals:")
print(df)
print(f"\nTotal stock value: TZS {total_stock:,.0f}")
Output:
Stock Report:
item_name quantity price
0 Rice 50 25000
1 Sugar 30 15000
2 Salt 100 5000
3 Cooking Oil 20 35000
Item-wise totals:
item_name quantity price total_value
0 Rice 50 25000 1250000
1 Sugar 30 15000 450000
2 Salt 100 5000 500000
3 Cooking Oil 20 35000 700000
Total stock value: TZS 2,900,000
This program successfully reads the data, processes it, and produces useful information for the shop owner.
- Use
csv.reader()orpd.read_csv()to read CSV files - Use
pd.read_excel()to read Excel files - Use
pd.read_json()orjson.load()to read JSON files - Handle missing values using
isnull()andfillna() - Handle incorrect formats using
pd.to_numeric()with theerrors='coerce'parameter
In Tanzania, market vendors and small shop owners can use file reading to track their inventory. For example, a duka in Mwanza can maintain an Excel sheet of goods received, and a program can read this file daily to automatically calculate remaining stock and alert the owner when items are running low — helping them restock on time and avoid lost sales.
Jinsi utakavyotathminiwaTathmini ya mchakato · Tathmini ya matokeo · Tathmini ya maarifa
Tathmini ya mchakato
- Retrieve data correctly from various file types.
- Handle file errors correctly.
Tathmini ya matokeo
A Python program that reads, cleans, and processes data from files such as CSV, Excel, or JSON.
Tathmini ya maarifa
Detailed knowledge of:
Method used: The student should explain how to:
- read and write files using Python
Principles: The student should explain the principles of
- data integrity during retrieval.
Theories: The student should explain: engineering processes.
Circumstantial knowledge:
Detailed knowledge about:
- handling large or corrupted files.
Swali
What does the abbreviation CSV stand for?
Ingia ili kuwasilisha jibu lako na lihesabiwe katika umahiri wako.
Ingia ili kufanya mazoeziMwalimu
Umekwama? Niulize chochote kuhusu mada hii.
Ingia ili kumuuliza Mwalimu wa AI wa Sonza kuhusu swali hili.
Ingia ili kuulizaMajadiliano
Hakuna maswali bado