Mada za sehemu hiiAnalysing dataMada 2
- Performing basic statistical analysis
- Using specialised libraries to analyse data
Using Specialised Libraries to Analyse Data
When we analyse data in programming, we often need more than simple calculations. Specialised Python libraries like statsmodels, scipy, and seaborn provide ready-made functions for hypothesis testing, correlation analysis, and advanced statistics. These tools let us make evidence-based decisions from data.
- Writing statistical tests from scratch is complex and error-prone
- Libraries like
scipyandstatsmodelsimplement trusted, tested algorithms - They save time and produce accurate results quickly
- They are used by data scientists worldwide
First, we import the libraries we need:
import pandas as pd
import numpy as np
from scipy import stats
import statsmodels.api as sm
import seaborn as sns
import matplotlib.pyplot as plt
Hypothesis testing helps us determine whether observed differences in data are real or due to chance.
The T-Test (Comparing Two Groups)
A t-test compares the means of two groups to see if they are significantly different.
# Example: Test scores from two schools in Tanzania
school_a_scores = [72, 75, 68, 80, 71, 77, 74, 69, 76, 73]
school_b_scores = [58, 62, 65, 60, 59, 63, 61, 67, 64, 66]
# Perform independent t-test
t_statistic, p_value = stats.ttest_ind(school_a_scores, school_b_scores)
print(f"T-statistic: {t_statistic:.4f}")
print(f"P-value: {p_value:.4f}")
Interpreting the result:
- If p-value < 0.05, the difference between the two schools is statistically significant
- If p-value > 0.05, we cannot conclude there is a real difference
One-Sample T-Test
This tests whether a sample mean differs from a known value:
# Example: Test if average maize yield is different from expected 5000 kg/ha
maize_yields = [4800, 5200, 4900, 5100, 4700, 5300, 4950, 5050]
t_stat, p_value = stats.ttest_1samp(maize_yields, 5000)
print(f"P-value: {p_value:.4f}")
When we want to compare more than two groups, we use ANOVA.
# Example: Sales at three markets in Dar es Salaam (in TZS thousands)
market_kariakoo = [120, 135, 128, 142, 130]
market_mwenge = [98, 105, 112, 99, 108]
market_mikocheni = [145, 152, 148, 155, 150]
# Perform one-way ANOVA
f_statistic, p_value = stats.f_oneway(market_kariakoo, market_mwenge, market_mikocheni)
print(f"F-statistic: {f_statistic:.4f}")
print(f"P-value: {p_value:.4f}")
Correlation measures the relationship between two variables. We use Pearson correlation for linear relationships.
# Example: Hours studied vs. exam scores for Form 4 students
hours_studied = [2, 3, 4, 5, 6, 7, 8]
exam_scores = [45, 55, 62, 70, 78, 85, 92]
# Calculate Pearson correlation
correlation, p_value = stats.pearsonr(hours_studied, exam_scores)
print(f"Correlation coefficient: {correlation:.4f}")
print(f"P-value: {p_value:.4f}")
A correlation of +1 means perfect positive relationship, -1 means perfect negative, and 0 means no relationship.
Seaborn makes it easy to create informative statistical plots:
# Create a scatter plot with regression line
data = pd.DataFrame({
'hours': hours_studied,
'scores': exam_scores
})
sns.regplot(x='hours', y='scores', data=data)
plt.xlabel('Hours Studied')
plt.ylabel('Exam Score')
plt.title('Relationship Between Study Time and Exam Performance')
plt.show()
| Situation | Recommended Test |
|---|---|
| Compare two groups (e.g., two schools) | Independent t-test |
| Compare one group to a known value | One-sample t-test |
| Compare three or more groups | ANOVA |
| Measure relationship between two numeric variables | Pearson correlation |
A small shop owner in Arusha recorded daily sales (in TZS) for three months:
import pandas as pd
from scipy import stats
import seaborn as sns
import matplotlib.pyplot as plt
# Sample data
weekday_sales = [150000, 165000, 140000, 155000, 160000]
weekend_sales = [210000, 195000, 230000, 205000, 220000]
# Perform t-test
t_stat, p_value = stats.ttest_ind(weekday_sales, weekend_sales)
print(f"Weekday average: {sum(weekday_sales)/len(weekday_sales):,.0f} TZS")
print(f"Weekend average: {sum(weekend_sales)/len(weekend_sales):,.0f} TZS")
print(f"P-value: {p_value:.4f}")
if p_value < 0.05:
print("Conclusion: Weekend sales are significantly higher.")
else:
print("Conclusion: No significant difference between weekday and weekend sales.")
Output:
Weekday average: 154,000 TZS
Weekend average: 212,000 TZS
P-value: 0.0003
Conclusion: Weekend sales are significantly higher.
In Tanzania, shop owners and market traders can use these statistical libraries to analyse daily sales data and determine whether weekend sales are truly higher than weekday sales, or whether price changes actually affect customer buying patterns. A maize farmer in Morogoro could record harvest yields from different fertilizer types and use ANOVA to determine which fertilizer produces significantly better results, then invest wisely in the most effective one.
Jinsi utakavyotathminiwaTathmini ya mchakato · Tathmini ya matokeo · Tathmini ya maarifa
Tathmini ya mchakato
- Apply specialized libraries to solve statistical problems.
- Interpret statistical test results.
Tathmini ya matokeo
Python scripts demonstrating use of specialized libraries for statistical analysis and visualizations.
Tathmini ya maarifa
Detailed knowledge of:
Method used: The student should explain how to
- apply specialized libraries for real-world data analysis problems.
Principles: The student should explain the principles of:
- Statistical testing for hypothesis validation
Theories: The student should explain:
- Fundamentals of correlation, regression, and p-values.
Circumstantial knowledge:
Detailed knowledge about:
- assumptions and limitations of statistical methods used.
Swali
Which Python library from the following provides functions for performing t-tests, ANOVA, and correlation analysis?
Ingia ili kuwasilisha jibu lako na lihesabiwe katika umahiri wako.
Ingia ili kufanya mazoeziMwalimu
Umekwama? Niulize chochote kuhusu mada hii.
Ingia ili kumuuliza Mwalimu wa AI wa Sonza kuhusu swali hili.
Ingia ili kuulizaMajadiliano
Hakuna maswali bado