Loading and Inspecting Datasets with Pandas: A Practical Guide

Previously in this course, we explored Introduction to NumPy for Data Handling: Arrays and Vectorization to manage numerical data. While NumPy is excellent for mathematical operations, real-world data is rarely a clean, homogeneous array of numbers. It is usually messy, mixed-type, and stored in tabular formats like CSVs.

This is where Pandas comes in. It is the industry-standard tool for data manipulation in Python, built on top of NumPy but designed specifically for labeled, tabular data. In this lesson, we will focus on loading data into a DataFrame and performing initial EDA (Exploratory Data Analysis) to understand what we are actually working with.

Understanding the DataFrame

A DataFrame is essentially a 2D labeled data structure—think of it as an intelligent, programmatic spreadsheet. Unlike a raw NumPy array, a DataFrame allows you to have columns with different data types (e.g., integers, floats, strings) and provides row and column labels that make indexing intuitive.

In the Machine Learning Workflow, we established that data is the lifeblood of our models. Before we can train anything, we must be able to ingest that data reliably.

Loading Data with Pandas

To start, ensure you have your environment configured as described in our Setting Up the Python ML Environment lesson.

We use pd.read_csv() to import data. Here is how you load a typical dataset:

PYTHON
import pandas as pd

# Load the dataset
# We'll use a placeholder filename here; replace with your actual file
df = pd.read_csv(CE9178">'data/housing_prices.csv')

Inspecting Data Structure

Once the data is loaded, your first job is to "look under the hood." Never assume a dataset is clean. Use these three essential methods to audit your data:

1. df.head(): Displays the first 5 rows. It’s the quickest way to see if the file loaded correctly and what the data looks like.
2. df.info(): Shows the number of rows, column names, data types, and non-null counts. This is your primary tool for identifying missing values and ensuring numbers aren't being read as strings.
3. df.describe(): Generates summary statistics (mean, min, max, std) for numerical columns. It helps you spot outliers or impossible values immediately.

Worked Example

Let's see these in action. Imagine we have a housing dataset.

PYTHON
# 1. Peek at the top of the data
print(df.head())

# 2. Check structure and data types
print(df.info())

# 3. Get summary statistics
print(df.describe())

When you run df.info(), pay close attention to the Dtype column. If a column that should be numerical (like "price") shows up as object, it usually means there is a non-numeric character (like a currency symbol or a typo) hiding somewhere in that column.

Hands-on Exercise

1. Download a sample CSV file (e.g., from Kaggle or a public repository).
2. Write a script to load this file into a Pandas DataFrame.
3. Use df.info() to count how many columns are numerical versus categorical (objects).
4. Identify if any columns have fewer non-null values than the total number of rows (this indicates missing data).

Common Pitfalls

• Assuming Data Integrity: Never trust the raw file. Always check df.info() to verify that integers aren't being treated as floats or strings.
• Working with Large Files: If your file is gigabytes in size, pd.read_csv() will crash your RAM. Use the chunksize parameter or load only a subset of rows for testing.
• Case Sensitivity: CSV column names are case-sensitive. df['Price'] and df['price'] are different; always check your column headers using df.columns if you get a KeyError.

Recap

In this lesson, we transitioned from basic numerical arrays to real-world data handling. You now know how to:

• Load CSV files using Pandas.
• Instantiate a DataFrame for analysis.
• Use head, info, and describe to perform initial EDA.
• Identify data types and potential issues before the modeling phase.

These skills are the gatekeepers to effective machine learning. If you cannot load and understand your data, you cannot model it.

Up next: Exploratory Data Analysis Fundamentals — where we will learn how to visualize the distributions and relationships we just discovered.