Python Dictionaries and Sets for Data Analysis

This notebook demonstrates how Python dictionaries and sets can be used for data analysis, including lookup operations, data transformation, duplicate removal, and dataset comparison.

Data Analysis Examples of Python Dictionaries

Dictionaries are frequently used for:

1. Converting state codes → state names
2. Mapping FIPS codes → geographic names
3. Recoding survey responses
4. Standardizing inconsistent categories
5. Creating labels for charts and dashboards
6. Preparing data before merging with external datasets

Create a dictionary for lookup

import pandas as pd

# Create population dataset
population = pd.DataFrame({
    "State_Code": ["CA", "TX", "FL", "NY"],
    "Population": [39000000, 30000000, 22000000, 19500000]
})

# Create lookup table (dictionary)
state_lookup = {
    "CA": "California",
    "TX": "Texas",
    "FL": "Florida",
    "NY": "New York"
}

# Add State_Name using the lookup table
population["State_Name"] = population["State_Code"].map(state_lookup)

# Display the transformed dataset
population

	State_Code	Population	State_Name
0	CA	39000000	California
1	TX	30000000	Texas
2	FL	22000000	Florida
3	NY	19500000	New York

Data Analysis Examples of Python Sets

Sets are commonly used for:

1. Removing duplicate IDs
2. Checking whether records exist in another dataset
3. Comparing lists of variables between datasets
4. Finding missing categories
5. Validating data values

Remove duplicate states using a Python set

# List of states with duplicates
states = [
    "California",
    "Texas",
    "Florida",
    "New York",
    "California",
    "Texas"
]

# Remove duplicates
unique_states = set(states)

print(unique_states)

{'Texas', 'Florida', 'California', 'New York'}

Example: Create a set of U.S. state names

# Create a list of state names
states = [
    "California",
    "Texas",
    "Florida",
    "New York",
    "California",
    "Texas"
]

# Convert list to a set
state_set = set(states)

print(state_set)

{'Texas', 'Florida', 'California', 'New York'}

Create a population dataset and convert the state names into a set

import pandas as pd

# Create population dataset
population = pd.DataFrame({
    "State_Code": ["CA", "TX", "FL", "NY"],
    "Population": [39000000, 30000000, 22000000, 19500000],
    "State_Name": [
        "California",
        "Texas",
        "Florida",
        "New York"
    ]
})

# Create a set from the State_Name column
state_set = set(population["State_Name"])

print(state_set)

{'Texas', 'Florida', 'California', 'New York'}

Use case 1: Check if a state exists

if "Texas" in state_set:
    print("Texas is included in the dataset")

Texas is included in the dataset

Use case 2: Find missing states

Suppose you expect five states, but your actual dataset contains only four states. Which state is missing?

expected_states = {
    "California",
    "Texas",
    "Florida",
    "New York",
    "Ohio"
}

actual_states = {
    "California",
    "Texas",
    "Florida",
    "New York"
}

missing_states = expected_states - actual_states

print(missing_states)

{'Ohio'}

A common use of a Python set in data analysis is comparing the variable (column) names between two datasets.

Example: Compare variables between two datasets

import pandas as pd

# Create first dataset
population = pd.DataFrame({
    "State_Code": ["CA", "TX", "FL"],
    "State_Name": ["California", "Texas", "Florida"],
    "Population": [39000000, 30000000, 22000000],
    "Year": [2025, 2025, 2025]
})

# Create second dataset
census = pd.DataFrame({
    "State_Code": ["CA", "TX", "FL"],
    "State_Name": ["California", "Texas", "Florida"],
    "Population": [39000000, 30000000, 22000000],
    "Median_Age": [37, 35, 43]
})

# Convert column names to sets
population_vars = set(population.columns)
census_vars = set(census.columns)

# Variables common to both datasets
common_vars = population_vars.intersection(census_vars)

# Variables only in population dataset
only_population = population_vars - census_vars

# Variables only in census dataset
only_census = census_vars - population_vars

print("Common variables:")
print(common_vars)

print("\nOnly in population dataset:")
print(only_population)

print("\nOnly in census dataset:")
print(only_census)

Common variables:
{'Population', 'State_Code', 'State_Name'}

Only in population dataset:
{'Year'}

Only in census dataset:
{'Median_Age'}

from pathlib import Path
root = Path(r"C:\Explore")
for file in root.rglob("PythonDictionaryAndSet.ipynb"):
    if ".virtual_documents" not in str(file):
        print(file)

C:\Explore\Python\PythonDictionaryAndSet.ipynb