NB02: Data Automation in Python

Authored by Souvik Mandal, Ph.D.

Project Leader & Instructor, Computational Behavioral Sciences, LS100, FAS, Harvard University | Linkedin ID: souvik-mandal-phd

Last updated: 2026-08-25

Welcome to Week 2. In the previous notebook, you learned how to represent research data in Python using variables, data structures, pandas, and NumPy. That gave you the raw materials. In this notebook, we will start using those materials more strategically so Python can do repeated work for us.

The big idea this week is automation. Researchers often apply the same rule to many records: checking eligibility, filtering data, labeling observations, converting messy values, or summarizing repeated measurements. If you can express a rule clearly in code, Python can apply it faster, more consistently, and with fewer errors than your sleep-deprived spreadsheet self.

Treat this notebook as training in both coding and scientific reasoning:

1. Run cells in order.

2. Before each code cell, predict what you expect to happen.

3. After running a cell, compare your expectation with the output.

4. At each checkpoint, write one sentence about what rule or pattern your code is actually using.

Weekly Goal: move from simply storing and inspecting data to automating repeated decisions and repeated operations.

Time Plan (4-6 hours a week) This notebook has four sections. Each section is designed to take about 30-40 minutes for beginners, plus time for the checkpoint tasks and the final weekly challenge.

Table of Content:¶

1. Section 1: More Operations with pandas

   • 1.1. Inspecting and Summarizing a DataFrame

   • 1.2. Working with Data in a Dataframe

   • 1.3. Filtering Rows and Creating New Columns

   • Checkpoint 1

2. Section 2: More Operations with NumPy

   • 2.1. Arrays for Repeated Calculations

   • 2.2. Boolean Masks and When NumPy Helps More Than pandas

3. Section 3: Rules and Automation with Conditionals and Loops

   • 3.1. Conditional Logic

     • if, elif, and else

     • combining multiple decision rules

   • 3.2. Loops

     • using range

     • using enumerate

     • using zip

   • Checkpoint 3

4. Section 4: Reusable Research Helpers with Functions

   • 4.1. Why Functions Help Automation

   • 4.2. Functions with Inputs and Outputs

   • 4.3. Defensive Helpers for Messy Data

   • Checkpoint 4

   • Weekly Challenge: Build a Tiny Automation Pipeline

Section 1: More Operations with pandas¶

In Week 1, you learned how pandas can store tabular data in a DataFrame. That was the first step: getting data into a structure that is easier to inspect than a list of dictionaries. Now we will push a bit further and ask a more practical question: once the data is in a DataFrame, what do we actually do with it?

1.1. Inspecting and Summarizing a DataFrame¶

Before filtering, modeling, or reporting anything, you should first inspect the dataset. That means checking what columns are present, how many rows you have, and whether the values look roughly reasonable. This is one of the most boring habits in research coding, which is exactly why it saves you from very exciting disasters later.

We can also check the whole dataframe.

1.1.1. Using loc and iloc Indexing¶

• loc is label-based indexing. It is useful when you want to select rows and columns by their names or row labels.

• iloc is position-based indexing. It is useful when you want to select rows and columns by numeric position.

In this dataframe, we can use loc to get a participant by label and iloc to get the same participant by position. That helps us inspect one row at a time when we want to check specific values quickly.

• Access a whole row with the row index using loc

Remember counting generally starts with 0 in Python.

• Access just one column of a particular row

• Access a whole row with the row index using iloc

• Selecting data by position using .iloc

• Selecting data from multiple columns using iloc

Question: Now, think about the difference between participant_df.iloc[0,0] and participant_df.iloc[0, [0]].

When to use loc vs iloc¶

Use loc when you want to select data by labels. It is usually the better choice when your index or column names are meaningful and you want the code to read like a sentence.

Use iloc when you want to select data by position. It is the better choice when you only care about the row or column number, or when you are working with a dataframe whose labels may change.

loc advantages

• Easier to read when the dataframe has meaningful row or column names.

• More stable when the order of rows or columns changes.

• Good for selecting by label, name, or condition.

loc disadvantages

• Depends on labels being present and correct.

• Can be less convenient if you only know the numeric position.

iloc advantages

• Simple when you want the first row, second column, or another fixed position.

• Useful for slicing by index position in a predictable way.

• Works well when the dataframe structure is known and unlikely to change.

iloc disadvantages

• More fragile if rows or columns are reordered.

• Harder to understand at a glance when compared with named labels.

In practice, loc is often more advantageous for readable, label-based analysis, while iloc is often more advantageous for quick positional access, exploration, and examples where the exact row or column number matters.

1.1.2. Other ways to inspect a DataFrame¶

Besides loc and iloc, pandas gives us several quick inspection methods that are useful when we want to understand the shape and contents of a dataframe.

• head() shows the first few rows.

• tail() shows the last few rows.

• shape tells us the number of rows and columns.

• columns lists the column names.

• info() gives a compact summary of the dataframe structure, data types, and missing values.

• describe() summarizes numeric columns with common statistics like count, mean, standard deviation, minimum, and quartiles.

These are good tools to keep in mind as a to-do list for exploring a dataframe before doing more detailed analysis. You do not need a code example here yet; the goal is just to recognize the methods and know what each one is for.

1.2. Working with Data in a Dataframe¶

1.2.1. Using .apply() and lambda:¶

.apply() is a pandas method. When a column contains repeated objects like lists, .apply(...) lets us run the same function on each item in that column.

lambda is a quick way to write a small function right where we need it. In this example, we will use .apply() and lambda to calculate the mean score for each participant and then the overall mean score.

• Find the Mean of Lists Inside a Column:

In this dataset, each value in the score column is a list of scores for one participant (for example, [72, 81, 90]). We cannot call .mean() directly on the full column and expect the participant-level averages to appear automatically.

In short, we are computing:

• mean score per participant, then

• mean of those participant means.

We can rewrite the previous calculation in one compact line as shown below.

Let’s see another lambda example

Here we will use lambda to create new numeric summaries from each participant’s list of scores. In Section 3, we will build on this and use conditional logic (if/elif/else) to create rule-based labels.

Run the following lines of code to get the output for any particular row.

Notice that we added two new columns to participant_df: one for the number of recorded scores (n_scores) and one for each participant’s mean score (mean_score).

In Section 3, we will go one step further and use these kinds of values inside explicit decision rules.

1.2.2. Using append:¶

append() is a list method - it adds one new item to the end of a list. In this example, we use it inside a loop to store each participant’s mean score one by one.

This is useful when you want to start with an empty list and keep adding results as the loop runs. By the end, the list holds one value for each participant.

Notice that we are using the length of the entire dataframe, i.e. the number of rows, to iterate over each participant. We could also use len(participant_df["score"]) or len(participant_df["age"]) since the number of rows is the same as the number of score lists.

1.3. Filtering Rows and Creating New Columns¶

Once you trust the basic structure of your DataFrame, the next step is usually to apply a rule. Maybe you want only adult participants, or only participants who completed enough sessions, or only observations above a threshold. pandas makes this efficient by letting you filter rows using boolean conditions.

You can also create new columns from existing ones. This is useful when you want to label observations, mark eligibility, or create a quick indicator that will be reused later in your analysis pipeline.

Checkpoint 1¶

Use participant_df to create a filtered DataFrame containing only participants who both consented and scored at least 80.

Tasks:

• Print the columns id, score, and attendance.

• Create a new column called high_attendance using the rule attendance >= 4.

• Write one sentence explaining what your filtered output says about the sample.

Section 2: More Operations with NumPy¶

In Week 1, NumPy appeared as the fast numerical engine underneath many data operations. This week, we will use it more deliberately. The main question is not “Can NumPy do this?” The real question is “When does it help me think and compute more clearly than plain Python or even pandas?”

2.1. Arrays for Repeated Calculations¶

A NumPy array is especially useful when you want to apply the same numeric operation to many values at once. Instead of updating one score at a time, you can adjust the full set in one line. That is where automation starts feeling less like extra work and more like a tiny reward for writing clean code.

2.2. Boolean Masks and When NumPy Helps More Than pandas¶

A boolean mask is just a True/False pattern used to select values that meet a rule. NumPy is especially helpful when your data is mostly numeric and you want fast, repeated calculations. pandas is still better when the labels matter and you care about named columns or mixed data types.

So here is the practical rule of thumb: use pandas when you are reasoning about a table; use NumPy when you are reasoning about arrays of numbers. In the next section, we will build on that idea by writing explicit decision rules with conditionals and loops.

Section 3: Rules and Automation with Conditionals and Loops¶

So far, we have inspected data, filtered records, and applied repeated numeric operations. But real research workflows often require rules: include this participant, exclude that one, flag this record, or label a case for follow-up. That is where conditionals and loops become central.

3.1. Conditional Logic¶

Conditional logic means making a decision based on a rule. In Python, this usually appears with if, elif, and else. This is the language of inclusion criteria, exclusion criteria, quality flags, and many other research decisions.

Using the same decision rules on dataframes¶

Let’s use our participant_df dataframe again for this exercise.

We now want to apply the same decision rule to every row in participant_df.

Problem: How can we loop through the dataframe and label each participant as Eligible, Exclude: under 18, Exclude: no consent, or Follow up: low attendance for some activity?

In this example, we are not checking a single participant only. Instead, we want to walk through participant_df row by row and show how a placeholder statement can fit inside a loop.

The pass statement lets us leave a branch empty for now while keeping the structure of the decision rule in place. That can be useful when you know a condition matters, but you are not ready to define the action yet.

• A dataframe-based version of the same idea.

3.2. Loops¶

A loop lets you apply the same pattern across many values or many records. This is one of the first places where code starts to feel like automation instead of organized typing.

This section focuses on three especially useful loop patterns:

• range() when you want repeated steps tied to positions or counts

• enumerate() when you want both the item and its position

• zip() when you want to iterate through multiple iterables (e.g., list, tuples, arrays, Series, and generators) in lockstep, pairing values by position

zip() stops at the shortest input, so it is best when the items are already aligned by index or length.

You do not need to memorize every loop pattern today. The goal is to understand what each pattern is good for and why it can simplify a repeated research task.

Now, let’s use these loop patterns in a dataframe.

Checkpoint 3¶

Use conditionals and a loop to label each participant as "Eligible" or "Not eligible" using this rule:

• age >= 18

• consent == True

• attendance >= 3

Tasks:

• Print each participant’s status.

• Count how many are eligible.

• Print the final eligible count.

• Bonus: if a participant is not eligible, print the first reason they fail the rule.

Section 4: Reusable Research Helpers with Functions¶

By now, you have seen the same logic appear more than once: classify a score, check eligibility, convert a value, apply a rule across several records. When the same pattern appears repeatedly, it is usually a sign that a function can help.

4.1. Why Functions Help Automation¶

A function lets you package a rule into a named unit of code. That makes your workflow easier to test, easier to reuse, and much easier to revise when your criteria change. Instead of rewriting the same rule in five places, you update it once.

4.2. Functions with Inputs and Outputs¶

Good beginner functions usually do one clear thing: they take an input, apply a rule, and return an output. That sounds almost disappointingly simple, which is why it is so useful.

4.3. Defensive Helpers for Messy Data¶

Real data loves surprises. A value that should be numeric may arrive as text. A field may be missing. A label may contain something you did not expect. Defensive coding means writing helpers that fail gracefully instead of breaking the entire workflow.

In the next code cell, one function packages a classification rule, and another handles messy conversion more safely.

Checkpoint 4¶

Write a function eligibility_label(age, consent, attendance) that returns either "Eligible" or "Not eligible".

Then test at least four cases and briefly explain one edge case where writing the rule explicitly helps you avoid confusion.

Weekly Challenge: Build a Tiny Automation Pipeline¶

Now combine the main ideas from this notebook.

Your task is to build a small workflow that:

• converts raw text values to usable numbers when needed,

• applies an eligibility rule,

• uses a loop to process many participant records,

• uses at least one function so the logic is reusable,

• prints a short summary report.

Keep the code clear enough that a collaborator could read it without needing a decoder ring from a secret programming society.