How to Deal With Missing Data in Polars – Real Python

How to Work With Missing Data in Polars

When you’re dealing with missing data in Polars, there are several things you can do:

• Recover it
• Remove it
• Replace it
• Ignore it

By far the best plan of action is to review the techniques used to collect the source data, and try to find out why missing data exists in the first place. Perhaps it’s been caused by a manual data entry problem, or a failure of another system to provide the values. Once you know why it’s missing, you can take measures to ensure future data collections are complete, or even repeat the collection process and recover the current missing values.

If you can’t recover the missing data, the next best approach is to remove or replace it. Be aware that removing data may cause bias in calculations, so you should only do this if this isn’t a concern. A better option might be to replace it with an alternative, more usable value.

It’s also possible to ignore missing values. This approach might be fine in cases where you’re collecting text-based data that contains customer reviews or opinions. However, missing numerical data can skew calculations, and your code may even crash if it can’t cope with null values properly.

From your initial findings, you notice that there are two rows in the total column with nulls, and four in the tip column that you’ll deal with first. You decide to find out if there are any rows where there are null values in both of these columns:

After you import the Polars library and create a tips LazyFrame, you add further instructions to filter out everything apart from any rows that contain a null in both their total and tip columns. You still need to use .collect() to materialize your LazyFrame into a DataFrame to see the results.

To create the filter, you use .filter() to specify a filter context and pass in an expression to define the criteria. In this case, the expression pl.col("total").is_null() & pl.col("tip").is_null() tells the LazyFrame which columns to analyze. The Boolean AND (&) ensures that only those rows whose tip and total values are both null are included in the result.

One point to note is that applying the filter() method to tips doesn’t actually change the original tips LazyFrame in any way. Instead, it creates a fresh LazyFrame containing an updated set of instructions. This means several LazyFrames can be developed from the original tips LazyFrame by applying different methods to it.

Later in the tutorial, you’ll develop this idea further by adding successive refinements to a LazyFrame by appending multiple methods to a base LazyFrame. Each new method added has the effect of tweaking the LazyFrame further. This is commonly known as method chaining and is the recommended way of manipulating LazyFrames.

As you can see from the output, the records with a record_id of 4 and 18 meet the filter’s criteria.

After further investigation, you decide to drop these rows since they’re the only two where the total column contains a null value, as revealed by your earlier use of .null_count():

To drop these rows, you use the LazyFrame’s .drop_nulls() method. This adds an instruction into your LazyFrame to drop all rows whose total column contains null values. Performing the drop on only these null values works fine here because there are no other rows containing null in their total column. You’ll see later how to create an instruction to base the drop on the presence of nulls in both columns.

The .filter() method won’t be needed in the final code. You’re including it here to display the filtered data to confirm the rows have indeed been deleted. As you can see, this time the filter finds nothing.

Recall that your initial analysis revealed some nulls in the tip column that also have amounts in their total column. You decide these are genuine records so decide to update tip to contain zero for these records. This will make sure there won’t be any unnecessary bias should you need to analyze the tips later. To do this, you use .fill_null():

You add another instruction to your LazyFrame, this time telling it to fill any existing null values in the tip column with the number 0. Then, you apply the instruction to the tip column using the .with_columns() context and specify your exact requirements as its parameter. You use with_columns() because you were changing the value of the column’s data.

To verify the changes, a filter is applied once again to confirm that the tip column no longer contains any null values. As with your earlier filter, this one should also be removed in the final version of the code.

Now that you’ve learned the basics of removing and replacing null values, it’s time to explore other Polars techniques for handling missing data.

Using a More Strategic Approach

In this section, you’ll deal with the null values that exist within the time column. As you saw earlier, there are two of them. First of all, you’ll take a look at the rows you need to deal with:

Using the .filter() method, as before, reveals that records with record_id values of 3 and 15 contain null values in their time column. To deal with these, you decide to look for clues in the surrounding data:

You may have noticed that a pair of null values have reappeared in the total and tip columns of record 4. This is because the filter is applied to the original tips LazyFrame. This problem will sort itself out when you add the new instructions to your modified LazyFrame.

What stands out this time is that the same value surrounds each null in the time column. You decide to fill each of these null values with their surrounding value. To do this, you can make use of the strategy parameter of .fill_null():

Here, .with_columns() analyzes each row in the time column. When it reaches a cell whose time value is null, .fill_null() will replace the null using a forward strategy. This means it’ll forward-fill the null with the previous non-null value, which will be the Dinner value from record 2. Similarly, record 15 will have its null replaced with Lunch from record 14.

As before, so you can see the results of your work, you use a filter on these rows. The forward strategy is just one of several strategies that are available to you.

Next, you’ll look at how to deal with null data that exists across multiple columns.

Dealing With Nulls Across Multiple Columns

Earlier, you saw that records 4 and 18 in the original tips.parquet file have null values in both the total and tip columns. You removed them by using only the null values in the total column. This worked because these were the only two records with nulls in this column.

Suppose there were other rows where total contained a null value but tip had a valid entry. These would have been removed as well, even though they may not have needed to be. It would be safer to remove only those records where the null values exist in both columns. To do this, you can make use of the .all_horizontal() function:

The .all_horizontal() function allows you to compute a Boolean AND horizontally across columns. You use it here to select rows that have nulls in both their total and tip columns.

By passing in both total and tip columns to pl.col() and using .is_null(), you’re effectively passing both pl.col("total").is_null() and pl.col("tip").is_null() as two separate parameters to .all_horizontal(). Each parameter will yield a True or False value for each row depending on whether or not their total and tip columns contain null values.

If both parameters are True, .all_horizontal() will return True and that row will be included in the output. However, if either of these return False, .all_horizontal() will also return False for those rows, meaning they will be ignored in its output. That’s why no other rows are present—there are no other rows with both a total of null and a tip of null.

You used .all_horizontal() to produce the same two rows as before. If you want to go one step further and instruct your LazyFrame to ignore these rows, you just need to make one small change to your code:

Can you spot the difference? Take a close look at the highlighted line. You added the NOT operator (~) to the all_horizontal() function. This has the effect of negating it.

In the earlier version of this code, you added instructions to your LazyFrame causing it to display only two rows. This time, the NOT operator ensures that the LazyFrame displays everything except those two rows. As a result, rows where both the total and tip columns contain null values are no longer included.

You could have used this code in place of the earlier version to remove these values immediately. The full version of your null-cleansing code now looks like this:

In a production environment, you’d probably use something like .collect() to view the cleansed version of your data, or .collect().write_parquet("tips_clean.parquet") to update your original data file with it. Here, you use .null_count().collect() so you can be satisfied that the original null values no longer exist.

Congratulations. You’ve successfully dealt with all of those annoying null values, but there are other techniques that you may find handy to know. The final method you’ll learn about for removing null values involves using the column’s data type.

Dealing With Nulls by Column Data Type

In this section, you’ll learn about Polars selectors. Selectors allow you to select columns based not only on their names but also on their data types and other properties.

To begin with, you’ll need a new LazyFrame. The code below creates one named scientists:

The LazyFrame contains details of five renowned scientists from various disciplines. While creating the LazyFrame, several null values were included in it. To create these null values, you used None instead of normal data values.

Suppose you wanted to update the null values to “Unknown” in those columns with a data type of str, and replace null values in the i64 columns with zero. To do this, you can use the Polars selectors:

To use selectors, you must first import polars.selectors. It’s standard practice to do so using an alias of cs. Think of this as short for column selectors.

As before, because you’re going to change a column, you once again use .with_columns(). This time, you’ll choose each of the string columns using the string data type selector cs.string(), and each of the integer columns using cs.integer(). To update the null values within these columns you again use .fill_null().

In this case, you replace the nulls in each of the str columns with the string Unknown, and each null in the i64 columns (or i32 columns if there were any) with a 0. If you look closely at the output, you’ll see the first_name and last_name columns no longer contain any null values. Instead, you’ve replaced them with Unknown. Also, the null values that were in birth_year and death_year have all been replaced with 0.

Although nulls can be a nuisance, they’re not the only thing that can mess you about. You’ll learn about another annoyance you may encounter next.

Dealing With Those Pesky NaNs and Infs

Sometimes, the data you’re analyzing contains the results of calculations. If incorrect data enters a calculation, it may cause erroneous results to appear. The most common examples are non-numeric values or NaNs, usually caused by trying to perform a numeric calculation in non-numeric data, and infinity values (inf, -inf), which are usually caused by a zero-division.

As with null data, you should try and find out why the invalid data has occurred and see if you can resolve the situation. If not, you’ll probably want to convert the offending values to null values, then use the techniques you learned earlier to remove them or replace them with something more suited to your analysis.

The code below scans the sales_trends.csv file included in your downloads into a LazyFrame named sales_trends:

This LazyFrame contains sales volumes for five products labeled A though E. The sales volumes for last_year and the current_year are complete. However, the next_year figures show some erroneous data. Before going any further with this data, you decide to replace it with null values:

To replace each of the different offending data in the next_year column, you passed a list of them as a parameter to the replace() method. As you can see from the output, the unknown data is now set to null.

Now that all of the erroneous data has been replaced with null values, you can use .fill_null() to update them all using the same logic. For example, in the absence of anything better, you decide to predict next year’s data by extrapolating it from the known data:

Once the erroneous data has been replaced with null values, you add some additional instructions telling your LazyFrame how to deal with them. You replace each null by adjusting the current_year figure. You add it to the difference between the last_year and current_year sales figures to produce a next_year estimate. Your data has now been cleansed of those pesky NaNs and infs.

It’s time to show off your prowess—it’s time for your review.

Practicing Your Skills

Now it’s your turn to demonstrate your skills and have some fun handling null data in Polars. After leaving the Monty Python comedy troupe, after which the Python language is named, John Cleese wrote the successful television show Fawlty Towers. In this section, you’ll analyze some data relating to it.

Well done if you managed to complete that exercise—and indeed, well done for making it this far!

Conclusion

In this tutorial, you’ve learned various techniques for removing or replacing null values in Polars.

You also learned that while there are a range of ways for dealing with null values, the best way is to avoid them in the first place. In other words, see if you can find out what caused them and take steps to prevent this from happening in the future. You should also see if you can figure out what’s missing and try to recover it.

Congratulations on completing this tutorial. Enjoy applying these newfound skills to better prepare your LazyFrames for analysis.

Frequently Asked Questions

Now that you have some experience with handling missing data in Polars, you can use the questions and answers below to check your understanding and recap what you’ve learned.

These FAQs are related to the most important concepts you’ve covered in this tutorial. Click the Show/Hide toggle beside each question to reveal the answer.