• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

The Programming Expert

Solving All of Your Programming Headaches

  • HTML
  • JavaScript
  • jQuery
  • PHP
  • Python
  • SAS
  • Ruby
  • About
You are here: Home / Python / Using Python to Remove Duplicates from List

Using Python to Remove Duplicates from List

August 15, 2022 Leave a Comment

To remove duplicates from a list in Python, the easiest way is by converting the list to a set with set() and then back to a list with list().

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = list(set(l))

print(l_duplicated_removed)

#Output:
[0,1,2,3,4,5,6,7]

If need to preserve the order of the elements in the list, then you can use comprehension.

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = []

[l_duplicates_removed.append(x) for x in l if x not in l_duplicates_removed]

print(l_duplicates_removed)

#Output:
[0, 7, 2, 3, 1, 4, 5, 6]

When working with collections of data in Python, a very common task to do is remove duplicates.

To remove duplicates from a list in Python, the easiest way is by converting the list to a set with set() and then back to a list with list().

A set is an unordered collection of unique elements. On the other hand, lists are ordered and can contain duplicates.

Converting a list to a set creates a set with the same items as the list and removes all duplicate values.

Below is how to remove duplicates from a list with the set() function in Python.

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = list(set(l))

print(l_duplicated_removed)

#Output:
[0,1,2,3,4,5,6,7]

Removing Duplicate from List and Preserving Order of Elements

If need to preserve the order of the elements in the list, then we need to do a little more work.

When converting a list to a set, the duplicates of the list are removed and also sorted.

To preserve the order of the elements in the list, you need to use a loop. To loop in Python, you can use comprehension or a standard for loop.

The process here to preserve the order of the elements of the original list is to create a second list.

We will append items to the second list if they aren’t already in the second list and this will preserve the order of the original list.

Below shows you how to remove all duplicates from a list and preserve the order of the elements with comprehension in Python.

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = []

[l_duplicates_removed.append(x) for x in l if x not in l_duplicates_removed]

print(l_duplicates_removed)

#Output:
[0, 7, 2, 3, 1, 4, 5, 6]

This is the same as the following for loop.

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = []

for x in l:
    if x not in l_duplicates_removed: 
        l_duplicates_removed.append(x)

print(l_duplicates_removed)

#Output:
[0, 7, 2, 3, 1, 4, 5, 6]

Hopefully this article has been useful for you to remove duplicates from a list in Python.

Other Articles You'll Also Like:

  • 1.  Print Time Elapsed in Python
  • 2.  PROC REG Equivalent in Python
  • 3.  Run Something Every 5 Seconds in Python
  • 4.  How to Read CSV File from AWS S3 Bucket Using Python
  • 5.  Create Unique List from List in Python
  • 6.  How to Measure Execution Time of Program in Python
  • 7.  Add Tuple to List in Python
  • 8.  Remove Every Nth Element from List in Python
  • 9.  Python Infinity – How to Use Infinity in Python
  • 10.  Cartesian Product of Two Lists in Python

About The Programming Expert

The Programming Expert is a compilation of a programmer’s findings in the world of software development, website creation, and automation of processes.

Programming allows us to create amazing applications which make our work more efficient, repeatable and accurate.

At the end of the day, we want to be able to just push a button and let the code do it’s magic.

You can read more about us on our about page.

Reader Interactions

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Primary Sidebar

About The Programming Expert

the programming expert main image

Welcome to The Programming Expert. We are a group of US-based programming professionals who have helped companies build, maintain, and improve everything from simple websites to large-scale projects.

We built The Programming Expert to help you solve your programming problems with useful coding methods and functions in various programming languages.

Search

Learn Coding from Experts on Udemy

Looking to boost your skills and learn how to become a programming expert?

Check out the links below to view Udemy courses for learning to program in the following languages:

Copyright © 2023 · The Programming Expert · About · Privacy Policy

x