Using Python to Remove Duplicates from List

To remove duplicates from a list in Python, the easiest way is by converting the list to a set with set() and then back to a list with list().

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = list(set(l))

print(l_duplicated_removed)

#Output:
[0,1,2,3,4,5,6,7]

If need to preserve the order of the elements in the list, then you can use comprehension.

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = []

[l_duplicates_removed.append(x) for x in l if x not in l_duplicates_removed]

print(l_duplicates_removed)

#Output:
[0, 7, 2, 3, 1, 4, 5, 6]

When working with collections of data in Python, a very common task to do is remove duplicates.

To remove duplicates from a list in Python, the easiest way is by converting the list to a set with set() and then back to a list with list().

A set is an unordered collection of unique elements. On the other hand, lists are ordered and can contain duplicates.

Converting a list to a set creates a set with the same items as the list and removes all duplicate values.

Below is how to remove duplicates from a list with the set() function in Python.

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = list(set(l))

print(l_duplicated_removed)

#Output:
[0,1,2,3,4,5,6,7]

Removing Duplicate from List and Preserving Order of Elements

If need to preserve the order of the elements in the list, then we need to do a little more work.

When converting a list to a set, the duplicates of the list are removed and also sorted.

To preserve the order of the elements in the list, you need to use a loop. To loop in Python, you can use comprehension or a standard for loop.

The process here to preserve the order of the elements of the original list is to create a second list.

We will append items to the second list if they aren’t already in the second list and this will preserve the order of the original list.

Below shows you how to remove all duplicates from a list and preserve the order of the elements with comprehension in Python.

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = []

[l_duplicates_removed.append(x) for x in l if x not in l_duplicates_removed]

print(l_duplicates_removed)

#Output:
[0, 7, 2, 3, 1, 4, 5, 6]

This is the same as the following for loop.

l = [0,7,7,7,0,2,3,1,1,4,5,6,7]

l_duplicates_removed = []

for x in l:
    if x not in l_duplicates_removed: 
        l_duplicates_removed.append(x)

print(l_duplicates_removed)

#Output:
[0, 7, 2, 3, 1, 4, 5, 6]

Hopefully this article has been useful for you to remove duplicates from a list in Python.