Python - Remove Duplicates from a List

Given a list that contains repeated values, the goal is to extract only the unique elements. For Example:

Input: [10, 20, 10, 30, 20, 40]
Output: [10, 20, 30, 40]

Let's explore different methods to remove duplicates from a list in Python.

Using set()

This method converts the list into a set using set(), which automatically removes duplicate values because sets store only unique items. However, since sets are unordered, the original order is not maintained.

a = [1, 2, 2, 3, 4, 4, 5]
u = list(set(a))
print(u)

[1, 2, 3, 4, 5]

Explanation:

• set(a) removes duplicates by storing each element once.
• list(set(a)) converts the set back into a list.

Using dict.fromkeys()

dict.fromkeys() builds a dictionary where list elements become keys. Since keys cannot repeat, duplicates are removed while keeping the first occurrence.

a = [1, 2, 2, 3, 4, 4, 5]
u = list(dict.fromkeys(a))
print(u)

[1, 2, 3, 4, 5]

Explanation:

• dict.fromkeys(a) creates keys for each element, automatically removing duplicates.
• list(...) extracts the ordered unique values.

Using List Comprehension + Set

This method tracks seen items using a set while building a new list. It removes duplicates and preserves order efficiently.

a = [1, 2, 2, 3, 4, 4, 5]
s = set()
u = [x for x in a if not (x in s or s.add(x))]
print(u)

[1, 2, 3, 4, 5]

Explanation:

• 's' stores already-seen elements.
• 'x' in 's' checks if an element appeared earlier.
• s.add(x) adds the element to s at its first encounter.
• The list comprehension builds an ordered list of unique items.

Using a Loop

This method checks each element and adds it only if it hasn't appeared earlier. It preserves order but is slower due to repeated membership checks.

a = [1, 2, 2, 3, 4, 4, 5]
u = []
for x in a:
    if x not in u:
        u.append(x)
print(u)

[1, 2, 3, 4, 5]

Explanation:

• 'x' not in 'u' ensures no repeated element is added.
• u.append(x) stores unique items in order of appearance.