python remove duplicates from list example
Jul 26, 2025 am 07:53 AMTo remove duplicate elements in Python list, 1. Use set(): It is suitable for situations where elements are immutable and do not care about order. The syntax is list(set(original_list)), but it will disrupt the original order; 2. Use dict.fromkeys(): Recommended method, applicable to Python 3.7, which can maintain the order of the first occurrence of elements. The syntax is list(dict.fromkeys(original_list)), and the code is concise and efficient; 3. Use list derivation and seed collection: It is suitable for scenarios where deduplication logic is required. By traversing the list and recording the seen elements with the set, ensuring uniqueness; 4. When dealing with non-hashable types such as dictionaries, you can deduplicate based on specific keys (such as 'id'), manually traversing and using the seed collection to make sure that the key value is not added repeatedly. Summary: In general, it is recommended to use dict.fromkeys() to maintain order. If you do not need order, you can use set(). For complex structures, manually traversal and add seed sets to complete deduplication.
There are several ways to remove duplicate elements from Python lists, and the following are several common and practical examples.

1. Use set()
(easiest, but not keeping order)
If the elements in the list are all immutable types (such as integers, strings, tuples), and you don't care about the order of the elements , you can directly convert them to collections and then return to the list:
original_list = [1, 2, 2, 3, 4, 4, 5] unique_list = list(set(original_list)) print(unique_list) # The output may be [1, 2, 3, 4, 5] (the order may vary)
?? Note:
set
will disrupt the original order because the set is unordered.
2. Use dict.fromkeys()
(recommended, keep order)
This is the simplest way to keep the first occurrence of elements , and it works in Python 3.7 (because the dictionary guarantees the insertion order starting from 3.7):
original_list = [1, 2, 2, 3, 4, 4, 5] unique_list = list(dict.fromkeys(original_list)) print(unique_list) # Output: [1, 2, 3, 4, 5]
? Advantages: The code is concise, keeps order and is fast.

3. Use list to derive seen
sets (custom logic, suitable for complex scenarios)
If you want to have more flexibility in controlling deduplication logic (such as handling dictionary lists), you can use a secondary collection to record the seen elements:
original_list = [1, 2, 2, 3, 4, 4, 5] see = set() unique_list = [] for item in original_list: if item not in see: seen.add(item) unique_list.append(item) print(unique_list) # Output: [1, 2, 3, 4, 5]
This method is suitable for scenarios where custom "judgment repetition" logic is required.
4. Remove duplicates of non-hashable types such as dictionaries in the list
If the list is a dictionary or a list that is not hashable, you cannot use set
or dict.fromkeys()
directly. You can deduplicate based on a certain key:
data = [ {'id': 1, 'name': 'Alice'}, {'id': 2, 'name': 'Bob'}, {'id': 1, 'name': 'Alice'}, # Repeat] see = set() unique_data = [] for item in data: key = item['id'] # If key not in seed: seen.add(key) unique_data.append(item) print(unique_data) # Output: [{'id': 1, 'name': 'Alice'}, {'id': 2, 'name': 'Bob'}]
Summary: Recommended usage
- Generally (elements can be hashed, and order must be maintained)? Use
list(dict.fromkeys(lst))
- Don't care about order, pursue speed? Use
list(set(lst))
- Complex structures (such as dictionaries)? Manually traverse the
seen
collection
Basically all this is not complicated but it is easy to ignore the order issue.
The above is the detailed content of python remove duplicates from list example. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undress AI Tool
Undress images for free

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

Maven is a standard tool for Java project management and construction. The answer lies in the fact that it uses pom.xml to standardize project structure, dependency management, construction lifecycle automation and plug-in extensions; 1. Use pom.xml to define groupId, artifactId, version and dependencies; 2. Master core commands such as mvnclean, compile, test, package, install and deploy; 3. Use dependencyManagement and exclusions to manage dependency versions and conflicts; 4. Organize large applications through multi-module project structure and are managed uniformly by the parent POM; 5.

SetupaMaven/GradleprojectwithJAX-RSdependencieslikeJersey;2.CreateaRESTresourceusingannotationssuchas@Pathand@GET;3.ConfiguretheapplicationviaApplicationsubclassorweb.xml;4.AddJacksonforJSONbindingbyincludingjersey-media-json-jackson;5.DeploytoaJakar

First, use JavaScript to obtain the user system preferences and locally stored theme settings, and initialize the page theme; 1. The HTML structure contains a button to trigger topic switching; 2. CSS uses: root to define bright theme variables, .dark-mode class defines dark theme variables, and applies these variables through var(); 3. JavaScript detects prefers-color-scheme and reads localStorage to determine the initial theme; 4. Switch the dark-mode class on the html element when clicking the button, and saves the current state to localStorage; 5. All color changes are accompanied by 0.3 seconds transition animation to enhance the user

@property decorator is used to convert methods into properties to implement the reading, setting and deletion control of properties. 1. Basic usage: define read-only attributes through @property, such as area calculated based on radius and accessed directly; 2. Advanced usage: use @name.setter and @name.deleter to implement attribute assignment verification and deletion operations; 3. Practical application: perform data verification in setters, such as BankAccount to ensure that the balance is not negative; 4. Naming specification: internal variables are prefixed, property method names are consistent with attributes, and unified access control is used to improve code security and maintainability.

To generate hash values using Java, it can be implemented through the MessageDigest class. 1. Get an instance of the specified algorithm, such as MD5 or SHA-256; 2. Call the .update() method to pass in the data to be encrypted; 3. Call the .digest() method to obtain a hash byte array; 4. Convert the byte array into a hexadecimal string for reading; for inputs such as large files, read in chunks and call .update() multiple times; it is recommended to use SHA-256 instead of MD5 or SHA-1 to ensure security.

Yes, a common CSS drop-down menu can be implemented through pure HTML and CSS without JavaScript. 1. Use nested ul and li to build a menu structure; 2. Use the:hover pseudo-class to control the display and hiding of pull-down content; 3. Set position:relative for parent li, and the submenu is positioned using position:absolute; 4. The submenu defaults to display:none, which becomes display:block when hovered; 5. Multi-level pull-down can be achieved through nesting, combined with transition, and add fade-in animations, and adapted to mobile terminals with media queries. The entire solution is simple and does not require JavaScript support, which is suitable for large

Use datetime.strptime() to convert date strings into datetime object. 1. Basic usage: parse "2023-10-05" as datetime object through "%Y-%m-%d"; 2. Supports multiple formats such as "%m/%d/%Y" to parse American dates, "%d/%m/%Y" to parse British dates, "%b%d,%Y%I:%M%p" to parse time with AM/PM; 3. Use dateutil.parser.parse() to automatically infer unknown formats; 4. Use .d

Use the uuid module to obtain the MAC address of the first network card of the machine across the platform, without the need for a third-party library, and convert it into a standard format through uuid.getnode(); 2. Use subprocess to call system commands such as ipconfig or ifconfig, and combine it with regular extraction of all network card MAC addresses, which is suitable for scenarios where multiple network card information needs to be obtained; 3. Use the third-party library getmac, call get_mac_address() after installation to obtain the MAC, which supports query by interface or IP, but requires additional dependencies; in summary, if no external library is needed, the uuid method is recommended. If you need to flexibly obtain multi-network card information, you can use the subprocess solution to allow you to install the dependency getma.
