Java 8 – How to remove duplicates from ArrayList ?

In this article, we will discuss how to remove duplicate element/objects from ArrayList along with various examples

Removing duplicates from ArrayList :

  1. Using Java 8 Stream method distinct()
  2. Using Set approach
  3. Using java.util.Comparator interface
  4. Overriding equals() & hashCode() methods

Let’s discuss one-by-one in detail with example/explanation

1. Stream distinct() method

  • Stream’s distinct() method returns a stream consisting of the distinct elements (according to Object.equals(Object)) of this stream
  • Below example removes duplicate String elements and maintains original insertion order
  • Terminal operation :- Another Stream method collect() is used to collect String elements into new list
  • Similarly, sorted() method of Stream helps to sort String elements in alphabetical order once after removing duplicates

RemoveDuplicatesFromArrayList.java

package net.bench.resources.java8;

import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;

public class RemoveDuplicatesFromArrayList {

	public static void main(String[] args) {

		// create test data
		List<String> students = new ArrayList<String>();

		// add values to list
		students.add("Roger");
		students.add("Rafael");
		students.add("Djokovic");
		students.add("Roger");
		students.add("Murray");
		students.add("Rafael");


		// pretty print
		System.out.println("1. Original list with duplicate values :\n");
		students.forEach(student -> System.out.println(student));

		// Java 8 - distinct() method
		List<String> uniqueList = students
				.stream() // get stream for original list
				.distinct() // distinct method removes duplicates
				.collect(Collectors.toList()); // distinct elements stored to new list

		// pretty print
		System.out.println("\n2. New list with unique values"
				+ " maintaining original insertion order:\n");
		uniqueList.forEach(uniqueStudent -> System.out.println(uniqueStudent));

		// Java 8 - sorting
		List<String> sortedList = students
				.stream() // get stream for original list
				.distinct() // distinct method removes duplicates
				.sorted() // uses natural-ordering to sort
				.collect(Collectors.toList()); // distinct elements stored to new list

		// pretty print
		System.out.println("\n3. New list with unique values"
				+ " in natural sorting order :\n");
		sortedList.forEach(sortedStudent -> System.out.println(sortedStudent));
	}
}

Output:

1. Original list with duplicate values :

Roger
Rafael
Djokovic
Roger
Murray
Rafael

2. New list with unique values maintaining original insertion order:

Roger
Rafael
Djokovic
Murray

3. New list with unique values in natural sorting order :

Djokovic
Murray
Rafael
Roger

2. Using Set approach

  • Original list contains duplicate String elements
  • Using set approach, we can get stream for original list and then we can collect it into set using Collectors.toSet() method
  • Collecting into Set helps to remove duplicate elements but at the same time loses insertion order of original list
  • This is where Stream‘s distinct() method wins big when comparing with Set approach
  • Finally, we are sorting elements in reverse order using another Stream method called sorted() by passing Comparator.reverseOrder(); comparator

UsingSetApproach.java

package net.bench.resources.java8;

import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;

public class UsingSetApproach {

	public static void main(String[] args) {
		
		List<String> students = new ArrayList<String>();

		// add values to list
		students.add("Roger");
		students.add("Rafael");
		students.add("Djokovic");
		students.add("Roger");
		students.add("Murray");
		students.add("Rafael");


		// pretty print
		System.out.println("1. Original list with duplicate values :\n");
		students.forEach(student -> System.out.println(student));

		// Java 8 - Collector.toSet()
		Set<String> uniqueSet = students
				.stream() // get stream for original list
				.collect(Collectors.toSet()); // distinct elements stored to new SET

		// pretty print
		System.out.println("\n2. New SET with unique values"
				+ " which doesn't maintains order:\n");
		uniqueSet.forEach(uniqueStudent -> System.out.println(uniqueStudent));

		// Java 8 - sorting in reverse order
		List<String> sortedList = uniqueSet
				.stream() // get stream for unique SET
				.sorted(Comparator.reverseOrder()) // reverse-ordering
				.collect(Collectors.toList()); // elements are stored to new list

		// pretty print
		System.out.println("\n3. New list with unique values"
				+ " in REVERSE sorting order :\n");
		sortedList.forEach(sortedStudent -> System.out.println(sortedStudent));
	}
}

Output:

1. Original list with duplicate values :

Roger
Rafael
Djokovic
Roger
Murray
Rafael

2. New SET with unique values which doesn't maintains order:

Roger
Rafael
Djokovic
Murray

3. New list with unique values in REVERSE sorting order :

Roger
Rafael
Murray
Djokovic

3. Custom object – remove duplicates using Comparator

  • Student class defined with 4 attributes namely id, name, percentage, rank and constructor, getter/setters & toString() method
  • In the Student list, there are 5 Student objects and 1 duplicate Student with Id=2
  • To remove duplicate Student, we are converting original list into set which doesn’t allow duplicate by comparing Student’s Id attribute
  • Then we are printing again by comparing Student’s rank attribute in ascending order

Student.Java

package net.bench.resources.java8;

// Student class
class Student {

	// private member variables
	private int id;
	private String name;
	private double percentage;
	private int rank;

	// public 4-arg constructor

	// getters & setters
	
	// toString() method
}

TestCustomObject.java

package net.bench.resources.java8;

import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Set;
import java.util.TreeSet;
import java.util.stream.Collectors;

public class TestCustomObject {

	public static void main(String[] args) {

		// create a list of Student objects
		List<Student> studentList = new ArrayList<>();

		// add student object to List
		studentList.add(new Student(1,"Arun", 67.36, 2));
		studentList.add(new Student(2,"Sethu", 88.58, 1));
		studentList.add(new Student(3,"Ajith", 55.74, 4));
		studentList.add(new Student(4,"Vikcy", 61.32, 3));
		studentList.add(new Student(1,"Arun", 67.36, 2)); // duplicate Arun

		// pretty print
		System.out.println("1. Original Student list with duplicates :\n");
		studentList.forEach(student -> System.out.println(student));

		// Java 8 - Collector.toCollection()
		Set<Student> uniqueStudentSet = studentList
				.stream() // get stream for original list
				.collect(Collectors.toCollection(//distinct elements stored into new SET
					() -> new TreeSet<>(Comparator.comparing(Student::getId)))
						); //Id comparison

		// pretty print
		System.out.println("\n2. New SET with unique Student objects"
				+ " in natural order of Id :\n");
		uniqueStudentSet.forEach(uniqueStudent -> System.out.println(uniqueStudent));

		// Java 8 - sorting in ascending order of Student's Rank
		List<Student> sortedList = uniqueStudentSet
				.stream() // get stream for unique SET
				.sorted(Comparator.comparing(Student::getRank)) // rank comparing
				.collect(Collectors.toList()); // elements stored to new list

		// pretty print
		System.out.println("\n3. Sorted according to ascending order"
				+ " of Student's Rank :\n");
		sortedList.forEach(sortedStudent -> System.out.println(sortedStudent));
	}
}

Output:

1. Original Student list with duplicates :

Student [id=1, name=Arun, percentage=67.36, rank=2]
Student [id=2, name=Sethu, percentage=88.58, rank=1]
Student [id=3, name=Ajith, percentage=55.74, rank=4]
Student [id=4, name=Vikcy, percentage=61.32, rank=3]
Student [id=1, name=Arun, percentage=67.36, rank=2]

2. New SET with unique Student objects in natural order of Id :

Student [id=1, name=Arun, percentage=67.36, rank=2]
Student [id=2, name=Sethu, percentage=88.58, rank=1]
Student [id=3, name=Ajith, percentage=55.74, rank=4]
Student [id=4, name=Vikcy, percentage=61.32, rank=3]

3. Sorted according to ascending order of Student's Rank :

Student [id=2, name=Sethu, percentage=88.58, rank=1]
Student [id=1, name=Arun, percentage=67.36, rank=2]
Student [id=4, name=Vikcy, percentage=61.32, rank=3]
Student [id=3, name=Ajith, percentage=55.74, rank=4]

4. Custom object – remove duplicate by overriding equals/hashCode

  • Student class defined with 4 attributes namely id, name, percentage, rank and constructor, getter/setters & toString(), equals(), hashCode() method
  • Here, we are overriding equals() & hashCode() methods based on Student’s Id attribute so as to prevent storing duplicate Student object in the Set
  • In the Student list, there are 6 Student objects and 2 duplicate Student with Id=2 & Id=3
  • To remove duplicate Student, we are using Stream‘s distinct() method and then collecting into Set which stores Student object in natural sorting order of Student’s Id attribute

Student.java

package net.bench.resources.java8;

class Student {

	// private member variables
	private int id;
	private String name;
	private double percentage;
	private int rank;

	// public 4-arg constructor

	// getters & setters

	// toString() method

	// hashCode() method
	@Override
	public int hashCode() {
		final int prime = 31;
		int result = 1;
		result = prime * result + id;
		return result;
	}

	// equals() method
	@Override
	public boolean equals(Object obj) {
		if (this == obj)
			return true;
		if (obj == null)
			return false;
		if (getClass() != obj.getClass())
			return false;
		Student other = (Student) obj;
		if (id != other.id)
			return false;
		return true;
	}
}

TestStudentObject.java

package net.bench.resources.java8;

import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;

public class TestStudentObject {

	public static void main(String[] args) {

		// create a list of Student objects
		List<Student> studentList = new ArrayList<>();

		// add student object to List
		studentList.add(new Student(1,"Arun", 67.36, 2));
		studentList.add(new Student(2,"Sethu", 88.58, 1));
		studentList.add(new Student(3,"Ajith", 55.74, 4));
		studentList.add(new Student(4,"Vikcy", 61.32, 3));
		studentList.add(new Student(1,"Arun", 67.36, 2)); // duplicate Arun
		studentList.add(new Student(4,"Vikcy", 61.32, 3)); // duplicate Vicky

		// pretty print
		System.out.println("1. Original Student list with duplicates :\n");
		studentList.forEach(student -> System.out.println(student));

		// Java 8 - Collector.toCollection()
		Set<Student> uniqueStudentSet = studentList
				.stream() // get stream for original list
				.distinct() // removes duplicate
				.collect(Collectors.toSet()); 

		// pretty print
		System.out.println("\n2. New SET with unique Student objects"
				+ " in natural order of Id :\n");
		uniqueStudentSet.forEach(uniqueStudent -> System.out.println(uniqueStudent));
	}
}

Output:

1. Original Student list with duplicates :

Student [id=1, name=Arun, percentage=67.36, rank=2]
Student [id=2, name=Sethu, percentage=88.58, rank=1]
Student [id=3, name=Ajith, percentage=55.74, rank=4]
Student [id=4, name=Vikcy, percentage=61.32, rank=3]
Student [id=1, name=Arun, percentage=67.36, rank=2]
Student [id=4, name=Vikcy, percentage=61.32, rank=3]

2. New SET with unique Student objects in natural order of Id :

Student [id=1, name=Arun, percentage=67.36, rank=2]
Student [id=2, name=Sethu, percentage=88.58, rank=1]
Student [id=3, name=Ajith, percentage=55.74, rank=4]
Student [id=4, name=Vikcy, percentage=61.32, rank=3]

Related Articles:

References:

Happy Coding !!
Happy Learning !!

Java 8 - Stream forEachOrdered() method
Java 8 - Stream forEach() method with examples