JavaScript - remove duplicated lines

Hello there! 👋 😊

In this article, I would like to show you two ways to remove duplicated lines from some text in JavaScript.

We're gonna use two methods: 

• filter()
• reduce()

1. filter() based example

This approach uses a functional programming pattern.

On the text I've used set of operations to remove duplicated lines:

• split() method to split text into lines that take a newline character in 4 different variants which is a universal approach,
• filter() method that creates new array of elements tested with provided function
• join() method to merge items back.

Note:
You can read more about splitting text in this article.

Practical example:

// ONLINE-RUNNER:browser;

const newLineExpression = /\r\n|\n\r|\n|\r/g;

const removeDuplicatedLines = (text) => {
  	return text.split(newLineExpression)
  		.filter((item, index, array) => array.indexOf(item) === index)
    	.join('\n');
};

// Usage example:

const text = `a
b
b
a
a
c
c`;

console.log(removeDuplicatedLines(text)); // a
                                          // b
                                          // c

2. reduce() based example

This approach was created to show that it is possible to get the same effect as in the example above with reduce() method.

Practical example:

// ONLINE-RUNNER:browser;

const newLineExpression = /\r\n|\n\r|\n|\r/g;

const removeDuplicatedLines = (text) => {
  	const blocker = {}; // prevents lines dupplication
  	return text.split(newLineExpression)
    	.reduce((result, line) => {
        	if (blocker.hasOwnProperty(line)) {
				return result;
            }
            blocker[line] = true;
            return result + line + '\n';
        }, '');
};

// Usage example:

const text = `a
b
b
a
a
c
c`;

console.log(removeDuplicatedLines(text)); // a
                                          // b
                                          // c

References

• JavaScript - remove duplicated lines

• JavaScript - how to split string by newline?