As we enter the last quarter of 2017, TestProject’s team decided to round up the best open-source test automation frameworks out there, to help you choose the right one for you!

Here are the pros and cons of 7 different open-source test automation frameworks.

1. Robot Framework

Robot Framework (RF) is a test automation framework for acceptance testing and acceptance test-driven development (ATDD). This framework is written in Python, but can also run on Jython (Java) and IronPython (.NET), and therefore is cross-platform (Windows, Linux, or MacOS).


  • It simplifies the test automation process by utilizing the keyword-driven testing (KDT) approach, which helps testers create readable tests that are easily made.
  • Has easy to use test data syntax.
  • Has a rich ecosystem around it consisting of various generic test libraries and tools that are developed as separate projects.
  • Has many APIs that make it highly extensible.
  • Although it’s not a built-in ability, RF enables to execute parallel tests via pabot library or Selenium Grid.


  • It’s not easy to customize HTML reports.

Bottom line: This cross-platform framework is highly recommended if you are targeting for KDT automation with a wide range of libraries and extensions. If you want to add new keywords (via RF test library APIs), a basic knowledge in Java/Python/C programming language is required.

2. JUnit

JUnit is a framework for unit testing of Java applications, used to write and run repeatable tests.


  • Tests are written in pure Java which is known as the leading programming language worldwide.
  • Supports test-driven development (TDD).
  • Enables you to create your own unit test case suite.
  • Integrates very well with other tools (for example, Maven) and with IDEs (for example, IntelliJ).
  • Has history – so it has a large user base that makes it easy to find documentation on it.


  • If a mocking ability is required, one needs to add Mockito (or some other mocking library).
  • Tests are not readable by non-technical people, since for example the methods names in JUnit are constrained by Java conventions.

Bottom line: If you’re looking to write unit testing for your Java application, it’s probably the best choice. However, for functional testing or non-Java applications, you should consider other solutions.

3. Spock

Spock is a testing and specification framework for Java and Groovy applications. It is based on JUnit.


  • Creates readable tests and supports plain English sentences, making it easy to read.
  • Provides the surrounding context, so it easily enables you to understand how to fix a failure.
  • Has built-in mocking and stubbing capabilities.
  • Supports data-driven-tests (DDT).


  • Requires a basic knowledge of the Groovy programming language.

Bottom line: If your application is based on JVM and you are targeting for BDD test automation with DSL, this framework is just for you!

4. NUnit

NUnit is a unit testing framework for all .Net languages. Originally inspired by Junit, it is written entirely in C#, and has been completely redesigned to take advantage of many .NET language features.


  • Quick initiation and test execution.
  • Comes with assertions and annotations.
  • Enables parallel testing.
  • Supports test-driven development (TDD).


  • It is not cross-platform as it is used only for .Net languages.
  • It doesn’t integrate into the Visual Studio ecosystem, so using it means more maintenance.

Bottom line: A nice open-source framework for C# unit testing, with a long history and good reputation. However, in case you’re already using .NET languages, you may consider MSTest.

5. TestNG

TestNG is a test automation framework for Java that is inspired by JUnit and NUnit, but includes improved and new functionalities (NG – Next Generation). It is designed to cover all the test automation categories: unit testing, functional test, end-to-end, integration testing, etc.


  • It is easily integrated into the Maven cycle.
  • Gives the developer the ability to write flexible and powerful tests.
  • Supports Data Driven testing (DDT).
  • Annotations are easy to understand.
  • Test cases can be grouped easily.
  • Allows you to create parallel tests.


  • Only supports Java, so you need to have at least a basic knowledge of the Java programming language.
  • You have to invest time in framework setup & design.

Bottom line: If you use Java, are looking for end-to-end test automation framework and willing to invest some time in framework setup – you should definitely consider using TestNG.

6. Jasmine

Jasmine is a JavaScript unit testing framework. It is also known as a Behavior Driven Development (BDD) testing framework for JavaScript. It’s suited for websites, Node.js projects, or anywhere JavaScript can run. It is mainly paired with AngularJS.


  • Additionally to JavaScript, it can run in Python and Ruby, which can greatly assist you if you want to run your client-side tests alongside your server-side ones.
  • Is supported by many CIs (Codeship, Travic, etc.).
  • Has built-in syntax for assertions.


  • In most scenarios it requires a test runner (such as Karma).
  • Has difficulties with asynchronous testing.

Bottom line: Jasmin might be a perfect fit for your needs if you’re searching for a unified (client-server) unit testing solution.

7. Mocha

Mocha is a JavaScript unit testing framework, that runs tests on Node.js. It is mainly paired with ReactJS.


  • Has its own test runner built-in.
  • Supports asynchronous testing.
  • Allows flexibility since you can use any assertion library (Chai, expect.js, Must.js, etc.) that fits your needs (as a replacement to Node’s standard ‘assert’ function).


  • Relatively new to the field (developed in 2012), which means it’s still changing and its user base and support might be lacking in some aspects.
  • Provides just the base test structure, thus requires additional setup and configuration (may be an advantage for some).

Bottom line: If you are looking for JavaScript stand alone unit testing framework, Mocha is your go-to framework!