As we enter the last quarter of 2017, TestProject’s team decided to round up the best open-source test automation frameworks out there, to help you choose the right one for you!
Here are the pros and cons of 7 different open-source test automation frameworks.
1. Robot Framework
Robot Framework (RF) is a test automation framework for acceptance testing and acceptance test-driven development (ATDD). This framework is written in Python, but can also run on Jython (Java) and IronPython (.NET), and therefore is cross-platform (Windows, Linux, or MacOS).
Pros:
- It simplifies the test automation process by utilizing the keyword-driven testing (KDT) approach, which helps testers create readable tests that are easily made.
- Has easy to use test data syntax.
- Has a rich ecosystem around it consisting of various generic test libraries and tools that are developed as separate projects.
- Has many APIs that make it highly extensible.
- Although it’s not a built-in ability, RF enables to execute parallel tests via pabot library or Selenium Grid.
Cons:
- It’s not easy to customize HTML reports.
Bottom line: This cross-platform framework is highly recommended if you are targeting for KDT automation with a wide range of libraries and extensions. If you want to add new keywords (via RF test library APIs), a basic knowledge in Java/Python/C programming language is required.
2. JUnit
JUnit is a framework for unit testing of Java applications, used to write and run repeatable tests.
Pros:
- Tests are written in pure Java which is known as the leading programming language worldwide.
- Supports test-driven development (TDD).
- Enables you to create your own unit test case suite.
- Integrates very well with other tools (for example, Maven) and with IDEs (for example, IntelliJ).
- Has history – so it has a large user base that makes it easy to find documentation on it.
Cons:
- If a mocking ability is required, one needs to add Mockito (or some other mocking library).
- Tests are not readable by non-technical people, since for example the methods names in JUnit are constrained by Java conventions.
Bottom line: If you’re looking to write unit testing for your Java application, it’s probably the best choice. However, for functional testing or non-Java applications, you should consider other solutions.
3. Spock
Spock is a testing and specification framework for Java and Groovy applications. It is based on JUnit.
Pros:
- Creates readable tests and supports plain English sentences, making it easy to read.
- Provides the surrounding context, so it easily enables you to understand how to fix a failure.
- Has built-in mocking and stubbing capabilities.
- Supports data-driven-tests (DDT).
Cons:
- Requires a basic knowledge of the Groovy programming language.
Bottom line: If your application is based on JVM and you are targeting for BDD test automation with DSL, this framework is just for you!
4. NUnit
NUnit is a unit testing framework for all .Net languages. Originally inspired by Junit, it is written entirely in C#, and has been completely redesigned to take advantage of many .NET language features.
Pros:
- Quick initiation and test execution.
- Comes with assertions and annotations.
- Enables parallel testing.
- Supports test-driven development (TDD).
Cons:
- It is not cross-platform as it is used only for .Net languages.
- It doesn’t integrate into the Visual Studio ecosystem, so using it means more maintenance.
Bottom line: A nice open-source framework for C# unit testing, with a long history and good reputation. However, in case you’re already using .NET languages, you may consider MSTest.
5. TestNG
TestNG is a test automation framework for Java that is inspired by JUnit and NUnit, but includes improved and new functionalities (NG – Next Generation). It is designed to cover all the test automation categories: unit testing, functional test, end-to-end, integration testing, etc.
Pros:
- It is easily integrated into the Maven cycle.
- Gives the developer the ability to write flexible and powerful tests.
- Supports Data Driven testing (DDT).
- Annotations are easy to understand.
- Test cases can be grouped easily.
- Allows you to create parallel tests.
Cons:
- Only supports Java, so you need to have at least a basic knowledge of the Java programming language.
- You have to invest time in framework setup & design.
Bottom line: If you use Java, are looking for end-to-end test automation framework and willing to invest some time in framework setup – you should definitely consider using TestNG.
6. Jasmine
Jasmine is a JavaScript unit testing framework. It is also known as a Behavior Driven Development (BDD) testing framework for JavaScript. It’s suited for websites, Node.js projects, or anywhere JavaScript can run. It is mainly paired with AngularJS.
Pros:
- Additionally to JavaScript, it can run in Python and Ruby, which can greatly assist you if you want to run your client-side tests alongside your server-side ones.
- Is supported by many CIs (Codeship, Travic, etc.).
- Has built-in syntax for assertions.
Cons:
- In most scenarios it requires a test runner (such as Karma).
- Has difficulties with asynchronous testing.
Bottom line: Jasmin might be a perfect fit for your needs if you’re searching for a unified (client-server) unit testing solution.
7. Mocha
Mocha is a JavaScript unit testing framework, that runs tests on Node.js. It is mainly paired with ReactJS.
Pros:
- Has its own test runner built-in.
- Supports asynchronous testing.
- Allows flexibility since you can use any assertion library (Chai, expect.js, Must.js, etc.) that fits your needs (as a replacement to Node’s standard ‘assert’ function).
Cons:
- Relatively new to the field (developed in 2012), which means it’s still changing and its user base and support might be lacking in some aspects.
- Provides just the base test structure, thus requires additional setup and configuration (may be an advantage for some).
Bottom line: If you are looking for JavaScript stand alone unit testing framework, Mocha is your go-to framework!
Source: www.sitepoint.com
