Specflow Support In Visual Studio For Mac

This section guides you through the process of installing SpecFlow and SpecFlow+ Runner¹ and setting up a simple project in Visual Studio. The getting started guide for SpecFlow+ Excel can be found here.

Can I use my existing Visual Studio license on a Mac? Support Policies (1) What are the support policies for Visual Studio for Mac? Support links. Installation Help. Live Chat English only Mon – Fri, excluding holidays. Installation chat. Getting Started, references, user manuals. Navigating to SpecFlow Features from Test Runner Follow. It is possible to run NUnit based SpecFlow Feature tests from either Visual Studio Test Explorer, or from the RS Unit Test Runner. If using the VS Test Explorer, if you double-click on an individual test, it takes you to the relevant scenario in the Feature file.

The sample project is intentionally kept very simple and consists of a basic “calculator” that adds two numbers and returns the result. You will define a test scenario to automate the testing of the calculator’s code.

This guide contains the following sections:

If you are new to BDD/ATDD/Specification by Example, we also suggest reading up on best practices for writing your tests. This is not covered in this guide; you can find a short article on how to write your specifications in Gherkin here.

^{1. Instead of SpecFlow+ Runner, you can also use other test engines, like MsTest, xUnit, or NUnit. These packages are installed in exactly the same manner as SpecFlow+ Runner. However to follow all the steps in this guide, you need to install SpecFlow+ Runner. The unlicensed version of SpecFlow+ Runner delays the execution of your tests by a number of seconds each time. You can request a free license key to remove this restriction (please include your name and company name, as this information is included in the license).}

Installation and Setup

Installing SpecFlow consists of two steps:

1. Install the Visual Studio Extension
2. Set up your Visual Studio project to work with SpecFlow

Installing the Visual Studio Extension

The SpecFlow extension for Visual Studio provides a number of helpful features, such as syntax highlighting for Gherkin (feature) files. This extension is not required to use SpecFlow, but we recommend you install it if you are using Visual Studio. If you are not using Visual Studio on Windows, you will not be able to install the extension.

To install the extension, download the extension for your version of Visual Studio:

Either choose to open the download directly, or double-click the extension once it has downloaded to install it in Visual Studio.

Setting Up your SpecFlow Project

This section guides you through the first steps of setting up a Visual Studio project with SpecFlow and defining and executing your first test scenario. In this example, we will be using SpecFlow+ Runner, but you can use a number of other test execution frameworks, including NUnit, Visual Studio Test Window and Team Foundation Server (TFS) Build integration for SpecFlow. SpecFlow+ Runner’s advantages include integration with Visual Studio Test Runner and extensive integrated reports available from within Visual Studio.

SpecFlow+ Runner is available free of charge. You can request your license here. Don’t forget to register your license!

SpecFlow tests are usually placed into one or more separate projects in your solution, and these projects are referred to as a “specification project” below. The easiest and most convenient way to set up these projects is to use our SpecFlow NuGet package or one of the specific helper packages, like SpecRun.SpecFlow or SpecFlow.NUnit.

To set up your specification project:

1. Add an “MSTest Test Project (.NET Core)” to your solution (e.g. “MyProject.Specs”).
   Note: Adding this simplifies the setup, as we will be using the .NET Core framework for this project. For a Full Framework project, select “Unit Test Project (.NET Framework)” instead.
2. Remove the UnitTest1.cs file, as it is not required.
3. Right-click on your solution (e.g. “MyProject.Specs”) and select Manage NuGet Packages for Solution.
4. Install the following packages (use the search field to filter the search results):
   • SpecFlow
   • SpecFlow.Tools.MsBuild.Generation
   • SpecRun.SpecFlow
5. Microsoft .NET Test SDK 15 is also required. If you have not installed this package already, please install it as well.

SpecRun.Runner Package

This package is added to your project automatically when you install SpecRun.SpecFlow. This package configures SpecFlow+ Runner as your unit test provider.

Note: Instead of SpecFlow+ Runner, you can also use other test engines, like MsTest, xUnit or NUnit. These packages are installed in exactly the same manner as SpecFlow+ Runner. However to follow all the steps in this guide, you need to install SpecFlow+ Runner. The evaluation version of SpecFlow+ Runner delays the execution of your tests by a number of seconds each time. You can request a free license key to remove this restriction (please include your name and company name).

SpecFlow.Tools.MsBuild.Generation

In order to run your tests, SpecFlow needs to generate so-called “code-behind” files. The SpecFlow.Tools.MsBuild.Generation NuGet package is responsible for generating these files whenever you build your solution.

Adding a Feature File

You now need to add a feature file to your specifications project that outlines a specific feature and includes a test scenario:

1. 1. Right-click on your specifications project and select Add New Item from the popup menu.
   2. Select SpecFlow Feature File (restrict the entries by entering “SpecFlow” in the search field), give it a meaningful name (e.g. “Calculator.feature”) and click on Add.
      Note: Do not choose one of the feature files with a language in the name, as these files do not contain the skeleton code we will be using.

The feature file is added to your specification project. It includes a default scenario for adding two numbers. You can leave most of the content as is, but add the word “also” to the scenario on the line beginning with “And” (see red text below):

We will use this scenario to demonstrate the first development iteration. By adding “also” to the second Given statement, we have two separate statements that will be parsed differently (otherwise both statements would match the regular expression “I have entered (.*) into the calculator”). This both simplifies the code in our example and illustrates how different statements linked with “And” are handled.

You can also assign tags to scenarios (e.g. “@mytag” in the example above), which can be used to filter scenarios, and control how scenarios are executed and automated. For more details on using tags, see Hooks, Scoped Bindings, FeatureContext and ScenarioContext in the documentation.

Generating Step Definitions

In order to test our scenario, we need to create step definitions that bind the statements in the test scenario to the application code. SpecFlow can automatically generate a skeleton for the automation code that you can then extend as necessary:

1. Right-click on your feature file in the code editor and select Generate Step Definitions from the popup menu. A dialogue is displayed.
2. Enter a name for the class, e.g. “CalculatorSteps”.
   
3. Click on Generate and save the file. A new skeleton class is added to your project with steps for each of the steps in the scenario:
   

Executing Your First Test

SpecFlow+ Runner integrates with Visual Studio Test Explorer. After adding your first specification and building the solution, the business readable scenario titles will show up in Visual Studio Test Explorer:

1. Build your solution.
2. Select Test Windows Test Explorer to open the Test Explorer:
   Scenarios are displayed with their plain text scenario title instead of a generated unit test name.
3. Click on Run All to run your test. As the automation and application code has not yet been implemented, the test will not pass successfully.
   Note: The evaluation version of SpecFlow+ Runner delays the execution of your tests by a number of seconds each time. You can request a free license key to remove this restriction (please include your name and company name).

Note: If you cannot see your tests, make sure there are no spaces or dashes in your project name!

Implementing the Automation and Application Code

In order for your tests to pass, you need to implement both the application code (the code in your application you are testing) and the automation code (binding the test scenario to the automation interface). This involves the following steps, which are covered in this section:

1. Reference the assembly or project containing the interface you want to bind the automation to (including APIs, controllers, UI automation tools etc.).
2. Extend the step definition skeleton with the automation code.
3. Implement the missing application code.
4. Verify that the scenario passes the test.

Adding a Calculator Class

The application code that implements the actual functions performed by the calculator should be defined in a separate project from your specification project. This project should include a class for the calculator and expose methods for initialising the calculator and performing the addition:

1. Right-click on your solution in the Solution Explorer and select Add Project from the context menu. Choose to add a new class library and give your project a name (e.g. “Example”).
2. Right-click on the .cs file in the new project and rename it (e.g. “Calculator.cs”), and choose to rename all references.
3. Your new class should be similar to the following:
   

Referencing the Calculator Class

1. Right-click your specification project and select Add Reference from the context menu.
2. Click on Projects on the left of the Reference Manager dialogue. The projects in your solution are listed.
3. Enable the check box next to the Example project to reference it from the specifications project.
4. Click on OK.
   A reference to the Example project is added to the References node in the Solution Explorer.
5. Add a using directive for the namespace (e.g. “Example”) of your Calculator class to the CalculatorSteps.cs file in your specification project:
   
6. Define a variable of the type Calculator in the CalculatorSteps class prior to the step definitions:
   

   Defining a variable outside of the individual steps allows the variable to be accessed by each of the individual steps and ensures the variable is persistent between steps.

Implementing the Code

Now that the step definitions can reference the Calculator class, you need to extend the step definitions and implement the application code.

Binding the First Given Statement

The first Given statement in the scenario needs to initialise the calculator with the first of the two numbers defined in the scenario (50). To implement the code:

1. Open CalculatorSteps.cs if it is not already open.
   The value defined in the scenario is passed as a parameter in the automation code’s associated function, e.g.:
   
2. Rename this parameter to something more human-readable (e.g. “number”):
   
3. To initialise the calculator with this number, replace ScenarioContext.Current.Pending(); in the step definition as follows:
   
4. Switch to the file containing your Calculator class (e.g. Calculator.cs) and add a public integer member to the class:
   

You have now determined that the FirstNumber member of the Calculator class is initialised with the value defined in the scenario when the test is executed.

Binding the Second Given Statement

The second Given statement in the scenario needs to initialise the second number with the second value defined in the scenario (70). To implement the code:

1. Open CalculatorSteps.cs if it is not already open.
2. Locate the function corresponding to the second Given statement and rename the p0 parameter to “number”, as before.
3. To initialise the calculator with the second number, replace ScenarioContext.Current.Pending(); in the step definition as follows:
   
4. Switch to the file containing your Calculator class and add another public integer member to the class:
   

You have now determined that the SecondNumber member of the Calculator class is initialised with the value defined in the scenario when the test is executed.

Binding the When Statement

The step for the When statement needs to call the method that performs the actual addition and store the result. This result needs to be available to the other final step in the automation code in order to verify that the result is the expected result defined in the test scenario.

To implement the code:

1. Open CalculatorSteps.cs if it is not already open.
2. Define a variable to store the result at the start of the CalculatorSeps class (before any of the steps):
   

   Defining a variable outside of the individual steps allows the variable to be accessed by each of the individual steps.

3. Locate the function corresponding to the When statement and edit it as follows:
   
4. Switch to the file containing your Calculator class and define the Add() method:
   

You have now determined that the Add() method of the calculator class is called once the initial Given steps have been performed.

Binding the Then Statement

The step for the Then statement needs to verify that the result returned by the Add() method in the previous step is the same as the expected result defined in the test scenario. To implement the code:

1. Open CalculatorSteps.cs if it is not already open.
   As the result will be verified using Assert, you need to add “using Microsoft.VisualStudio.TestTools.UnitTesting;” to the top of your automation code.
2. Locate the function corresponding to the Then statement. Rename the p0 parameter in the function call (this time to “expectedResult”) and edit the step definition as follows:

You have now implemented the final piece of the jigsaw – testing that the result returned by your application matches the expected result defined in the scenario.

Final CalculatorSteps.cs Code

Your CalculatorSteps.cs code should be similar to the following:

Final Calculator.cs Code

Your Calculator.cs code should be similar to the following:

Executing the Tests Again

Now that the test steps have been bound to your application code, you need to rebuild your solution and execute the tests again (click on Run All in the Test Explorer). You should see that the test now passes (green).

Click on Output in the Test Explorer to display a summary of the test:
This example is obviously very simple; at this point you would want to refactor your code before proceeding with the implementation of your remaining scenarios.

Sample Projects

A number of SpecFlow+ Runner sample projects, including the project covered in this tutorial (Getting Started), can be found here. You can find samples using other test unit providers here.

Living Documentation (Optional)

An easy way to share your feature files with all members of your team is to use the SpecFlow+ LivingDoc VSTS extension to generate living documentation from your feature files. This documentation can then be accessed from within DevOps/TFS. Your Gherkin tests constitute an important part of the documentation of your system, describing the intended behaviour in human-readable form and providing the basis for discussion with other project stakeholders. Many stakeholders will not be able to access your feature files directly in your repository, nor will they be using Visual Studio, so sharing them on a common platform is a big advantage.

Generating documentation involves two steps:

1. Define a build process. This build process needs to include the SpecFlow+ build step (included when installing the extension).
2. Queue the build. Once it is complete, select Test SpecFlow+ from the menu in DevOps/TFS to access the living documentation.

The generated documentation includes syntax highlighting for Gherkin keywords, tables and the ability to preview scenarios using example data.

Additional SpecFlow+ Runner Features

SpecFlow+ Runner includes a number of features that make working in Visual Studio easier.

Grouping Tests by Category

When using SpecFlow+ Runner to execute your tests, you can group scenarios according to various attributes:

• Select Group By Traits to group the SpecFlow scenarios by the tags defined at the scenario or feature level:
  
• Select Group By Class to group the SpecFlow scenarios by feature (with the feature title in plain text:
  

SpecFlow Reports

SpecFlow+ Runner generates an advanced execution report for each test run. To view the report, select Tests in the Show output from field in the Output window:

Click on the link to the report file to view the report in Visual Studio:

You can configure the name of the report file in the SpecFlow+ Runner settings (ReportFile element in the .runsettings file). More details are available here.

The report provides a convenient side-by-side view of the individual scenario steps and the trace output for each step:

Step definitions can write to the trace output to provide a lower level technical representation of the executed step (e.g. a more verbose representation of data structures that are processed in a step).

Running SpecFlow Tests from the Command Line

You can also run runtests.cmd from the command prompt to execute your specification tests.

Additional SpecFlow+ Runner Features

SpecFlow+ Runner offers several additional features that improve SpecFlow’s integration with Visual Studio, such as:

• Parallel test execution
• Detect flickering (intermittently failing) scenarios
• Smart test execution order (run failed tests first)
• Execute tests in different runtime environments
• Automatic deployment configuration
• SpecFlow integration with Team Foundation Server

SpecFlow+ requires you a license to use. You can request a free license key here.

Alternatively, you can instead use other test engines, like MsTest, xUnit or NUnit. These packages are installed in exactly the same manner as SpecFlow+ Runner, but do not require a license. However, they also lack the extended features found in SpecFlow+ Runner.

Prerequisites

You can install SpecFlow via the NuGet Gallery

• Create a new project in Visual Studio.
• In the Visual Studio Tools menu, go to Library Package Manager > Manage Nuget Package for Solution.
• This will open the Manage NuGet Packages dialog. Click Online, then Next.
• In the Search Packages field, enter SpecFlow and click Search.
• Select SpecFlow from the search results and click Install.

For more details, please visit SpecFlow installation Document.




Sample Test

This is a sample test case in SpecFlow:




Now you need to create the TestingBotDriver we will be using:




Now we need to create the config file which the code above uses to determine various settings.

Configuring capabilities

To let TestingBot know on which browser/platform you want to run your test on, you need to specify the browsername, version, OS and other optional options in the capabilities field.

To see how to do this please select a combination of browser version and platform in the drop-down menus below

Testing Internal/Staged Websites

We've built TestingBot Tunnel, to provide you with a secure way to run tests against your staged/internal webapps.
Please see our TestingBot Tunnel documentation for more information about this easy to use tunneling solution.

The example below shows how to easily run a Java WebDriver test with our Tunnel:

1. Download our tunnel and start the tunnel:

2. Adjust your test: instead of pointing to 'hub.testingbot.com/wd/hub' like the example above - change it to point to your tunnel's IP address.
Assuming you run the tunnel on the same machine you run your tests, change to 'localhost:4445/wd/hub'. localhost is the machine running the tunnel, 4445 is the default port of the tunnel.

This way your test will go securely through the tunnel to TestingBot and back:

Other Options

We offer many other test options, for example: disable video recording, specifying a custom Firefox Profile, loading Chrome/Firefox/Safari extensions, running an executable before your test starts, uploading files, ..
See our list of test options for a full list of options to customize your tests.

Parallel Testing

Parallel Testing means running the same test, or multiple tests, simultaneously. This greatly reduces your total testing time.

You can run the same tests on all different browser configurations or run different tests all on the same browser configuration.
TestingBot has a large grid of machines and browsers, which means you can use our service to do efficient parallel testing. It is one of the key features we provide to greatly cut down on your total testing time.

Parallel Example

To run multiple tests at the same time with SpecFlow, modify the config file like this:

The test cases you want to run in parallel need to be written like this:

You can now run the tests in parallel by running the tests with fixture parallel from Test Explorer.

Queueing

Every plan we provide comes with a limit of concurrent VMs (how many tests you can run in parallel).
For example: if you have a plan with 5 concurrent VMs, it is possible to start more tests. TestingBot will queue the additional tests and run the tests as soon as slots become available.

Pick a C# Test Framework

• NUnit

  An unit testing framework that is open source written in C#.

• PNunit

  With PNUnit you can run several tests in parallel.

• SpecFlow

  SpecFlow allows you to run Automated .NET tests using Cucumber-compatible Gherkin syntax.