Device Cloud Testing using Appium and SmartBear BitBar

Learn how to scale automated tests for mobile games and apps created in Unity using the SmartBear BitBar platform

This article will cover how to combine mobile test execution with running on real devices hosted by the SmartBear BitBar cloud service. For information on how to configure your Android builds for working with GameDriver see this article, and for working with iOS see this article. If following these steps doesn’t get you where you need to be, please let us know by opening a support ticket or emailing the steps taken and results to support@gamedriver.io.

Pre-reading

Read Unity’s documentation for working with Android here.
You will also need to build your Android project with the GameDriver agent. For more information regarding the installation and use of GameDriver, please refer to the installation instructions here.
For a great primer on setting up Appium for testing locally (macOS-focused), read this article.
For testing using a real device cloud, read BitBar’s Appium Tips for using C# here.
Read the steps for running server-side Appium tests with BitBar here.

Initial Setup

For this example, we used the Tennis Mobile game by Codeer Studio. This game can be purchased on the Unity Asset Store here. The test script used can be downloaded at the bottom of this article, along with a sample zip file containing the NUnit console and `run_script.sh` example you can modify for your tests.

The following packages.config settings were used for our testing. Note that the Selenium and Appium packages are version-sensitive when working with BitBar, and should not be updated to the latest version.

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Appium.WebDriver" version="3.0.0.2" targetFramework="net472" />
  <package id="Castle.Core" version="3.3.3" targetFramework="net472" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net472" />
  <package id="NUnit" version="3.13.2" targetFramework="net472" />
  <package id="NUnit.Console" version="3.12.0" targetFramework="net472" />
  <package id="NUnit.ConsoleRunner" version="3.12.0" targetFramework="net472" />
  <package id="NUnit.Extension.NUnitProjectLoader" version="3.6.0" targetFramework="net472" />
  <package id="NUnit.Extension.NUnitV2Driver" version="3.8.0" targetFramework="net472" />
  <package id="NUnit.Extension.NUnitV2ResultWriter" version="3.6.0" targetFramework="net472" />
  <package id="NUnit.Extension.TeamCityEventListener" version="1.0.7" targetFramework="net472" />
  <package id="NUnit.Extension.VSProjectLoader" version="3.8.0" targetFramework="net472" />
  <package id="NUnit3TestAdapter" version="3.17.0" targetFramework="net472" />
  <package id="NUnitTestAdapter" version="2.3.0" targetFramework="net472" />
  <package id="Selenium.Support" version="3.11.2" targetFramework="net472" />
  <package id="Selenium.WebDriver" version="3.11.2" targetFramework="net472" />
</packages>

Run > nuget install Test123/packages.config -OutputDirectory packages after updating packages.config to ensure your packages are installed/updated as needed.

Next, update the test to accept external parameters, to differentiate between test modes. This is required since GameDriver’s ApiClient class supports multiple modes, and needs to know whether to connect to the Unity editor or a standalone device. For example:

public string testMode = TestContext.Parameters.Get("Mode", "IDE");

This will accept a new parameter from the NUnit Console Runner, with the default value “IDE”. We can then override the parameter when running the test with the following format:

mono ./ext/nunit3-console.exe --testparam:Mode=standalone

To extend this concept, we then add several options for the test mode, test host, and OS we want to test. Note that for proper scoping, you will need to create the following outside of any test classes including your [OneTimeSetup]. This will also create the ApiClient and AppiumDriver needed throughout the test:

ApiClient api;
AppiumDriver<IWebElement> driver;

//Some test parameters and their overrides
static string mode = "IDE";
//static string mode = "standalone";
//static string mode = "appium";

static string host = "localhost";
//static string host = "192.168.68.78";

static string OS = "Android";
//static string OS = "iOS";

public string testMode = TestContext.Parameters.Get("Mode", mode);
public string testHost = TestContext.Parameters.Get("Host", host);
public string testOS = TestContext.Parameters.Get("OS", OS);

Note: The commented lines above are provided as examples.

We then need to add the DesiredCapabilities for our specific mobile devices. Be sure to read the documentation regarding Appium DesiredCapabilities for BitBar here.

For our testing, we used the following DesiredCapabilities for Android:

public static DesiredCapabilities appiumAndroidCapabilities()
{
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.SetCapability("platformName", "Android");
    capabilities.SetCapability("app", "./MobileTennis.apk");
    capabilities.SetCapability("appPackage", "com.GameDriver.MobileTennis");
    capabilities.SetCapability("appActivity", "com.unity3d.player.UnityPlayerActivity");
    capabilities.SetCapability("appium:automationName", "UiAutomator2");
    capabilities.SetCapability("newCommandTimeout", 600);

    return capabilities;
}

...and the following for iOS:

public static DesiredCapabilities appiumIOSCapabilities()
{
    DesiredCapabilities capabilities = new DesiredCapabilities();
    capabilities.SetCapability("os_version", "16");
    capabilities.SetCapability("device", "iPhone 14");
    capabilities.SetCapability("app", "./MobileTennis.ipa");
    capabilities.SetCapability("appium:automationName", "XCUITest");
    capabilities.SetCapability("newCommandTimeout", 600);

    return capabilities;
}

Note that both of these can be stored in a separate class in order to keep the test code relatively clean. In our example, we're using a class called Configs to store these capabilities.

Be sure to update appPackage as defined in the Unity Project Settings. It should be formatted as com.CompanyName.ProductName as shown above. We will also need to add the following code and call it during the [OneTimeSetup] section. This allows us to forward communications intended for the GameDriver agent to the app running on the device, via the Android debug bridge (ADB) for Android:

string arguments = "forward tcp:19734 tcp:19734";
var process = new Process();
var startInfo = new ProcessStartInfo
{
    CreateNoWindow = true,
    WindowStyle = ProcessWindowStyle.Minimized,
    UseShellExecute = false,
    RedirectStandardOutput = true,
    RedirectStandardError = true,
    FileName = "adb",
    Arguments = arguments
};

process.StartInfo = startInfo;
Console.WriteLine($"Waiting {DateTime.Now.ToString()}");
Thread.Sleep(15000);
Console.WriteLine($"Attempting Port Forward... {DateTime.Now.ToString()}");
process.Start();

...and using the iProxy service for iOS:

string arguments = $"-u {Environment.GetEnvironmentVariable("APPIUM_UDID")} 19734:19734";
var process = new Process();
var startInfo = new ProcessStartInfo
{
    FileName = "iproxy",
    Arguments = arguments
};
process.StartInfo = startInfo;
Console.WriteLine($"Waiting {DateTime.Now.ToString()}");
Thread.Sleep(15000);
Console.WriteLine($"Attempting Port Forward... {DateTime.Now.ToString()}");
process.Start();
// Stabilization
Thread.Sleep(15000);
string stdout = process.StandardOutput.ReadToEnd();
process.WaitForExit();

Both of these pieces can be moved into helper methods, and called during [OneTimeSetup], along with the DesiredCapabilities we covered above. For example, our final [OneTimeSetup] might look like this:

[OneTimeSetUp]
public void Connect()
{
 api = new ApiClient();

 try
 {
     if (testMode == "appium")
     {
        Console.WriteLine("WebDriver request initiated. Waiting for response, this typically takes 2-3 mins");

            if (testOS == "iOS")
            {
                driver = new IOSDriver<IWebElement>(Configs.appiumUri(), Configs.appiumIOSCapabilities(), TimeSpan.FromSeconds(300));
                SetUpPortForwardingIos();
            }
            else
            {
                driver = new AndroidDriver<IWebElement>(Configs.appiumUri(), Configs.appiumAndroidCapabilities(), TimeSpan.FromSeconds(300));
                Console.WriteLine("WebDriver response received.");
                Console.WriteLine($"Installing app: {Environment.GetEnvironmentVariable("APPIUM_APPFILE")}");
                driver.LaunchApp();
                SetUpADBForwardingAndroid();
            }
    }

    Console.WriteLine($"Waiting {DateTime.Now}");

    if (testMode == "IDE")
    {
            api.Connect(testHost, 19734, true, 30);
    }
    else api.Connect(testHost, 19734, false, 30);

}
catch (Exception e)
{
    Console.WriteLine(e.ToString());
}

api.UnityLoggedMessage += (s, e) =>
{
    Console.WriteLine($"Type: {e.type.ToString()}\r\nCondition: {e.condition}\r\nStackTrace: {e.stackTrace}" );
};

api.Wait(3000);
api.EnableHooks(HookingObject.MOUSE);
api.Wait(3000);
}

Running tests using SmartBear BitBar

In order to run our tests on the BitBar platform, we will need to package them up for server-side execution as described in the BitBar documentation here.

After running your tests against the Unity editor to verify they are functioning as expected, copy the <Test>/bin/Debug directory contents of your test into your Appium <Test>/tests folder. For example:

Copy the <Test>/packages/NUnit.ConsoleRunner.3.12.0/tools directory contents to your Appium <Test>/ext folder.

Create the run_tests.sh as outlined in the BitBar instructions above. For our testing with Android, the following was used:

#!/bin/bash
unzip tests.zip # BitBar zips up the tests folder on upload, so this needs to be unzipped

## Test-specific variables
export APPIUM_APPFILE=$PWD/GameDriver_Demo.apk #Replace "GameDriver_Demo" with your Apk filename located in the current working directory
export TEST=${TEST:="GameDriver_BitBar_Tests.dll"} #Replace "GameDriver_BitBar_Tests" with the name of your NUnit Test Library name

## Desired capabilities:
export APPIUM_URL="http://localhost:4723/wd/hub" # Local & Cloud
export APPIUM_DEVICE="Local Device"
export APPIUM_PLATFORM="android"

## Mac Environment variables - comment out for Bitbar execution
#export ANDROID_HOME=/Users/$(whoami)/Library/Android/sdk
#export PATH=$PATH:$ANDROID_HOME/platform-tools
#export PATH=$PATH:$ANDROID_HOME/tools
#export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home

echo "Starting Appium ..."
appium --log-timestamp &

ps -ef | grep appium

echo "RUN#1"
pwd

# use with --result attribute for BitBar execution, without for local Appium testing
#mono ./ext/nunit3-console.exe ./tests/$TEST --testparam:Mode=appium
mono ./ext/nunit3-console.exe ./tests/$TEST --testparam:Mode=appium --result="TEST-all.xml;transform=nunit3-junit.xslt"

echo "Testing script complete"

For local Appium server troubleshooting, be sure to run the following command from the ANDROID_HOME directory. See the example environment variable from the run_tests.sh file above for adb location.

adb forward tcp:19734 tcp:19734

To remove adb port forwarding, in case you need to get back to local Unity editor testing, run the command:

adb forward --remove tcp:19734

Once you have configured your test to run using the local Appium environment, be sure to comment/uncomment the lines indicated in the run_tests.sh for execution in BitBar before zipping up your test script. Also, if you are using macOS be sure to zip the file contents and not the folder containing the files, or you're run_tests.sh will not be at the root of your unzipped folder and the tests will fail.

Once registered, go to Automation > Create Automated Test in your Mobile App Testing Dashboard

Select the Target OS of “Android” and the Framework “Appium Android Server Side”

Select the zip file you created above along with the .apk or .ipa file for your application. Leave the default selection for Choose Devices, at least for the trial.

Start your tests. From here, you should be able to view the test running on the physical devices for a short period of time. Once the test is completed, you can view the results for each device, including the resource utilization and logs.

That's it. Time to celebrate! If you have any issues with this process, please let us know via support@gamedriver.io or by leaving a comment here.