Generating cucumber yard BDD Documentation automagically with Jenkins

When I first read about BDD documentation cucumber-yard here, I fell in love. I then decided to try it out and write my experience in setting it up. That worked great, however I always thought that cucumber-yard would add even more value if I could automate the process of generating BDD documents on fly and host it somewhere. The search began and I decided to setup a Jenkins job which can poll for changes to my git repository, run cucumber yard and publish documents automagically. That way team can refer to an always up-to-date living, runnable BDD documentation.

So I decided to play with Jenkins. Any change to automation Github repo, will trigger a job that will generate cucumber-yard docs.  My initial attempts to run ruby commands in shell failed because Jenkins runs all of its jobs with user “jenkins” and the ruby version for “jenkins” user was not compatible with cucumber-yard. After further research I came across RVM plugin for Jenkins which I thought would solve the ruby version issue. I further decided to use Rake plugin to invoke rake tasks and HTML Publisher plugin to publish cucumber yard docs on Jenkins. So to summarize, here is my Jenkins setup looked like,

1. RVM plugin – to select suitable ruby version for cucumber-yard
2. Rake Plugin- to invoke rake tasks
3. Publish HTML Reports – to publish cucumber yard docs on jenkins

This seemed straightforward to setup, however I stumbled upon a newer road block. I wasn’t able to use RVM plugin properly since “jenkins” user did not have enough privileges to install ruby in System directory. I got below errors at RVM step

[workspace] $ bash -c " source ~/.rvm/scripts/rvm && rvm use --install --create 2.0.0 && export > rvm.env"
ruby-2.0.0-p451 is not installed - installing.
Searching for binary rubies, this might take some time.
Found remote file https://rvm.io/binaries/osx/10.9/x86_64/ruby-2.0.0-p451.tar.bz2
Checking requirements for osx.
ERROR: '/usr/local/bin' is not writable - it is required for Homebrew, try 'brew doctor' to fix it!

I solved this by giving “jenkins” user special permissions like

sudo chown –R jenkins:admin /usr/local/bin

Also make sure to configure Yard for your jenkins user like,

$ mkdir ~/.yard (create .yard in your home directory)
$ yard config load_plugins true
$ yardoc 'src/test/resources/features/*.feature'

After this, RVM plugin was able to install and switch to provided ruby version. After this was resolved, the rest of it was fairly straightforward and it worked great. Here are some sample screenshots of my Jenkins configuration to generate cucumber yard documents.

This entire setup is working great for my team and I hope this post helps you setup cucumber yard on Jenkins.

To goto cucumber yard reports, navigate to

http://your.jenkins.server/job/cucumber-yard/Cucumber_Documentation/?

cucumber yard Config  Jenkins

html report link

BDD Living Documentation using Yard-Cucumber

BDD Framework like Cucumber aims to bridge gap between business and technology. In BDD we describe behavior of software as plain text in .feature file. Cucumber helps us wrap business language around test code so that there is better visibility and tests are not lost in code. Sharing documentation within and across teams is an important aspect and there are currently different solutions available to host and share .feature files. I have used Relish in the past for hosting feature files online. Cucumber.pro also looks promising for collaboration. I came across a post from Shashikant here about living documentation using Yard-Cucumber and wanted explore more.  Yard-Cucumber is  a free tool which can be used to generate documentation interface to view features, tags etc. It generates HTML files and is a nice interface to search/view features and tags. In his blog post Shashikant shows how to use Yard-Cucumber for a ruby cucumber project. I attempted to use Yard-Cucumber for a cucumber-jvm project and share my experience in this blog post. Here are the steps,

Make sure you have ruby 2.0.0 installed

Go to your project and install bundle if its not installed already.

gem install bundle
bundle init

This will create a Gemfile in your project. Add gems below,
gem "yard-cucumber"
gem "redcarpet"
gem "rake"

Then execute,
bundle install

Now lets create Rakefile and add contents below,

require 'yard'
YARD::Rake::YardocTask.new(:yard) do |t|
t.files = ['src/test/resources/features/**/*.feature', 'src/test/java/**/*.java']
end

Now Yard-Cucumber is configured successfully and now you should be able to run the tool to generate reports,
bundle exec rake yard

Now you should see doc folder under your project. index.html should be the launching pad. My sample project is hosted on github here and you can take a look at my docs over here (give it a few seconds to load).

Some images from my project are like,

featuressearch

Cultural shift in Software Development for good

I have been working in agile teams for sometime now as a Test engineer. I have been pondering a lot recently about roles of development and test engineers in the context of delivering business value at a nimble speed in a given sprint.

Although I am a test engineer in my team, I consider myself as a development engineer with test focus. When I have free cycles, I develop features, help fix bugs, write unit tests to improve code coverage, review code (pull requests in github). I took conscious efforts to get to that level and gained respect of my fellow developers. I believe in modern software development, to be a good tester, you need to be a decent developer and vice versa. So while I showed interest in contributing development activities, similarly I wanted developers to show interest in testing activities too. I convinced my fellow developers that they should not only write unit tests but should also contribute to functional tests to a certain extent. They cannot depend on one person for writing, maintaining all functional tests. Quality is shared responsibility and everyone should contribute. We are still in the early transformation of this cultural shift of engineers performing hybrid roles, however the more I think about it the more it makes sense.

Think of this scenario, let’s say a developer has some of free cycles in a sprint; can he/she help with current burning issues like,

  • Increase functional test coverage
  • Fix a failing functional test
  • Review build report, make sure test failures are not caused due to code issues
  • Or something else?

Similarly think of this scenario, let’s say a tester has free cycles in a sprint, can he/she help with current burning issues like,

  • Increase code coverage by writing unit tests
  • Fix bugs
  • Or something else?

If other engineers can perform hybrid roles, then current burning tasks no longer need to wait for the special engineer’s attention.  Lets say we have a database developer in a scrum team and he is the only person who knows about database development, issues etc. If the person leaves the company or goes on vacation, then it’s naturally going to slow down team’s progress. This analogy applies to a tester, developer too. The moment we draw stronger lines and define rigid roles, we add dependencies, which eventually thwart faster development.

Thankfully my team is naturally going in the direction where engineers are able to perform each others roles. Communication between quality and development engineer is much better than before. Although as a team we are still working on many areas, this approach has been working really well for us and as result of it we are able to deliver business value at a better speed. I strongly feel that in agile world the line between a quality engineer and a development engineer is getting blur. This notion is not new. Modern companies like Facebook; Google etc. have successfully implemented this approach and are able to release software at a greater speed.  Sure, test engineers are hired to test and development engineers are hired to develop, however they should be able to perform each other’s tasks when its required in order to release faster.  What are your thoughts? What is working for you in your company?

Introducing Accessibility scanning using Selenium WebDriver

Ever wanted to do accessibility scanning in your WebDriver tests in Java? Introducing webdriver-accessibility, a Java library that helps you run accessibility audits using Selenium WebDriver and Google chrome accessibility developer tools. It uses Google chrome accessibility developer tools to run accessibility audits and then parses the report into a logical Java object.  webdriver-accessibility marks elements which violate accessibility rules on your webpage and takes a screenshots for better reporting. The output result of this tool contains accessibility errors, warnings for you to assert in your tests along with screenshot pointing to problematic elements on your web page.

How to use this library? It’s very simple. Launch your site using WebDriver, and pass along WebDriver object to the tool.

AccessibilityScanner scanner = new AccessibilityScanner(driver);
Map<String, Object> audit_report = scanner.runAccessibilityAudit();

And you could assert like below in your test,

if (audit_report.containsKey("error")) {
 List<Result> errors = (List<Result>) audit_report.get("error");
 assertThat("No accessibility errors expected", errors.size(),equalTo(0));
}

Output Map<String,Object> has following keys,

 /** @type {List<Result>} */
error, //contains all errors
/** @type {List<Result>} */
warning, //contains all warnings
/** @type {String} */
plain_report, //contains plain report
/** @type {byte[]} */
screenshot, //contains screenshot for reporting

Result object is made of following,

 /** @type {String} */
  rule, //contains specific rule information
  /** @type {List<String>} */
  elements, //contains all element locators with errors/warnings
  /** @type {String} */
  information_link, //link to [GoogleChrome accessibility-developer-tools audit rules][3] wiki for more details

If you want specific details of the error, you could scan through the List of errors like below.

List<Result> errors = (List<Result>) audit_report.get("error"); 
for (Result error : errors) {
 log.info(error.getRule());//e.g. AX_TEXT_01
 log.info(error.getUrl());//e.g. Url explaining the error
 for (String element : error.getElements()) //violated elements
  log.info(element);//e.g. #myForm > P > INPUT
}

Similarly you could scan all warnings for details. Just get the “warning” key from the audit_report map. Rest remains the same.

List<Result> warnings = (List<Result>) audit_report.get("warning");

Pros:

  • Light weight library
  • Easily extend your Selenium WebDriver suite to run accessibility audits
  • Use any version of Selenium WebDriver
  • No hard assertions, they are totally up to the user

Cons:

  • Selenium WebDriver uses browsers native APIs to simulate user interactions with the web app. webdriver-accessibility injects Java scripts to run accessibility audits and marking elements on the webpage. Injecting external Java script libraries pollutes global scope of the page under test. Therefore keep your functional and accessibility tests separate.

Here is a sample cucumber report of the accessibility audit that I ran on my sample site. In my sample site, I deliberately introduced some accessibility errors and warnings to demo how the tool captures them and highlights corresponding elements in the screenshot. In the screenshot, you can notice that input text boxes violated missing label rule and are therefore marked with red border. There are in fact 24 violations of missing label rule, in my example below however GoogleChrome accessibility-developer-tools at most provides 5 errors/warnings of each type. Therefore only 5 input boxes with this violation are marked with red border. Also the small pizza image violated missing ALT attribute rule as a warning and therefore marked as yellow.

report

I have hosted the project on github here. I am not an accessibility expert and open for suggestions to make this tool better. If you want to contribute, fork away. Feedback appreciated!

Internationalization: Automating localized UI using Selenium WebDriver

In my last company, we had a web app that supported seven different languages. The challenge was to develop UI automated tests which could run on all localized UIs. We used Selenium WebDriver and its Java binding to simulate browser interactions with the web app. At the time we already had automated significant portion of the app for English UI. All test development was done on windows and in Eclipse IDE. Eclipse wouldn’t understand native characters from most of the languages, although I played around changing encoding to UTF-8 etc.  The text used to appear as garbled (My latest experience on OSX is that native text in Eclipse appear correctly, perhaps has to do with default encoding in OSX).

So my research began and I started to looking into internals of WebDriver and how it worked. I learnt that Selenium Webdriver uses JsonWireProtocol to communicate with browser and it supports UTF-8. A quick note from JsonWireProtocol documentation, “Although the server may be extended to respond to other content-types, the wire protocol dictates that all commands accept a content-type of application/json;charset=UTF-8. Likewise, the message bodies for POST and PUT request must use an application/json;charset=UTF-8 content-type“. That tells me that the underlying WebDriver implementation for any browser is going to convert request body into UTF-8 and communicate to the browser. However I had to deal with Eclipse errors and funky characters. So my next step was to find a tool that can convert native characters into UTF-8. I found that JDK provides an excellent utility called native2ascii which can easily convert native characters into their ASCII representation. Since this was a one time or once a while process, I decided to leave this step as manual. However there is a native2ascii maven plugin or an API which can be potentially used if you desire to automate this step. You can also provide desired encoding. So lets take below for example (If you are developing on OSX this step is perhaps not required)

source.txt
ラドクリフ、マラソン五輪代表に1万m出場にも含み

command
native2ascii -encoding utf8 source.txt output.txt

output.txt
\u30e9\u30c9\u30af\u30ea\u30d5\u3001\u30de\u30e9\u30bd\u30f3\u4e94\u8f2a
\u4ee3\u8868\u306b1\u4e07m\u51fa\u5834\u306b\u3082\u542b\u307f

Try this sample quick code below,

WebDriver driver = new FirefoxDriver();
driver.get("http://www.google.com.hk/");
String textToSend = "\u30e9\u30c9\u30af\u30ea\u30d5\u3001\u30de\u30e9\u30bd\u30f3\u4e94\u8f2a
\u4ee3\u8868\u306b1\u4e07m\u51fa\u5834\u306b\u3082\u542b\u307f";
WebElement searchButton = driver.findElement(By.name("q"));
searchButton.sendKeys(textToSend);

You will notice that the search will actually be populated with ラドクリフ、マラソン五輪代表に1万m出場にも含み. Nice and nice!!!! Now I have a way to easily transform something that Selenium WebDriver + my IDE can understand. This was promising indeed. So I decided to externalize text that I am verifying in properties files (key-value pair); one per locale. Obviously keys across locale will remain the same, just the values would differ. Based on the locale under test, I decided to load the appropriate property file and the automation will run seamlessly.

I was happy with this approach and went ahead with it. Our application contained many pages and two weeks later I realized that my property files are growing big. I had seven property files for seven languages to maintain and things were going out of control. I wanted to retrospect on my approach and find if I could change my approach quickly before its too late. Agile taught me well! I posted questions on forums but didn’t get any convincing answers.

I went to our front end developers to understand the actual implementation of localized UIs and learnt that backend implementation is independent of locale. They received locale files from external vendors who translated text into desired languages. These locale files were essentially a key/value pair, similar to my earlier approach. Keys across different locale files were same, but values would change based on the language. All Html files for individual locales were pre-compiled as a part of build and based on the users language preference at the login screen corresponding pre-compiled resources would be loaded, backend would remain the same. In some pages, based on user actions javascript would return dynamic text (locale specific success/error messages etc). I then realized that I was reinventing the wheel earlier and I could just straight use these resource bundles from vendors. After further thoughts, I decided to use locale files from vendors directly, parse them as Properties file and read in my code. Before I parse them, I converted all resource bundles from native characters to ASCII characters (UTF-8) so that Eclipse wouldn’t complain plus WebDriver could understand them as well. Like I showed above,

native2ascii -encoding utf8 japanese_native.txt japanese_ascii.txt
native2ascii -encoding utf8 chinese_native.txt chinese_ascii.txt
and so on....

So to put things into perspective, Below is what we would get from the vendors. Columns indicate locale file and two rows demonstrate sample content of those files.

english.text japanese_native.txt chinese_native.txt
SUBMIT_LABLE= Submit SUBMIT_LABLE= 提出する SUBMIT_LABLE= 提交
ERROR= Error ERROR=エラー ERROR=錯誤

native2ascii tool easily converted these native source files to ASCII file which were essentially fed into my automation suite. So the conversion would look like below,

english.text japanese_ascii.txt chinese_ascii.txt
SUBMIT_LABLE= Submit SUBMIT_LABLE= \u63d0\u51fa\u3059\u308b SUBMIT_LABLE= \u63d0\u4ea4
ERROR= Error ERROR=\u30a8\u30e9\u30fc ERROR=\u932f\u8aa4

And here is some psuedo-code to give you an idea. Based on the language of preference, I load appropriate locale file,

if(language.equalsIgnoreCase("japanese"))
properties.load(new FileInputStream("japanese_ascii.txt"));
else if (language.equalsIgnoreCase("spanish"))
properties.load(new FileInputStream("spanish_ascii.txt"));
and so on...

My page objects would have something like below. So just like the application code, my automation suite is only dependent on loading the correct file, page objects were independent of the locale.

String key = properties.get("SUBMIT_LABLE");
submitButton.getAttribute("value").equals(key)

Benefits of this approach

  • No need to maintain and create locale files.
  • UI functional automated tests ran on all locales seamlessly
  • Straight forward, less complicated- UI automation is difficult in itself; don’t complicate 
  • Automation suite found many locale specific bugs
    • Some languages were missing few keys during translation due to obvious human errors. Such errors resulted in default english text to appear when locale specific text was expected
    • Integration bugs detected with Javascript like dynamic success/error messages etc.

The approach worked great and provided significant value from testing standpoint.  The lesson I learnt from this experience is that we need to retrospect constantly and improve testing practices. Fail fast so that you can recover quickly. Work closely with developers and product stake holders. Working closely with developers and product stake holders, I was able to understand internals of our web application and it led me to answers I was looking for. UI automation is a hard problem to solve, not to mention the ever changing/evolving web app and need for testing on different browsers, platforms etc. I believe complicating browser automation suites further with adding more layers to support testing different locales is just an overhead and destined to fail. In my case the web app used resource bundles, some apps may have different implementation, I believe that for testing localized UIs, we should leverage native application solutions for internationalization as much as possible .

What do you think? How would you approach?

Creating custom HttpMessageConverters for Spring’s RestTemplate

Yesterday, I struggled with one of ours REST API which is old and oddly designed. The API always returns, 200 HTTP status code in the response header, no matter the request is successful or not. Response body has a field called as “code” along with some other fields which represents the actual status code. In situations when the ‘actual’ response code is non-200 in the response body, HTTP response header code is still 200 and RestTemplate thinks that this is a genuine response and tries to deserialize the response body and fails! This made me write my own HttpMessageConverter that I could pass on to RestTemplate to deserialize the response object.

I looked at MappingJackson2HttpMessageConverter source and pretty much kept the source except for the readInternal() method. Rest of the methods in my converters are exactly like MappingJackson2HttpMessageConverter, hence I have taken out the duplicate methods below for brevity.

public class CustomHttpmsgConverter extends  AbstractHttpMessageConverter<Object> {

    public static final Charset DEFAULT_CHARSET
    = Charset.forName("UTF-8");
    private ObjectMapper objectMapper = new ObjectMapper();
    @Override
    protected Object readInternal(Class clazz, HttpInputMessage inputMessage) throws IOException, HttpMessageNotReadableException {

      InputStream istream = inputMessage.getBody();
      String responseString = IOUtils.toString(istream);
      JavaType javaType = getJavaType(clazz);
      try {

        return this.objectMapper.readValue(responseString, javaType);

      } catch (Exception ex) {

        throw new HttpMessageNotReadableException(responseString);

      }

    }

    Notice the use of string in object mapper for deserialization. Original implementation in MappingJackson2HttpMessageConverter uses input stream to deserialize, but I first use it to convert it to string since I need to preserve the response body. HttpInputMessage stream can only be used once because it is closed internally after it is fetched for the first time. Therefore, I can only use string to deserialize in my converter. This is important to note.

    In my @Test I just catch the HttpMessageNotReadableException which has the reponseString if the message is not readable

    @Test
    public void test() {

      String url = "htt://my.application.com";
      RestTemplate template = new RestTemplate();
      List<HttpMessageConverter> converters = new ArrayList<HttpMessageConverter>();
      converters.add(new MDLmsgConverter());
      template.setMessageConverters(converters);
      MultiValueMap header = new LinkedMultiValueMap();
      header.add("Content-Type", "application/x-www-form-urlencoded");
      HttpEntity requestEntity = new HttpEntity(null, header);
      try {
      ResponseEntity<Data> response = template.exchange(url, HttpMethod.POST,
      requestEntity, Data.class);
      Assert.isTrue(response.getBody().equals(expectedDataObject));
      } catch (HttpMessageNotReadableException e) {
      Assert.isTrue(e.getMessage().equals(
      "this is what the response body was"));
      }

    }