Breaking up my Lettuce feature tests

Often when writing test automation, you run into bugs.  Arguable the greatest usefulness of test automation is to catch regressions when new changes are made.  But finding bugs is part of it, too, of course.

In other schemes, I’d not been sure how to handle bugs that I know are there, have a bug written against them, and are likely to exist for a while longer.  Sure, it’d be great to have the bug fixed right away, but sometimes it won’t.  How do you mark it as a known failure?  Is it right to put a bug number in the source code to give a heads up where to look for more info on the failure?

My feeling has been that it IS ok to put bug numbers in the source code of your test automation.  But I had a friend who argued that this is wrong.  That source code should never contain bug numbers.  While I don’t agree, I admit, that the issue gave me pause.  It did seem kind of hacky.  But I wanted to have a clear reference to the bug for context.

Anyway, with lettuce the controversy is kind of removed.  Since you don’t define your tests in source code, you define them in .feature files.

However, having them mixed in with your passing tests can be misleading for non-test authors.  “I just want to know which ones are supposed to pass!” one might say.  So here is how I organized things with Lettuce:

In my features folder, I broke my .features files into 3 sections:

• Bugs – Here are the feature scenarios that are broken with known bugs.
• Passing – All the scenarios that are expected to pass.
• Staging – A working area for scenarios.  Ones I’m not done with yet.

This makes it easy for a CI tool like Jenkins, or the developer considering checking in a change, to just execute the passing tests like so:

lettuce features/passing/

You can also, of course, execute an individual feature by doing:

lettuce features/passing/
roles.feature

Break it down any way you like.  I’ve considered making a Smoke folder, too.  Since lettuce discovers all the files named .feature, it’s a snap.

BDD and swapping out steps files

In my current project for Mozilla, I am testing the REST API for our next Test Case Management system.  Using BDD with Lettuce, I’m writing .feature files which lay out each test scenario.

I hadn’t really thought of it when I first started, but it occurred to me that those feature files could serve two purposes.  What if I swapped out my “REST” steps with my “Selenium” steps?

• rest_steps.py would test the product directly on the back-end with the REST APIs.
• selenium_steps.py would use selenium to perform the same features and scenarios, but from a UI standpoint with Selenium

I’m sure I’m not the first to think of this.  I’m just musing here…    By being able to switch contexts, you can be sure that you’re getting testing redundancy based on different contexts.  I know that one might argue that in the YAGNI policy, you really only need the UI steps, but I’m not so sure I agree.  I think being able to test on two different levels helps track down problems more quickly.  Helps avoid the “Must be in the UI layer.” vs. “Must be in the back-end layer.” quarrels.  And you can directly equate your tests in each layer to see where you have equivalent coverage.

I’m not quite sure how implement swapping out steps.py files.  I’ll cross this bridge when I get there.  Luckily, we’re not in need of Selenium tests yet.  We will, though.  But I find the idea compelling.  Knowing that the UI test version is failing, but the REST version passes tells you where to look for the problem.  It could be:

• The UI is broken
• The UI test steps are out of sync with the REST steps
• or perhaps the REST steps are passing when they shouldn’t because the UI doesn’t work that way anymore.

Now, this begs the question: “What if a test is applicable in the UI context, but not in the REST context? or vice-versa?”  Wow, that’s a good question!  I may need to develop a way to specify contexts for Scenarios.  Or one could break the scenarios into different .feature files where some are UI only, some are REST only, and some are both.  Another bridge to cross when I get to it…

Anyway, just my musings on upcoming challenges and possibilities.

My Mock Servlet Object Scheme

Just to recap: I’m testing web services.  And all I have so far is an API of REST endpoints.  So I’m writing my tests against that API.  Since I don’t have an implementation to test, I need a Mock Object to write tests against.

Mock Servlet Object

I wanted a mock servlet that was simple enough not to need much maintenance or brains.  So I went with something simple:  At the beginning of each Scenario, I POST data to the servlet containing what requests and responses it will receive.  The POSTed object looks something like this:

[
  {
    "comment": "User does not exist",
    "request": "/api/v1/users?firstName=Jedi&lastName=Jones",
    "response": "No response expected",
    "status": "404"
  },
  {
    "comment": "Posting the user data",
    "request": "/api/v1/users",                              
    "response": "POST, w/o response",  
    "status": "200"
  },
  {
    "comment": "Check user exists, but not active",
    "request": "/api/v1/users?firstName=Jedi&lastName=Jones",
    "response": get_returned_user(["Jedi Jones"], "false"),     
    "status": "200"
  },
  {
    "comment": "request it to be activated",
    "request": "/api/v1/users?firstName=Jedi&lastName=Jones",
    "response": get_returned_user(["Jedi Jones"], "false"),     
    "status": "200"
  },
  {
    "comment": "expect it's now active",
    "request": "/api/v1/users?firstName=Jedi&lastName=Jones",
    "response": get_returned_user(["Jedi Jones"], "true"),
    "status": "200"      
  }
]

It’s ordered an array of Steps.  On the Servlet, each time it receives a request, it checks that it matches the next expected request, then returns the status and response for that step.

The “response” fields shown above are (mostly) function calls in python.  Those functions are helpers to create the response object the Servlet will return.  Using this structure is working well so far.  I may want to store this in a DB at some point.  But for now, in a source file is ok.

Lettuce and Python for BDD

So, on my project, I decided on Lettuce as the BDD framework for writing my automated tests in Python.  There were a lot of great solutions out there.  But I wanted to use Python because:

1. It’s popular in-house at Mozilla (JavaScript would have been another great choice regarding in-house support though, for that matter)
2. It’s a nice, graceful, easy-to-learn OO language
3. I wanted to learn it

I decided on Lettuce rather than a couple of the other Python BDD options because it’s so similar to the popular “Cucumber” for Ruby.  Also, it seems to be pretty well supported at the moment in terms of recent updates.  Hopefully that’ll continue.

So far, I like it.  Sometimes the output on the cmd line is a little choppy in its formatting.  But overall, it’s been a lot of fun.  I like the way that if it hits a step in a scenario that it can’t find in the steps.py source file, it will give me the Python stub code for it.  Makes it easy to generate the steps methods.  I’m attempting to keep things DRY (Don’t Repeat Yourself) so I’m using variables and enums in the steps to make this work.

For example, in the scenario of:

Scenario: Create a new Environment
 Given I am logged in as user "Walter Bishop"
 And user "Walter Bishop" has the role of "ENVIRONMENT_CREATOR"
 And environment "Walter's Lab" does not exist
 when I add a new environment with name "Walter's Lab"
 Then environment "Walter's Lab" exists

The step function for the 3rd and 5th steps look like this:

@step(u'environment "(.*)" (does not exist|exists)')
def check_environment_foo_existence(step, environment_name, existence):
    check_existence(step, "environment",
                    "name", environment_name,
                    "environment", existence)

def check_existence(step, uri, arg_name, arg_value, obj_name, existence):
 arg_value_enc = urllib.quote(arg_value)
 conn = get_connection()
 conn.request("GET", "/api/v1/" + uri + "?" + arg_name + "=" + arg_value_enc)
 response = conn.getresponse()

 if existence.strip() == "does not exist":
     assert_equal(response.status, 404, uri + " existence")
 else:
     assert_equal(response.status, 200, uri + " existence")
     environmentJson = get_single_item(response, obj_name)
     assert_equal(environmentJson.get(arg_name), arg_value, obj_name + " name match")

In the above step, I have abstracted out the check_existence function so I can check existence on several different types of objects (test cases, permissions, users, etc).

I know that as I progress I will need to refactor more, but so far it’s working quite well for me.

Behavior Driven Development with Python and Lettuce

I have been reading up a bit on BDD and I find it very interesting for several reasons.  Now, just to restate: I’m not the developer on my project, I’m the test automation creator.  So I suppose this is more accurately BDT? Regardless…  Whenever I’ve automated systems in the past, I generally create libraries with functions that do each of the nitty gritty tasks I must do to perform an operation.  Surely, the more elegant the system, often the less nit and grit there will be.  But it usually comes down to making functions like

Create a user
Change address

And the like.  This is really what BDD is doing for you.  You’re making functions that correspond to text strings.  OK, cool.  Not rocket science.  But it IS more elegant than just a method, I think.  I like it because it’s easier to read, for everyone.  It’s nice because it’s a smoother read than reading method signatures.  (Yes, a descriptive method signature works very well, but it’s not as smooth a read when you’re trying to get an overview of the tests involved.  That’s my feeling, anyway…)

But the thing that I’m not sure I like about this kinda goes back to my Java days.  I never liked using strings literals as keys for key-value pairs.  It’s too easy to introduce a bug with a typo that the compiler won’t catch.  So using constants is MUCH smarter.  Plus, an IDE will auto-complete for you, making it easier.  I’m not far enough in BDD to see if this is going to be a pitfall or not.  My fear is I won’t find out I had a typo till I’m 12 minutes into the tests, fix that typo, then hope there’s not another one later on.  But I bet there’s something elegant to handle this.

The thing here I like a lot is that it separates the order of the steps from the implementation of each of those steps.  You want to delete a user on step 3, but the implementation of how that deletion is done is abstracted away.  Keeps it readable and clean.

So I like this so far.  It feels like it fits well with my needs at this point.

Automating Web Services

Here at Mozilla QA, we (obviously) do a lot of testing.  We have a test case management system called Litmus, which serves pretty well to keep track of all the tests we want to cover on a given release cycle.  But it’s aging and we have some improvements that we feel would be better done starting from the ground up with a new tool.  So, we are going to build a fresh-new test case management system that will (if all goes well) meet our needs much better.

The first thing to hit us will be the web-based back-end created by uTest.  These will be a collection of REST APIs that our web developers will write the UI to.  Part of my job is to write some tests against these APIs to make sure they do what they claim.

So I’m venturing forth into the HTTP API testing frontier.  Well… ok… it’s not really a frontier.  This is a pretty well-established country, actually.  There are lots of tools out there.  Several languages to choose from and several design patterns available.  So I’m starting my journey to pick what feels like it will work best for us.

Yet, we don’t actually have the implementation of the REST APIs.  We have the spec from the development team, though.  So my first act was to write some kind of Mock objects that would serve those APIs, against which I can write some automated tests.  When we get the real back-end, I will point my tests at the new server and it will work perfectly.  Right?

For the mock server, I chose to write a few servlets.  This was something I’d done before, so it seemed an easy way to start.  So far I haven’t implemented all the APIs, just a few to get me started.

Next, I wanted to try out a few of the testing frameworks out there.  One obvious choice to try (seeing as I already know Java) was the use JUnit and HttpUnit.  Works great, and easy to execute in Eclipse.  But to get some other flavor, I also implemented the same simple tests in Python with PyUnit.  Both were really simple to implement.  The hardest part (though not even really hard, only relatively) was parsing the XML.  Since I’ve used DOMs before, I started with minidom in Python and Xerxes in Java.  Yeah, old-school, I know.  I may choose a different XML parser in the future.

But the next thing I want to try is using a Behavioral Driven Development framework.  Of course, JBehave is there for Java and RSpec and Cucumber are the Ruby favorites. But does anybody have favorites in JavaScript or Python?  I’ve heard of Leaves for Python, Node and Jasmine for JavaScript.  What’s the favorite out there?