AgileIQ Blog

Agile Software Development: BDD With Spock and Selenium

by Tim Myer

Goal

To develop a web site specification-first using Spock and Selenium WebDriver.

tl;dr

The sample code for this project is available on github and will be a helpful resource for following this article. The interesting technologies showcased include Spock, Selenium WebDriver, Selenium PageObjects, Sauce OnDemand, Gradle and Grails 2.0.

The Project

Suppose our product owner would like to publish a website for teachers to schedule their classes and for students to register for those courses. At the first backlog grooming, the product owner has prioritized the stories that would constitute a minimum viable product.

User Stories

During the team's first sprint planning meeting, we sized the stories, accepted four into the sprint, and have added tasks for each. Our story board for the first sprint contains sticky notes for our user stories, the acceptance criteria and the individual tasks for each story.

Project Structure

We will create a top-level project course_registry and the two sub-projects web and specifications. We will also create a build.gradle file in the root along with settings.gradle and a build.gradle file in the specifications subproject for running automated acceptance tests (While this is not strictly necessary, it will help to keep our project boundaries clear).

course_registry/build.gradle

course_registry/settings.gradle

course_registry/specifications/build.gradle

We will create a src/test/groovy folder in the specifications project. At this point we can generate Eclipse .project and .classpath files for all the projects from the project root in order to work within an IDE.

The Teacher Registration Spec

We are now able to write the first specification for our acceptance criteria.

timezra/course_registry/TeacherRegistrationSpec.groovy

We can run this spec and watch it fail.

Satisfying the Spec

Suppose the team decides to use Grails 2.0 to implement the specification. This is not a restriction based on our other technology choices, since gradle is a general purpose build tool and since our specification defines how a user will interact with our web project entirely through a browser. The decision is based on the convenience of the framework, community support, the plug-in ecosystem and the skills of the developers.

First, we need to create an empty web/grails-app directory to indicate to the grails bootstrap that the project will be a grails application.
Then, we will configure our web build for Grails 2.0 in a new web/build.gradle file.

web/build.gradle

From the project root we will initialize the grails project.

We can satisfy the spec simply by modifying application.properties with the expected web application deployment path and project version, the homepage to include the expected title, and messages.properties to contain the expected messages.

web/application.properties

web/grails-app/views/index.gsp

web/grails-app/i18n/messages.properties

We can now add hooks to the specifications project to stop and start the web application before and after running the specifications, respectively.

specifications/build.gradle

These hooks depend on a specific interface in the web project, i.e., the existence of the tasks webStart and webStop. We can deploy the Grails 2.0 artifact to an embedded Jetty server through gradle to satisfy this interface.

web/build.gradle

The first specification should now be satisfied.

NB: The Jetty plugin must be applied before the grails plugin, or else you will see an error similar to the following:

Adding another test for our second acceptance criterion should be straightforward now that our infrastructure is in place.

timezra/course_registry/TeacherRegistrationSpec.groovy

We can satisfy this specification by generating a Grails Teacher domain object with fields for the name, email and password, along with a Grails controller that uses dynamic scaffolding to generate the view and actions available, and by adding a link to the scaffolded Create Teacher page on our web/grails-app/views/index.gsp. Since the point of this tutorial is not to cover basic Grails development, we do not need to go into those specifics here, but sample code can be found on the project site or in any Grails tutorial for further reference.

Page Objects

Before we get too far with automating our acceptance criteria, we should begin to look towards a more abstract representation of the testable components of our application. From the second acceptance test above, we can already see patterns emerging. For example, when we navigate to a particular page, it would be convenient to identify that we are on the correct page, perhaps by its title. It would also be convenient to represent each view in the application as its own object, and each view could encapsulate its specific WebElements. Finally, there is a small set of actions available for any WebElement on a page, and it would be helpful to build these actions into a testing DSL. Fortunately, by combining the Page Object pattern and Groovy's dynamic language features, we can achieve all these goals with a minimal amount of code.

As we are satisfying the specifications for all the acceptance criteria, a base Page Object emerges.

specifications/src/test/groovy/timezra/course_registry/pages/CourseRegistryPage.groovy

NB: The constructor verifies that the page is correct by comparing the class name to the page title; in addition, the static goTo method demonstrates how to use the PageFactory for initializing a Page Object; the methodMissing declaration also showcases Groovy's ability to call dynamic methods based on the combination of the click, type and choose actions with individual WebElement names. Such phrases become first-class elements of the testing DSL.

We can see the use of WebElement injection and dynamic methods in action with a CreateUser page that serves as the base for the CreateTeacher and CreateStudent pages.

specifications/src/test/groovy/timezra/course_registry/pages/CreateUser.groovy

Other Browsers

So far, we have done all our testing with a single WebDriver, i.e., the FirefoxDriver. For simple testing, a single driver works fine, but if we wish to ensure that our website works in multiple browsers, we will need to configure our tests to use multiple WebDrivers. Fortunately, since we are using Gradle to manage our dependencies, the InternetExplorerDriver and ChromeDriver should be available to us automatically, along with the AndroidDriver and IPhoneDriver for testing our application on mobile devices. In addition, an OperaDriver is also available.

NB: Some of these drivers require that additional platform-specific software be installed, so please read the project pages for documentation on additional requirements.

Configuring our specifications to use multiple WebDrivers should just be a matter of parameterizing each Spock feature.

NB: Unlike JUnit tests which are parameterized by fixture, Spock specifications are parameterized per feature.

We will modify our TeacherRegistrationSpec to use the Spock where: block for parameterization, we will configure multiple WebDrivers, and we will share the instance driver field among the parameterized features.

timezra/course_registry/TeacherRegistrationSpec.groovy

Testing In The Cloud

Now that our specifications are running through multiple browsers on a single machine, we can move a step further towards testing multiple browser versions on multiple OSes with cloud-based services such as Sauce Labs. For such a powerful service, the configuration changes to our existing application are surprisingly simple. We will modify our TeacherRegistrationSpec to use a RemoteWebDriver for the specific browser and OS combination we would like to test along with our Sauce Labs username and API key as described in the Sauce OnDemand documentation. We will also need to add a Gradle plugin to start SauceConnect from the build, just as we start our web application before running acceptance tests. Finally, we should ensure that all references to localhost in our application are changed to the IP address of the machine where we will be running our tests; otherwise, we might see the SauceConnect proxy freeze (you might have a different experience, and this freezing might just be attributable to the Gremlins in my machine).

timezra/course_registry/TeacherRegistrationSpec.groovy

course_registry/specifications/build.gradle

Conclusion

This post took us through the definition of user stories for a small website, the setup of a Gradle project for automating the website build and the run of its corresponding acceptance tests, the creation of our first WebDriver-based Spock specification for our acceptance criteria and the fulfillment of that spec with a Grails 2.0 application, the use of Selenium PageObjects to create abstract representations of our web pages before those web pages are even written, all the way to the creation of a Gradle plugin for running our acceptance tests on various browser/OS combinations in Sauce Labs. Each of those items in itself is worthy of a tutorial series. The combination of all these elements demonstrates with a remarkably small amount of configuration and code the type of robust and scalable ATDD infrastructure that is possible and should be at the core of any enterprise web application.

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

When the bottleneck of an Agile team is the team itself

by Jayaprakash Puttaswamy

Most often, we have observed that agile implementations fail in spite of putting efforts towards educating the team about Agile principles, training the team on Agile practices, and getting the right people for Agile roles. Have you ever wondered why? There are chances that the culprit is the team itself.  The team could be plagued with several dysfunctions (refer to  “the five the dysfunctions of team” model from Patrick Lencioni). If you discover this in your teams, how would you go about dealing with it? How can we overcome the dysfunctions and thus help the team implement agile successfully?

Well, there are ways to do it. There are instruments and techniques to do it. But all these would fail if you don’t have good leadership skills. I would like to share my experience with similar situations, with success stories, challenges and the lessons learnt. Here is a case study which I would describe in three sections:

1. SITUATION: Problem statement
2. APPROACH: Key skills used
3. SOLUTION: Results obtained, challenges faced, lessons learnt

SITUATION

The situation was exactly as outlined below. It was a nine-member team, suffering from dysfunctions (mentioned inside the pyramid). As a result, the team was exhibiting the dysfunctional characteristics (mentioned on right hand side of the pyramid).

Problem statement

• The team was slipping on deadlines, unable to fix critical issues and could not removeits  high dependency on two key technical people.
• There were also side effects:
  • Decrease in the morale of one of the key technical people
  • The other key person not being able to focus on continuous improvement

APPROACH

The team members were trying to follow Scrum as their way of implementing Agile, and there was a technical person who was struggling to play the ScrumMaster role. I had joined this team as a development manager. Irrespective of my role, I had to take charge of the situation to identify the bottleneck. The first thing I did was to take out the teams for a couple of lunch sessions (we decided not to wait for company sponsorship with the budget approval).

Those sessions, coupled with a team assessment instrument, helped us figure out as a team what the dysfunctions were that were affecting us. The rest was all about leadership skills (refer the inverted pyramid below).

Key skills used:

• Situational leadership
• Conflict management
• One-on-one feedback
• Time management

SOLUTION

As a responsible leader, I had to drive the team in a slow and steady manner on the following aspects:

• Making retrospective meetings effective
• Open evaluation of team dysfunctions
• Consensus on tackling “trust” and “conflicts” (which were affecting the team most)
• Enabling the team to share personal details (strengths and weaknesses)
• Effective sprint planning
• One-on-one discussions between me and other team members

Results obtained

After a period of 6 months of continuous experiments (I call them experiments since there was no single way to address “trust” and “conflict” issues), we achieved the following:

• Completion of tasks on-time, with better quality
• Significant reduction of technical dependency on individuals
• Improved “self-organizing” capability of the team
• Increased morale of the team, as well as the “key technical people”
• More critical issues were identified and fixed
• The other key technical person got opportunity to improve his “leadership” skills

Challenges faced

• Convincing the people on abstract concepts like “dysfunctions”
• Time management
• Dealing with different personality styles

Lessons learnt

• Most of the radical changes require “paradigm shifts”
• “Going first” or “being vulnerable” is the first step
• “Constant focus” is needed to bring real changes

Conclusion

Any successful Agile implementation requires a good leader who can drive the team by identifying dysfunctions, helping the team overcoming them, and facilitating the process of change management if needed. Any of the team members (especially management people) can do this and they should do ONLY this and nothing else. Rest, the team figures itself out. Ending with a quote from Henry Kissinger:

“The task of the leader is to get his people from where they are to where they have not been.”

 

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

Join us in February at Agile India 2012 in Bangalore

by Pam Dyer

Agile India 2012 is Asia's largest international conference on Agile and Lean software development methods. The conference will held 17th-19th February 2012 at Le Meridien, Bangalore, India. It offers many sessions from Agile experts, including SolutionsIQ CEO Charlie Rudd and SolutionsIQ Agile Consultant/Coach William Rowden:

Confronting Business Uncertainty

Charlie Rudd
Friday 17 February at 10:45am

We can never manage uncertainty. Yet in today’s business environment we confront it all the time. This talk explains why Agile principles are needed to make good management decisions in an uncertain business environment and why past practices no longer work. We will review what drives business uncertainty and why it’s here to stay. We will introduce Agile management principles and explain how they help us navigate uncertainty. We will learn how to use these principles to exploit new business opportunities, reduce business risk and increase returns, while decreasing investment requirements.

Lean Workflow: A Parable in Pictures

William Rowden
Friday 17 February at 10:45am

This talk is the tale of two Scrum teams — team Shu that watched utilization and team Ha that watched lead time — diagramed on their task boards. Watching work in progress on a task board is sufficient to explain throughput, local sub-optimization, constraints, utilization, efficiency, service time, lead time—and why even Scrum teams would benefit from a Kanban. Additional maps illustrate wait time, value streams, and wastes. The result is an introduction to Lean, its pillars and principles, primarily in pictures.

Distributed Scrum: Why Some Teams Make it Work Well and Others Don't

William Rowden
Saturday 18 February at 10:45am

How does a geographically-distributed team collaborate across distance, culture, and even language? What can an organization do to encourage successful teamwork? What technologies and practices increase collaboration? This workshop will explore the patterns that enable companies to successfully deliver software with a distributed team. It will be presented by an Agile coach that assisted with the transition of a company that adopted Agile for teams in three time zones with two languages and cultures, and (travel permitting) the VP Technology of that company.

Agile Governance

Charlie Rudd
Saturday 18 February at 3:00pm

Ironically, conflicts often arise between Agile practitioners and the governance policies of their organizations, even though Agile can provide an excellent (often superior) governance framework. This presentation will explain why these conflicts arise and how they can be resolved. Agile methods will be presented as a governance framework and compared against past practices. Special emphasis will be given to Agile practices as a governance framework for geographically distributed teams.

Don't miss this fantastic event!

At Agile India 2012, more than 500 participants from all over the world will share their Agile knowledge through presentations, workshops, and tutorials. Twenty-two international research papers from renowned institutes will be presented as well. The full schedule of events is available here.

Learn more and register today!

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

Reflection Tools for .NET

by Robert Zormeir

For almost as long as there has been a .NET development environment there have been free developer tools that used Reflection to help developers understand the nature of the beast they were dealing with. The tool that almost everyone started with was .NET Reflector, which was originally written by Lutz Roeder. In 2008, Red Gate Software announced they were taking over future Reflector development, and that Reflector would remain free.

Early in 2011 Red Gate announced that .NET Reflector would become a paid-for product, and that the free version would no longer be available. They also announced that previously downloaded free versions would time out. After being on the receiving end of what had to be an amazing stream of bile and vitriol due to the idea of reneging on their original promise, they relented and announced that previously downloaded versions would be automatically updated to become permanent.

As we near the one year anniversary of Red Gate's questionable marketing decision, it probably didn't lead to many sales of their paid version, but sudden lack of an established free reflection tool did lead to the creation of several free alternatives:

• JetBrains dotPeek
• DevExtras CodeReflect
• ILSpy
• Telerik JustDecompile

A new version of JetBrains dotPeek was released on December 29th and is the cream of the replacement crop so far. It gives you some of the same functionality you'll find in their ReSharper product: The ability to navigate to a specific type, assembly, symbol, or type member; to move between declarations, implementations, derived, and base classes in inheritance chains. It has no installer, so once you download and unpack the ZIP file and you're good to go. DotPeek disassembled my test dll with very good fidelity, but didn't insert curly braces in all the places where I'd like to see them. Maybe a future version will give me the abiltiy to specify code formatting rules the same way ReSharper does. In general, the functionality of dotPeek gives you the idea that you might be in a mini version of Visual Studio.

DevExtras CodeReflect is available as an MSI installer or a ZIP file. It was quick to download and unpack. It performed well, but didn't do the full colorization of the code the way dotPeek and JustDecompile did. One nice feature is that it allows you to quickly switch back and forth between MSIL, C#, and VB.NET code. The decompilation fidelity was good, but it did generate some intermediate variables that didn't exist in the original code.

LSpy is an open-source project that requires .NET Framework 4.0. You can download the binaries or the source code from SourceForge. The program will check for updates once a week if you prefer. The performance was good, and the decompilation fidelity was very good, but the code colorization was a little weird. But I can live with that. You can quickly switch back and forth between generating MSIL and C# code.

The installer for Telerik JustComplile asks for too much registration info and takes too long to do its work, IMHO. Telerik appears to be viewing their product as too much of a marketing opportunity... to the point that it ceases to be a marketing opportunity with me. You'll see "Just kick back and relax. Good things take time. Please be patient." as the installation proceeds. You also need to be patient as the the installed program goes through the process of "Loading assemblies." JustDecompile had a bit of trouble with decompiling a relatively simple class, but it did come close and the software is still in beta. Once everything was loaded the performance seemed very good.

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

Mocking Frameworks for .NET

by Robert Zormeir

The four major competitors in the free/open source .NET mocking framework arena are NMock/NMock2, NMock3, Rhino Mocks and MOQ. NMock and NMock2 were actually built by different teams, but they kept the same design philosophy and are backwards compatible so they can be used almost interchangeably. If you're familiar with NMock/NMock2, you'll see that NMock3 is from the same gene pool, but it starts fresh and has somewhat different syntax from its siblings. All of the major mocking libraries are mature products with a decent-sized user base, so it's not tough to find good examples and help for all of them but NMock3.

Like many other .NET developers, I started using NMock, and then "upgraded' to nMock2, but both use "magic strings" which can't take advantage of Intellisense, and make tests brittle because they're not easily amenable to refactoring without the use of additional tools like ReSharper to replace name-similar text in strings. On the plus side, NMock doesn't rely on the explicit record/replay statements as Rhino Mocks does. The NMock2 codebase hasn't been updated since late in 2009, and should not be expected to see continued development.

On the plus side, Rhino Mocks has a syntax that supports code refactoring and compile-time checking. Unfortunately, I don't find the syntax to be particularly intuitive, and the variety of ways to construct and condition mocks can lead to some confusion when different people are writing tests. There is also the annoying need to write explicit record and replay statements. Another problem with Rhino Mocks is that the project appears to be running out of steam. Aside from a few bug-fix patches, not much has happened since early in 2009.

MOQ is one of the new kids, with an annoying name that needs to be spelled out so that people know what you're talking about. The latest release became available in April of 2011, and MOQ has active developers and community. It's different from the first two in several ways. When using NMock and Rhino you create mock objects of a specific type. When using MOQ you create mocks that contain an objects of a specific type, and that wrapper around the mock objects lends itself to simpler syntax in your tests. MOQ requires .Net 3.5 or greater due to its use of lambdas, so if you haven't learned to use lambda expressions, now is the time. I haven't used it extensively, but so far I like everything about MOQ... except for the name.

NMock3 is the other new kid, and the one who seems less well known. Maybe it isn't on the radar for most folks because NMock/NMock2 seem behind the times, and that may have kept expectations low. Versions of NMock3 are available for .NET 3.5 and 4.0, and the latest RTM became available in January of 2011 with a beta release in July of 2011. Like MOQ, you create mocks that contain objects of a specific type, and that wrapper around the mock objects leverages lambda expressions, so NMock/NMock2 users are in for some culture shock. Since the NMock3 examples are a little thin in the wild I've include sample code below. Note that in conditioning your mocks, the parameters in the lambda expressions are just placeholders, while the "real" parameters are specified in the "With" method.

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

Use-It Testing in Agile Software Development

by Dan Ebert

Agile Coach: “What kind of testing do the developers do?”

Development Team Member: “First we create some test data if needed. Then we go to the web page and use the feature to make sure it works.”

I call this anti-pattern “Use-It Testing.” I’m always surprised, and disappointed, when I encounter a team that relies solely on Use-It Testing when writing their code.

I’d bet everyone who’s written a line of code has done it at least once. You write a bit of code … then use it by running the program, going to the web page, etc. to see if it works. This *might* be OK for prototyping (I’m sure there are some who would argue that Use-It Testing is not sufficient even when prototyping) but it is definitely not sufficient for production work.

There are three main types of testing typically done by an Agile Team: Unit Testing, Integration Testing, and Functional Testing. All should be automated so they can be run frequently with little effort.

Unit Tests are the finest grained. The units under test should be very small, just a few lines of code, and are often written the same time as the production code. They isolate the code under test from all other components by using Test Doubles if necessary. There should be more Unit Tests than any other kind of test because they cover all paths through the code. Due to the quantity, these tests must execute quickly so they can be run frequently to provide feedback during development. They also provide the safety net when refactoring and changing functionality.

Integration Tests test the interaction between two components: testing a DAO’s (Data Access Object) interaction with the database, testing a client against a web service, etc. Fewer Integration Tests are needed because they don’t need to cover every path (Unit Tests are responsible for that); testing a few paths is sufficient to assert the integration works. Integration Tests usually take longer to run, but this is OK since there are fewer tests to run.

Functional Tests can be broken down into many categories: Acceptance Tests, UI Tests, End-to-End Tests, etc. These test the whole system, are usually the slowest running, and should be the fewest in number. They are good at detecting environment issues and testing the interaction of multiple components.

In addition to these automated tests, most teams do some Exploratory Testing as well. This *is* using the application … but with the main goals of assessing usability and finding features/behaviors which were missed. These findings often result in new stories for the backlog. Of course, bugs can be found during this testing. The appropriate (failing) test should be written, then code fixed, then the test run and the featured exercised to verify the fix.

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

Developing Eclipse Plug-ins: Program to Publish

by Tim Myer

Goal

The purpose of this blog entry is to demonstrate one workflow for taking an idea for an Eclipse plug-in from concept to product downloadable from the Eclipse marketplace.

tl;dr

If you want to look right away at working code for a simple Eclipse plug-in, feature, update site and their Tycho configurations, a sample project based on this tutorial is available. The plug-in can also be installed in Eclipse through the Marketplace (search for "save actions") or directly from an update site. In addition, the conventions for project setup contained in this tutorial have been extracted into a Maven archetype. Since this blog entry does not provide an in-depth introduction to the nuances of configuring and writing Eclipse plug-ins, having the project source as a reference, either locally or in a web browser, will be helpful for following the examples.

The Idea

Suppose we are starting a new project on an existing codebase and, as part of our Working Agreement, the team has decided to automate certain coding standards with Eclipse Java editor save actions. It would be convenient to bring all the existing code up to our standards before development even begins. Currently, formatting and import organization can be performed in bulk from the source submenu in the workbench. Eclipse users can also apply clean-up conventions from this submenu. Unfortunately, even though the clean-up and save participants have similar configurations, there is a disconnect between the two.

Exposing save actions for the bulk processing of Java/Groovy files through this source submenu is a simple enough feature to add to Eclipse and should provide the opportunity to experience one full cycle of Eclipse plug-in development, from the creation of a simple plug-in and its integration tests, to the addition of a feature to contain the plug-in, to the packaging of an update site, to the distribution of the product through the Eclipse Marketplace.

The Plug-in

We will begin by creating a new plug-in project named timezra.eclipse.apply_save_actions in our Eclipse workspace. Since we will eventually generate a Tycho configuration in order to automate the compilation, testing and packaging of our product, we will modify a few of the default settings for the plug-in. Our plug-in and fragment projects will be contained in a plugins subdirectory, here /path/to/workspace/plugins/timezra.eclipse.apply_save_actions. Our source folder will be src/main/java and output folder will be target/classes to follow the Maven convention.



We can either use the New Plug-in Project Wizard Hello, World Command Template or we can contribute a command to a menu with manual configuration. There are already quite a few resources for contributing commands through an extension point, so you can get more insight into specific configuration settings there.
For this example, the configuration is boilerplate. The internals of the ApplySaveActions command handler are somewhat interesting. There may be a more direct way to invoke the save participant, but this method works on both un-opened files and source buffers modified in memory for Java and Groovy, so in the interest of DTSTTCPW, we can use this simple implementation until it is no longer sufficient.

plugins/timezra.eclipse.apply_save_actions/src/main/java/timezra/eclipse/apply_save_actions/handlers/ApplySaveActions.java

The Test Fragment

We will similarly create a new integration test fragment alongside this plug-in, overriding the default configuration to store the fragment into the plugins subdirectory and to use Maven conventions for the source and output directories.



There are various approaches for testing Eclipse plug-ins, but the fragment approach has been embraced by the Tycho community, so we will use it here.
Again, the project configuration can be boilerplate for now. Of particular interest is the handler test case.

plugins/timezra.eclipse.apply_save_actions.tests/src/test/java/timezra/eclipse/apply_save_actions/handlers/ApplySaveActions.java

The Feature

Now that we have a tested plug-in, we will create an Eclipse feature to contain it for distribution. We can do practically all our configuration through the New Feature Project Wizard, except we want to put the feature project into a features subdirectory in the same way that we put our plug-in and fragment into the plugins subdirectory.

The Update Site

Now that we have a tested plug-in and a feature to contain it, we will create an Eclipse update site for publishing the feature. In the New Update Site Project Wizard, we will again override the default location so that our update site project is in an update-site subdirectory, just as we separated our plug-in, fragment and feature into plugins and features subdirectories.


NB: Whereas we may have multiple features or plug-ins for our project, we will have a single update site; thus the update-site project is not contained in a subfolder of update-site, but in this folder directly.

Tycho

Compiling, running integration tests and packaging an application entirely within an IDE does not scale to even a single programmer over time, let alone a team of programmers working on multiple plug-in projects. So far, the amount of ceremony for creating our menu contribution has been high, but the IDE has reduced a significant amount of the boilerplate. Before Tycho, the amount of ceremony and hackery required to get plugins, features and update sites packaged and unit and integration test suites running on a CI server had been prohibitively high. Tycho removes a significant amount of that pain. Generating meaningful poms for our projects is as trivial as going into each of the features, plugins and update-site directories and running a Tycho goal.


We can combine some of the boilerplate in each of the subproject poms in an über-parent pom at the root of our workspace. We will also add the Indigo p2 repository and a target platform configuration resolver since we are developing our Eclipse components Manifest-first.

pom.xml

We will also configure Tycho to use the UI test runner for our integration test suite, as well as add any platform-specific runtime plug-in dependencies or configuration to the test fragment pom.

plugins/timezra.eclipse.apply_save_actions.tests/pom.xml

NB: we would use a different configuration if our fragment contained unit-tests instead of integration tests.

JAR signing

If you do not have access to a certificate from a trusted authority, you can generate a self-signed certificate with 1-year validity by a command such as


In order to sign the jars deployed to our update-site before release, we will add a new profile with a plug-in management configuration to the über-parent pom.

pom.xml

Similarly, we will configure the plug-ins and features parent poms for jar signing.

plugins/pom.xml and features/pom.xml

Publishing

We are now ready to build and test our signed plug-ins and features and to package them in an update site for deployment.

There will now be a fully deployable update site in update-site/target/site.
For this particular project, I distribute the contents of this directory to the gh-pages branch of the project on github.
Since the update site for the project is publicly-available and since I have an Eclipse Bugzilla login, I can simply add a new solution listing to the Eclipse Marketplace to make the menu contribution even more discoverable by and accessible to Eclipse users.

Conclusion

This tutorial has provided the outline for the workflow of taking an idea for a single-featured Eclipse contribution from inception to delivery in a very short amount of time. While Eclipse tooling has for years provided the means to perform the tasks of plug-in, fragment, feature and update site publishing entirely within the IDE, it is the Tycho project that lowers the barrier to entry for scaling out plug-in development by making the build and test process far simpler to automate and to configure than other PDE-based build systems. Along the way, we have explored JAR signing and uploading of content to github and to the Eclipse Marketplace, and we have hopefully developed a menu contribution that others will find useful in their own projects.





Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

Agile Coaching: A Kanban Exercise

by Dhaval Panchal

Kanban at its essence is a signaling device that instructs the moving or creating in a “pull” based system. Kanban when applied within a system enables synchronization between the rhythm of demand (customer) and rhythm of production (producer). The pace or beat, of customer demand is takt time. In other words to enable just in time production of goods it is best for producers to produce goods at the same rate as the customer demands. A faster rate of production results in finished goods waiting to be consumed (inventory) and a slower rate of production results in missed opportunity costs or dissatisfied customers. This aspect of continuously tuning the producer’s rate of production to customers demand requires a continuous improvement (kaizen) mindset. My favorite article on Kanban applied to software development is by Kenji Hiranabe. Please read that article before you dive into the details of setting up and facilitating this exercise.

Time:

60 – 90 minutes

Input:

• 20 Coins (any denomination)
• 3x5 Index cards – 100 pieces
• Table length to accomodate 4 to 5 people, preferably a U-shaped table
• 3 Stopwatches
• Color paper – legal size, two different colors (say Purple & Brown) – 10 sheets for each color
• Facilitator
• 3 timekeepers
• Flip chart or whiteboard
• Whiteboard markers

Setup:

• Create a table as shown below on flip chart or whiteboard

	Round 1	Round 2	Round 3
1st Coin
1st Card
All Coins
All Cards

• There are two processing flows that are happening in opposite directions. One for the cards and another for the coins
• Processing steps for cards

Steps to process a card are intentionally complicated and one can invent any scheme do-able in a short time.

• Processing steps for coins is rather simple, however dependent on chance. At step n-1, the instruction is to flip a coin until it turns to ‘heads’. In step, participants flip all coins until the coin turns to ‘tails’. Repeat this loop until the last person in chain is done.
• For each round, you need less than 20 coins and the same number of blank index cards
• Setup and people arrangement (click images to view larger versions):

The motivation behind creating opposite flows is to amplify the bottleneck effect and demonstrate the viability of the Kanban signaling mechanism in intersecting workflows, the same as in reality where people have to switch context and work on a different stream.

Exercise Context:

I set up the context for this exercise by making myself (facilitator) the owner/CEO of this operation. My company produces a magic set which requires a mathematically valid index card setup (see card processing steps) and a coin that has been flipped appropriately. The magic product set must have one card and one coin that make it possible for customers to do magic tricks. (All made up )

There are five team members who are producing the magic set of card + coin and three timekeepers to record production time - one for recoding 1st coin processing time and last coin processing time, second for 1st card processing time and last card processing time, and third for backup and tabulating results during each round.

Round 1

In this round, all coins must be flipped to ‘heads’ by a person on one end of the table before (s)he can pass the entire set of 20 coins to the next person. On the other end of the table, the person who is starting to process cards must process all cards by executing only step 1 (demarcate quadrants) before all cards are passed onto next person for step 2.

Soon enough a situation similar to the one below will arise:

This causes a sever bottleneck mid-stream through the round and lot of wait periods for people when it happens.

Timekeepers tabulate the time taken for 1st coin that completes the flow, the 1st card that completes the flow, the last coin that completes the flow, and the last card that completes the flow.

Round 2

After Round 1, remove the constraint of processing all coins and all cards by a person before they can be passed on the next person in processing chain. A short debrief and retrospective on Round 1 participants will show that they naturally gravitate to select small batch size, in this case the participants chose to pass a coin and card as soon as it was processed by a person.

This is enabling reduced batch size in a PUSH based system since the participants move their processed item to next step as soon as they are done.

In Round 2, the problem of ‘waiting’ is resolved by reducing batch size, however pushing artifacts to next step causes unpredictable bottlenecks to pop up throughout the process. Don’t be surprised if this round feels chaotic with participants egging each other on to complete their steps.

As timekeepers tabulate time taken, it is clear that Round 2 processed the 1st coin and 1st card faster than round 1. Improved time to market!

But as an owner/producer, you know that just being first to market is not good enough - you have to continuously supply the market with your product so that you can keep the market advantage gained by faster time to market. (For those fixing production issues using the Kanban system, simply fixing the first production issue really quickly and demonstrating no predictable rhythm thereafter does not help your customers.)

The stage is now set for Round 3.

Round 3

As a facilitator/owner in this context, there is some new setup steps required prior to executing Round 3.

Setup and Rules:

Each person has two different colored papers next to them, except for the first and last person since they only need one (see diagram above). Each color is marked either for card or for coin. A work in progress (WIP) limit is enforced by ensuring that only one coin or only one card can be placed in on their respective color paper. Only when a participant PULLS an item from the board can the downstream person work to fill the empty color space.

Execution:

As this round begins, the following dynamic emerges:

Reduced batch size eliminates wasteful wait periods as in Round 2. Establishing Kanban cards to enable PULL based system normalizes flow, preventing bottlenecks from building up. In the exercises that I have facilitated, the time for the 1st coin and 1st card processed are extremely close (almost within seconds) and the total time to process all cards and all coins is also very close (again almost within seconds).

I was fortunate to have a video recording for one of these exercises. Following is a selected transcript from the debrief. (Full transcript is 5 pages long and peoples name have been replaced by random one letters, except my name is in full.)

Debrief

Y: We were producing cards and coins at the same pace, evenly in 3rd round. we got the production flow of cards and coins..

Z: We were making money sooner and continually having product available, so we got product sooner, a more consistent delivery..

X: Versus a lot of overhead of constantly keep stuff on shelf. This avoids the problem that we gave you a big batch in January but we can’t get anything for you until march

…….

Dhaval: You said something about predictability almost 3 times, what did you mean by that?

X: Inventory was predictable, it was consistent, we had a process that was going and even though one of them was a random process you still ended up with a consistent output, because the batching system. The Kanban system allowed you to control flow and really no one was idle. Work was evenly distributed

Dhaval: So, ummm.. I liked what you said. This (flips a coin) is sort of 50% chance, it is not wildly unpredictable as some software stuff could get. And you are saying that we could control something even without knowing how much flips you are going to need.

X: Yup!

A: Not everybody is overloaded, you have only two buckets to watch for; it's not like there are hundreds. It isn’t that you don’t have work at sometimes and you are overloaded at other times.

……….

Y: In Round 3, if there is a bottleneck, if somebody is not processing well then this .. this .. signal cascades down the line and makes it apparant, right? (looks around for acknowledgement and gets some) as opposed to let’s say, if I’m the bottleneck and J is tossing over and cards are queuing up it does not become apparent down the production line.

Dhaval: very interesting, so if I restate what you said was in Round 2 the output would have been a stack of queue in front of R but in Round 3 you (1st guy to process cards) could feel the effect of R not being able to process. All this with out having R to explicitly communicate that I’m blocked!

all: Yeah! Yeah!

Y: We could feel the flow, we were all interconnected. Builds cohesion and in the same vein build that dependency between us too.

Summary

There are two things very sacred about using Kanban:

1. Limit Work in progress: WIP could be 1 item or 2 items or at most 5-7 items but definitely not 100.
2. Have a signaling system that implicitly communicates upstream bottlenecks downstream. Never exceed your WIP limit even if this means that you have to idle (wait) for upstream process steps to free up their kanban bucket.

Most Kanban implementations either blatantly disregard their WIP limit or invent new workflows for exception handling which soon becomes the norm. Instead it would serve the organization better to plain and simply idle, twiddle thumbs. Of course this would make the organization optimizers concerned for fear of underutilization. A better alternative is to utilize unused capacity to find continuous improvement experiments that will prevent similar disruption of flow in future (a Kaizen event!)

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

Programming with Groovy: Trampoline and Memoize

by Tim Myer

Groovy 1.8 introduces two new closure functions. Memoization is the automatic caching of closure results. Trampolining permits a form of declarative tail-call optimization. This article introduces the two concepts and demonstrates how to combine them in order to create cached, tail-recursive closures.

tl;dr

The example code from this article is available on github.

Simple Memoization

Creating a closure that caches the result of some calculation is as easy as appending .memoize() or one of the alternate memoize...(...) methods that allows more fine-grained control over cache sizes to a closure declaration. One benefit of memoization includes the caching of results of long-running calculations that have no side effects. Memory leaks are a potential pitfall, which is why a maximum cache size should generally be prefered.
The specification below contains a closure with a side-effect. This side effect happens just once, despite the closure being invoked twice.

SimpleMemoizationSpec.groovy

Recursive Memoization

Suppose we want to memoize the results of a recursive closure call. For example, we can unroll the call tree of this naive implementation of the fibonacci function.

The call trace for the fourth fibonacci number looks like this.

NB: The closure here is entered nine times but we can see that it only needs to be entered 5 times because 4 calls are repeated. If we cache the results of the fibonacci calls, the complexity of even a naive implementation such as this will increase linearly, rather than exponentially.

Unfortunately, because, in Groovy 1.8, a closure cannot invoke another closure directly, memoizing this closure is not entirely straightforward. The example directly below does not work.

RecursiveMemoizationSpec.groovy

The stack trace when a closure calls itself.

Since methods can invoke memoized closures, the solution is to invoke the call method on the closure.

RecursiveMemoizationSpec.groovy

NB: The un-memoized version enters the closure 177 times, but the memoized version enters just 11.

Trampoline

Declarative tail-call optimization is as simple as adding .trampoline() to a closure declaration and ensuring that recursive calls to the closure invoke the trampoline method on the closure instead of the closure itself. Some problems are more clearly solved with recursive solutions, but without automatic tail-call optimization in the JVM, recursion can quickly lead to an explosion in the size of the call stack. Trampolining is one work-around for this design tradeoff (or defect).

A tail-recursive fibonacci closure:
By tracing the call stack, we can see its linear growth without memoization.

In order to avoid a java.lang.StackOverflowError for sufficiently large inputs, the tail-recursive closure must be explicitly trampolined.

TrampolineSpec.groovy

In this example, the trampolined closure is called 1001 times. If a method in Java were to call itself 1001 times, the stack would be overwhelmed. By trampolining, Groovy turns this recursion into iteration.

Memoizing a Trampolined Closure

Suppose we want to cache the results of a computationally expensive tail-recursive closure with no side effects. A trampolined closure that calls itself can easily be memoized in a separate closure declaration.

OneTimeTrampolineMemoizationSpec.groovy

NB: This solution caches the top-level trampolined closure, not the results of the intermediate calls.

Trampolining a Memoized Closure

Suppose we want to cache the intermediate results of a trampolined function call. For example, in our trace above, suppose we want to cache the results of fib 4 and fib 3, 1, 1 and fib 2, 1, 2 and fib 1, 2, 3 and fib 0, 3, 5. Again, this situation is not straightforward because no closure can call a memoized closure directly, so in this case, the trampolined closure cannot call a memoized version of itself directly. Again, we can use the call function on the memoized closure.

FullTrampolineMemoizationSpec.groovy

Conclusion

Trampolining and memoization are two powerful new features in Groovy 1.8, but the combination of the two is not always straightforward. This tutorial has presented strategies for combining the two and for working around some of the limitations of their combination.

 

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com

Managing Risk in Scrum, Part 2

by Valerie Morris

In my previous post, I discussed the five risk areas found on most projects and how Agile addresses them.

Managing risk is prevalent in Scrum on a daily basis.  Discovery, analysis, and mitigation for risk happens organically in Agile, and particularly in Scrum.  Let’s compare risk management practices between traditional/”waterfall” project management and Scrum.

Traditional	Scrum
Risk Management:  work with management and stakeholders to determine what the risk management approach will be for the project *formal documented results	Risk Management:  work with the product owner, delivery team, and scrummaster to determine what the risk management approach will be *no documentation or informal documentation usually preferred
Risk Identification:  identify all risks upfront at project initiation and planning (i.e. natural disasters, divorce, death, etc.)  risk identification is “big planning up front”  (BPUF) *the project manager holds a risk management meeting to review the risk document with stakeholders and project team *the project manager creates this deliverable	Risk Identification:  identify risk on multiple levels: • product vision • product roadmap • release planning • sprint planning • daily stand up risk is identified, and mitigated daily via the daily standup and beginning and end of sprint planning and review meetings *whole team is involved in the multiple levels and through transparency * whole team is involved in Scrum ceremonies and transparency
Risk Analysis:  review all of the risks identified during the identification meeting and perform quantitative and qualitative analysis prioritize risks by performing an exercise of pssibility and probability scoring of every risk *the project manager creates a scoring sheet for all risks and determine which risks to mitigate based on score	Risk Analysis:  agile projects generally focus on qualitative risk analysis because of the sprint time boxes and constant feedback loops provided in scrum *scrummasters help keep the team see the risks and determine what to do next
Risk Response Planning:  develop options and actions for the risks creating the biggest threats *the project manager or a part of the team to create ways to avoid, mitigate, plan contingency, or accept the risks	Risk Response Planning:  happens real-time as risk is identified *whole team is involved in brainstorming ways to avoid, mitigate, contain or evade the risks
Risk Monitoring and Controlling:  status meetings are the forum to discuss new risks and updates to the risk identification list *the project manager facilitates the status meeting that is usually weekly or monthly	Risk Monitoring and Controlling:  transparency of the delivery team’s work via task boards, burndowns, daily standups, and end of sprint reviews provide information and forums for continuously monitoring risk *whole team is involved in risk monitoring through their contributions to the data and feedback loops in scrum

Traditional project risk management is a knowledge area in PMBOK.  Risk management in Scrum is not mentioned as a formal tenet of the framework.  Scrum has enough structure to plan and deliver quality software incrementally with inspection and adaptation.  Scrum gives you front row access to identify and mitigate risk daily.  Scrum does not ask you to be stupid on purpose by ignoring risk.

Visit the AgileIQ Blog and the home for Agile Training and Development, SolutionsIQ.com