Simple Web Development In Java

Teams often are slowed down by the increasing size and complexity of the frameworks they use and the code they write. But do they really need frameworks?

The reason we often choose frameworks is that they feel easy to start with and seem to hide the complexity involved in solving complex problems. Often though they give headaches and are a pain to get rid of.

If the solution to your problem does not require a framework, do not use one. Do It Yourself Simply (DIYS).


Simplicity is the key to brilliance

http://thepresentationdesigner.co.uk/blog/portfolio-item/bruce-lee-slide-2/

Reading the following situations might evoke a feeling of déjà vu.

a. You start a new development effort by assembling 5 to 10 (or more) libraries as a first step. The cost of carrying those libraries proves to be high. It’s not just installing. It’s the overall cost of learning, knowledge transfer, keeping up to date, supporting. It’s all the added complexity.

b. A framework keeps getting in your way. You have to invest of lot of energy to work around that framework. Typically, it gets in the way of test-driven development. The framework feels like a prison: you resist abandoning it because you fear the cost of change will be to high. Your code is too tightly coupled to the framework. Instead of listening to the feedback from your tests, you let go of your good intentions.

c. A framework keeps surprising you with some magical behaviors. It has unintended and inappropriate behaviors. You wish you could explain what’s happening, but cannot. You spend horrendous amounts of time searching user forums for an answer, but cannot find any. The source code is so hard to follow it won’t help either. You browse Google in desperation…

The Little Story

Some time last year, several discussions on the Growing Object Oriented Software, Guided By Tests discussion group inspired me to take action and rewrite an existing Java web application using only simple tools and libraries.

I was curious and challenged to see how I could actually build a transactional web application in Java in a simple way. By simple way I mean without a web framework, an Object Relational Mapping (ORM) framework or a Dependency Injection (DI) framework.

The original application is a sample application used mainly for academic purposes – typically when teaching how to do Test-Driven Development from the outside-in in Java. It uses Spring as a DI framework, Spring MVC, Hibernate as the ORM framework and Maven as the build tool. Velocity views and SiteMesh templates render the HTML. For better or worse, it’s a pretty standard Java stack – similar to what a lot of my consulting clients are using for their day to day development.

The Good, the Bad and the Ugly

There is some good, some not so good and some ugly in the original application.

The Good, the Bad and the Ugly

On the good side, the code is pretty clean and very well tested. It has been test-driven developed from the outside-in. End-to-end tests cover typical usage scenarios. Integration tests with the database cover both data access code, Spring transaction demarcation, Hibernate configuration and mapping, as well as database migrations. Unit tests for the views are done with offline HTML rendering and assertion against CSS3 selectors expressions with custom Hamcrest matchers. The overall code coverage is around 99% when combining coverage from the unit, integration and end-to-end tests.

There are also those areas were it could have been better. The domain model does not reify the concept of user requests, which results in some web controllers doing too much. Startup cost of the database integration tests is noticeable because they load Hibernate and a trimmed down Spring context. Assembling the application for running end-to-end tests is a bit tricky. Startup cost of the application also makes the end-to-end tests somewhat painful to run.

Finally, there are the real pain points. The Maven build is one of them, as always. It packs more than 1100 lines of XML for a rather small application! Configuring the application for running the end-to-end tests requires overriding application configuration with test settings via system properties override in Spring. End-to-end and integration tests use static contexts to avoid loading spring and hibernate contexts multiple times. Last, but not least, the WAR file weighs in at 53 jars, for a rather small application.

The Challenge

For the rewrite, the challenge was to eliminate the pain points and to follow these guiding principles:

  • Use only simple tools and libraries that are tailored to the problem at hand
  • No framework, all libraries should have a single responsibility – doing only one thing and doing it well
  • Do It Yourself Simply (DIYS)
  • No XML
  • No annotations
  • Minimum startup and shutdown cost – so that tests can run fast
  • Ease of assembly and deployment – so that tests are easy to setup

The new version of the application is built on top of Simple for the webserver, standard JDBC, Mustache Java flavor for the view templates, and my own small command line interface library. Everything else was test-driven developed. That is:

  • Object assembly,
  • Data mapping,
  • Transaction management,
  • Environment configuration,
  • Routing,
  • Serving static assets,
  • Access logs,
  • Layout templating

That new version does not follow the Servlet specification. It uses a simple web server and builds as a single jar with embedded configuration. Most things that were left to frameworks in the original version – namely transaction demarcation, objet assembly, wiring, routing, and database mapping – are done programmatically in the rewrite.

The Benefits of Simplicity

Built for Speed

One of the things I like the most about the new version is how fast it starts up. Startup time is down from around 6-7s on my machine for the Spring application to a negligible amount of time for the Simple version. That small startup cost makes a huge difference. It means the end-to-end tests run much faster. It also means re-deploying the application results in almost no service interruption.

The application is self-contained – it is packaged as a single jar – and has no external configuration. Configuration is bundled with the application – including test configuration. As a result, the end-to-end tests exercise the entire application through its main entry point, which results in greater test coverage. Self-contained configuration also makes it very easy to setup, startup and shutdown the application from the end-to-end tests. That means the application can be easily installed as a daemon as any other Unix tool.

Tests are blazingly fast, and that too makes a huge difference. End-to-end tests run 50% faster. They are down from 18s to 12s using Firefox and only 5s when run with PhantomJS. Database integration tests run under 1s compared to 3s in the original application. Configuring the tests for running on the test database is now straightforward. The best part is that database integration tests run so fast that it is no longer necessary to maintain a static test context. There is no penalty to bootstrapping the data mapping code on every test.

Doing simple has also tremendous benefits in terms of total cost of ownership of the application:

  • The cost of understanding the tools and technologies is low, which makes getting new team members up-to-speed much easier
  • The increased startup speed has lots of benefits on testing and solves re-deployment headaches
  • Smaller codebase size means reduced maintenance and communication costs
  • Simplicity of the design means cost of change is kept low
  • The explicit programming model makes codebase easier to follow and debug
  • All-in-one packaging reduces cost of deployment and configuration

Final Thoughts and Takeaways

That being said, the solution is obviously not very “enterprisy”. One legitimate question is to ask is whether there is anything in the frameworks that we would be missing. For example, we could question how the solution might scale up. One the other hand, if you rely on a large framework and at some point late in development you discover that the performance – memory consumption, reliability or anything else – is not as good as expected, you’re … screwed. There’s not much you can do except trying to rewrite parts of your application without the framework. On the contrary, if you develop your application incrementally with no framework and you keep it simple, then if at some point it’s not fast enough you are in a good position for applying whatever optimization is needed.

For some teams, using large frameworks is a belief that they’d rather use a proven solution than develop their own. If chosen well, they have a tested and stable component that works. They might wonder if they have the skills and knowledge to develop their own solution. Will it work as expected? How do we know for sure? How easy can we make it evolve as our needs change? How reliable will our solution be? Do we have the skills to come up with a good API design? etc.

Another question that arises is that you probably want to avoid solving problems already taken care of. If you can find a well-tested, well-crafted library that is easy to use and addresses the problem you’re trying to solve and just that problem, and if that library has minimal to no dependency, you might as well use it. Where to draw the line between doing it yourself simply and using an existing solution is a judgment call.

For the rewrite, I ended up using only few off-the-shelf libraries – mainly one for the web server, one for rendering HTML templates and one for parsing command line parameters. I chose libraries that I liked and that solved a single problem in a simple and elegant way – no less, no more. When I could not find a suitable library, I followed the Do It Yourself Simply principle and coded my solution. Some problems are not hugely difficult and you can solve them easily with few lines of code. As an example, the code that replaces SiteMesh in the new version of the application is no more than 100 lines long.

To avoid reinventing the wheel you can scan through the conventional framework code looking for special case handling that you wouldn’t have thought of. Also look at how the framework solves the problems you’re already aware of, to draw on as many good ideas as possible. The point is to steal the good ideas, and rewrite just the code you need in a simple way. With mastery of test-driven-development, you will be able to incrementally add as you need more. Of course, be careful to avoid turning your own solutions into large frameworks.

One last consideration is that frameworks allow you to do so many things quickly and that’s a very good point. One situation where it might be desirable to use a framework is when you don’t care so much about the code, for instance to do a proof of concept for a product idea.


I believe there is often an illusion that frameworks will help much more than what they do in reality. You tend to want to use more of what the framework has to offer to keep your project in the same ecosystem. You often have to twist your code until for it to fit the framework. You end up paying a hefty price for the added complexity, difficulties in test automation, learning curve, keeping up to date and upgrading. All of that adds to the cost of ownership of your codebase.

Staying away from large frameworks as much as possible will reduce your risk and increase your options. You can still add a framework later if you find you really need to.

Simplicity is the best way to keep your agility: speed of delivery and ability to change.

Simplicity is the ultimate sophistication. – Leonardo da Vinci

My blog is moving

I’m moving my blog to a new domain. I will now be at http://vtence.com.

Please update your bookmarks and feeds.

See you there!

Focus on solutions, not problems

One of the key ingredients to becoming a great team is learning to focus on solutions, not problems. Unfortunately, this is really hard and requires practice. As humans, our natural response is to do just the opposite. In this post, I will explore why this is so and share things you can do to help.

We tend to focus on problems

Over the years, I have trained teams to become great teams that build great software; I have coached product owners, managers and executives to adopt Scrum and move away from predictive thinking to empirical management. During my coaching work, I have consistently observed the natural tendency people have to focus on problems rather than solutions. If you take a moment to think about your own experiences, you might recall situations like these:

  • A meeting without a clear goal in which problems are discussed at length, even if that leads nowhere closer to a solution;
  • A Daily Scrum during which team members share the work they did, the technical difficulties they encountered, but fail to create a plan for the next 24 hours;
  • A one-on-one meeting in which a manager provides « feedback » to an employee, identifies weak spots and asks for improvements;
  • A code review session during which the reviewer describes what he does not like about his peer’s code.

It does not matter whether you are a CEO or a software developer, we all suffer from the same brain bugs. In order to become more efficient, whether as a team member or individual, you need to learn how to overcome them. To do so, it helps to understand what those brain bugs are and where they come from.

The brain tries to maximize rewards and minimize danger

Our brain has one overarching organizing principle:

It tries to maximize rewards and minimize danger.

It also registers negative emotions much stronger than positive ones. This phenomenon is known as the Negativity bias. Negativity bias is pretty easy to explain considering that our brain has evolved over thousand of years and that during most of that evolution period, surviving amongst predators was the only thing that mattered. Our brain has been optimized to react quickly and efficiently to threat responses, even though we do not always react the way we would like to.

Threat responses generate negative emotions. In his latest book, Your Brain At Work, David Rock explains that negative emotions put people in what he calls an away state. Away responses make people defensive and reduce their cognitive resources. It becomes harder for them to think clearly.

The brain is a prediction machine

Our brain is also a formidable prediction machine, as Jeff Hawkins describes in his book, On Intelligence. It is continuously trying to predict the outcome of future actions to make decisions. Considering this, you can understand why your brain hates uncertainty and actively seeks to reduce it. Uncertainty leads to the inability to make decisions. Too much uncertainty often results in being stuck in a decision process. The more complex the situation, thus the more uncertainty, the more your brain has to focus attention and process stimuli, an energy hungry process. Your brain hates to do that.

Uncertainty creates a threat response. So there is a natural biological response to reduce uncertainty.

The past has lots of certainty and the future has little

Unfortunately for us, the future has very little certainty. It is scary to the brain. This is why so many leaders look for definitive answers and have such a hard time with empirical management. They prefer to live in the illusion that predictive management can actually work.

On the other hand, the past is all about certainty. It is about facts, things that are known. It is much easier for the brain to talk about the past than make decisions about the future.

Consider now that solutions are generally untested, and thus uncertain. They live in the future. It takes effort for your brain to dampen down the threat response that comes with uncertainty. On the opposite, it is very easy to discuss problems, because they come from the past.

Some things you can try

Next time you end up in a meeting, pay attention to how people tend to talk about problems, rather than focus on solutions. Here are some things you can do to help people put the focus back on solutions:

  • Simplify the situation;
  • Consider the whole;
  • Minimize perceived threats;
  • Take a break.

Set clear and concise goals

You can simplify situations by stating clear and concise goals. To be helpful, a goal has to be stated using a simple language that is common to the team.

Simplifying the situation using a goal statement will help people generate insights (see The Eureka Effect) and move them toward solutions.

Consider the big picture

Consider the big picture (for the system thinkers out there, the system), rather than the details. Take a step back and then focus only on the parts of the system that are relevant to achieving your goal.

On that subject, have a look at that great talk from Eric Berlow: How complexity leads to simplicity.

Minimize threats

Be careful to minimize perceived threats. Remember that negative emotions are registered much stronger than positive ones.

You can minimize perceived threats by discussing the positive, rather than the negative. Discuss things that are working well, what’s good in what you are doing and engage in a dialogue about how you would like things to be.

Take a break

It is surprisingly easy to get stuck on a problem. This is known as the impasse phenomenon (The Eureka Effect).

Whenever you feel the group is stuck, take a break and do something light and interesting, to see if an answer emerges.

100% de couverture de code par les tests

Voici les slides de la session que j’ai donné à Agile Tour Montréal 2011 et à Confoo 2012.

L’idée maîtresse est de considérer l’objectif de 100% comme un objectif asymptotique et un état d’esprit. Cet objectif amène l’équipe à continuellement se dépasser et à améliorer entre autres, ses pratiques d’ingénierie, ses outils, son infrastructure.

Le résumé de la session est le suivant :

S’adonner au développement itératif et incrémental sans automatiser les tests, c’est s’engager sur la voie du Scrum Flasque (Flaccid Scrum pour reprendre l’expression de Martin Fowler). C’est voir petit à petit la vélocité de l’équipe diminuer, la dette technique s’accumuler, la livraison de valeur d’affaires cesser. La base de code devient progressivement intravaillable. Bref, la qualité se détériore et les coûts de maintenance de l’application explosent. L’automatisation des tests à tous les niveaux est essentielle pour livrer itération après itération du logiciel de qualité qui enchantera vos clients. Plus facile à dire qu’à faire, à priori. Et quelle couverture de code devrait-on viser ? Seulement le chemin nominal ? 70% ? … ou bien 100% ! Certains diront déjà, 100% c’est irréaliste ! Au contraire, cette session vous présentera comment viser … et atteindre une couverture de test de 100% en tirant profit des différents types de tests et en construisant vos propres outils de tests.