Domain Specific Languages

This is the last day of the conference part of JAOO, tomorrow is a full day tutorial.

Today I've been mostly listening to talks about Domain Specific Languages. This is a notion that kind of is the next step from Domain Driven Development, but also requires something more to quite get there. I think the stuff that we're trying to do with DDD is the closest thing we can get to an "internal" DSL in Java (to use Martin Fowlers classification), without actually making a full-blown DSL.

Now, what is a full-blown DSL? A DSL is a specific syntax, typically for a small, well defined domain, aimed at making the software more expressive. For a more through description read Martin Fowlers article on Language Workbenches. Usually, a DSL, whether internal or external, will imply some kind of transformation into executable code, so in that sense what we do directly in Java is not exactly DSL by strict definition. Even so it is possible to make the argument that it is.

Language Oriented Programming is a term used on the style of programming promoted by the Language Workbenches and similar technologies. LOP uses multiple DSLs to create full programs. This is analogous to Eric Evans notion of different domains within an application.

Another topic that propped up towards the end of the DSL track as a side effect is our use of the word Model and what this means to different people. And that will be the subject of my next post.

The fundamentals

If there is one recurring theme from all the conversations I've had with people at this conference, it is that it is vital to understand the fundamentals of a problem before we throw ourself into the technology discussions.

How can you choose one technology over another if you don't know the fundamental issues that should guide your decision?

Anyone interessted in the fundamentals of programming languages and language design should check out Abelsons and Sussman's lecture series Structure and Interpretation of Computer Programs.

Ivar Jacobson on Software Agents

Todays keynote by Ivar Jacobson was entitled Beyond Agile: Smart.

Ivar, of course, promotes the unified process, and especially the practice of making knowledge explicit. This is, he says what makes his process better than agile methods. Dave Thomas commented to me that it is easy to arrive at such conclusions when you don't have to do it in practice.

Anyway, the main thing that got my attention (of course) was what Ivar calles the next thing. His idea is that all the information gathered and formalized by (R)UP is a good thing as long as it stays out of our way and is available when we need it. This is an interessting proposition, albeit a bit far fetched. Ivar is actually talking about (and have started developing) intelligent agents to be responsible for this information distribution. He is using software agents as virual pairs to suggest options for you and learn from what you are doing. How about that?

He envisions a set of tools that will let agents play different roles, such as

• virtual pair project manager
• virtual pair architect
• virtual pair designer
• virtual pair developer and so on...

If you are more interessted in his work on this have a look at the website of his company JacZone.

As far as I can tell, this is all foilware, but it is interessting that these kinds of things pop up.

What are the key things with patterns?

A pattern is a description of a solution to a recurring problem in a given context. The format is not important, but it should include the following sections:

• Name
• Context
• Problem description
• Forces and tradeoffs
• Solution

This gives us the possibility to name recurring solutions and communicate the different aspects of them, which I think is a good thing that we should do more.

This is also a sensible (and perhaps the only usable) way of organizing any technical writing. If you don't cover at least the three last bullets in that order in your writing it will be very difficult to understand what you are trying to say. Providing a context helps and giving it a name adds to the communicativeity of the concept.

I think we should start looking into better ways of communicating across projects and organizational boundaries. Maybe using a common repository of short presise descriptions of our learning. Adding to this repository should be part of the project end life cycle.

Patterns in Software

I spent the rest of the day yesterday listening to different talks about patterns. Most of them talked about what has happend since the GoF book (Design Patterns, 1995), and what they think will be the future of patterns.

The track started with a talk from Ralph Johnson (one of the Gang of Four). He gave us his personal take on the history of patterns up to today. After him, Dragos Marinescu from ThoughtWorks talked about the new book Pattern-Languages of Program Development volume 5, where a new collection of Pattern Languages will be published later this year.

Markus Völter then talked about Patterns in Modern Software, Klaus Marquardt talked about how to improve the usability of patterns, and finally Kevlin Henney and Frank Buschmann talked about their take on the past, present and future of patterns.

The main trend in the pattern community is the move of focus from single patterns to pattern languages (interconnected patterns). Kevlin and Frank will publish a new book in Frank's series Pattern-Oriented Software Architecture, volume 5, later this year. In this book they will connect patterns from the GoF book, patterns from Fowlers EA Patterns book and other new pattern languages.

Those of you more interessted in patterns can check out the Hillside Group. They are the group that run the PLoP conferences.

To me patterns is a little bit off in my peripheral vision at the moment, especially when I'm still so geared up from the agile stuff on Sunday. I do however think that there are some ideas in the patterns movement that we should embrace in Statoil as well and start to use for better communication both internally and externally.

Opening keynote: Simon Phipps on Open Source

Simon Phipps is the Chief Officer for Open Source at Sun Microsystems, which makes him a guy to listen to and with power to change stuff in the open source space.

He talked about what he calls the "Participation Age" to describe the era we're in now. In this era open source is the model for building software. But open source is not enough. Standards must be in place to enable the companies to add value to the open source commons. Many business models are possible around open source.

Also interoperability is required for the users to be able to achieve the freedom that open source is all about for them.

A very interessting talk, that for me at least opened some new perspectives on the open source space.

First day of the JAOO conference starting

Yesterday was the pre-conference tutorial. Today the conference starts in earnest. Multiple tracks with many different interessting speakers. Today I will mostly follow the Patterns track, so we'll see what kind of ideas my head is filled with tonight...

Fundation for adopting agile methods

Dave Thomas made this point repeatedly: If you don't have an infrastructure for doing continuous integration and automated tests then you can forget about transitioning to agile methods.

This is good news for us, since we're there and have been for some time. I personally think it has been the right way to go, introducing the foundation and gradually maturing towards a better understanding. I think understanding must be there internally before we can bring in someone from the outside to show us the way. I feel that we are ready now to move further with agile methods and start to challenge our document driven ways.

So, what am I advocating here? The death of PROMIT, certainly not, but it needs to be reworked seriously. PROMIT is derived from PROMIS. Why do we have PROMIS? To manage huge, costly oil field development projects with huge amount of infrastrucure costs. The cost distribution of these projects are low in the specification phase and extremely high once the infratructure starts to be implemented. On these projects it makes sense to have the decision gates to control the costs before implementation, but this doesn't translate into software development.

Software development cost distribution is no different in the implementation phase than in any other time of the project. The cost depends on how many people and how long it runs. The best way to control this cost is to set a date (fix the time), set the teams (fix the people) and let the scope vary within reason.

Scaling Agile Development

Robert C. Martin (aka Uncle Bob) is a carismatic speaker. He is clear and consise in his descriptions and entertaining in his presentations. I've obtained an electronic version of his talk and can provide this to those who are interessted. Just send me an email.

The talk was split into two parts. The first half of the day uncle Bob spent going thorough the different principles and practices of agile development. In the second half, Dave Thomas talked about the management view. If you don't know Dave Thomas (not the pragmatic programmer), he is the one who managed the Eclipse project starting with IBM Visual Age, so you know he knows a thing or two about scaling agile development.

Bob started off by talking about measurements, under the banner in order to manage we need data. I'm a little embarrased to say that this surprised me, because that means that I have bought into the misconceptions that is going around about agile development that it lacks dicipline. Bob uses story points (or function points) to measure what a team can perform in a week, and then accompany this with a chart showing the remaining function points. With this you can estimate your delivery date. We'll come back to how he comes up with these function points.

He then went on to talk about the fact that in most projects the delivery date is what is fixed earliest, and the requirements never are and never should be.

The problem with the waterfall and other document centric approaches (apart its history) is that it lets you spend a lot of time in the start of the project without producing tangible results. Yes, a requirments document is tangible and sometimes even helpful, but if you spend the first seven months of a project doing requirements and only have three left for implementation it will fail no matter how good the implementation team is. The problem is really that there is no way to determine if a requirements document is done, or when a design document is done. It is however no problem to define when an implementation is done, and hence the problems in the end. Also the practice of putting in the quality at the end is doomed since you rearly have time to do a good job when the implementation fase slips.

He then proceeded to talk about agile development and the different practices or principles in it. Now I don't want this blog entry to turn into a tutorial on agile development, so please follow the link above for an overview of agile development ala Robert Martin.

Finally, Bob, did a programming excercise to show test driven development. He set out to write a bowling scoring game (from his book). The point of the excercise was to show that you will arrive at a different and simpler solution that with up-front design. First we did a collaborative design and arrived at classes like Game, Frame, Roll, and Tenth Frame (to handle the special 3 roll possibility of the last frame). Bob then starts from scratch with TDD approach. Simple creation test for Game, first test case all rolls scoring zero (gutter game), nest test all rolls scoring 1, next test one spare, next test one strike and final test all strikes. This produces one class Game with two methods roll(), which registeres the pins taken down in each roll and score() which very accurately captured a clear bowling scoring algorithm. It was interessting to watch. I actually didn't realize he had finished.

Then we had Dave Thomas come in to tell us about scaling and how to fit in with management. Basically his statement was, give them whatever you have to to keep them happy, generate requirements documents from your source or Fitness cases, generate the plan, fake it if you have to to keep them off your back. Internally teams scale into teams of teams with aggregation of the practices. This comes with the price of longer time intervals between control at the top level, so you need to be able to trust the people running the teams and that they can perform. The team structure can be evolved starting with a few people or architected from the start. Stories at the top level are broken down and divided amongst the team according to the structure. This is not so relevant to us, as we don't have to scale that high, but some of it could be useful.

To sum up, this sessions answered a lot of questions for me personally, and I'm going to take with me these new ideas when we look at how we can take a step furter and improve the way we run IT projects.

Agile Development according to Uncle Bob

Uncle Bob preaches pure agile development XP style, so those of you who know XP will recognize the practices:

Team level practices:

• Planning game
• Acceptance tests
• Small releases
• Whole Teams

Development practices:

• Simple design
• Test Driven Development
• Refactoring
• Pairing

In addition there are four intermediary practices:

• Continuous integration
• Collective ownership
• Sustainable pace
• Metaphor

An agile project starts with a meeting where the overall project parameters are defined. This most often starts with setting the delivery date, but the date is not always there from the start. In this meeting the overall functionality of the project will be discussed and start to be formalized as stories. Each story represent a function that has business value and that can be estimated and tested.

Bob use a deck of index cards, one for each story. Cards will be added and removed as the project moves on. First each story gets an estimate. The estimates are arbitrary numbers sigifying the relative complexity of a story. These numbers constitute the function points or story points used to plan and measure project progress.

An initial plan is then made laying the stories out in weekly iterations. The initial number of story points in an iteration is a guess and depends on team experience amongst other tings. If a team starts out trying to do 50 points in a week and on wednesday only 22 are done, the customer is promptly told that he can expect only 44 points to be finished by the iteration and so on. When the iteration ends, the final number of points done is counted, put in a graph to show progress and a new plan is created based on this new knowledge. The customer will select the stories that are most important at the beginning of each iterations, and stories can be added or taken away. The project is done when the customer cannot find any new stories that are worth implementing or the delivery date, whichever comes first.

They also do an initial two weeks of analysis into the problem domain. This is consistant with what Eric Evans recommends. This is when the requirement harvesting begins, and this phase could even produce an initial object model. The important thing is that this model will be discarded when the implementation starts.

So when is a story done? This is down to the acceptance tests. When a story passes all its acceptance tests its done. Acceptance tests are automated using Fitnesse or another framework, and run automatically. Ideally they would be written by the customer in Fitness, but most often it will be Business Analysts writing the "happy path" and QA people and testers writing the exceptions and boundary conditions.

How do they handle project slippage? When you measure function points done per week and function points remaining, after around five weeks the number of points done per week in average will be acurate enough for predicting a delivery date. This is still pretty early in the project, about seven weeks into the schedule we can tell the customer that the end date is unrealistic. This is unheardof in waterfall projects, where the norm is to have this information in the end at best when the implementation phase is reestimated. Of course the customers will still complain about slippage, but when you discover this early it's possible to do something about it. Most often the date will stay fixed, adding more people can be an option this early, and it is never a good idea to compromise on the quality. What is left is the scope, and the story deck needs to be reshuffled and reprioritized.

User demonstrations after each weekly iterations and fully functional, ready-for-production releases every six weeks. This means that if the project is cancelled after one of these releases, it will provide value to the customer. Also it allows the customer to terminate the project before the end delivery date if he cannot find any more stories worth implementing.

Whole teams refer to the practice of co-locating the whole team in one room. This includes business analysts and QA people, project manager and testers.

The lower level practices are more known to us and we have implemented most of them in the course of our work with Eric Evans on Domain Driven Design. To me these team level practices are what we need to complete the picture and help us avoid project failures in the future.

Why do we do the waterfall?

The reason that the waterfall and its spawn document centric, specify then build processes has become popular has mainly to do with a paper by Dr. Winston Royce in 1970 where he began with an ideal proposition saying "wouldn't it be great if we could do this", and basically went on to describe the waterfall model. This is the most cited paper from people talking about the waterfall or similar processes.

What they mostly fail to mention is that Royce went on to saying that he didn't think that this was possible and that he instead recommended an iterative approach. Because what he found is that there will be feedbacks from development into design and from design into spec once you start working with the implementation.

The reason the waterfall turned into such an international hit is that the US Department of Defense asked a tech writer to compile the standard known as 2167, and when that was approved, countries and companies worldwide said "if the DoD is doing it it must be good."

Since then both Royce and the tech writer has, according to Bob Martin, spent their entire time trying to convince people that this was not their intention and appologizing for their errors. Even Royces son has been plagued by this, said uncle Bob.

Robert C. Martin on Agile Development

I'm really happy that I signed up for this tutorial.

Bob Martin is describing his take on agile development, and in addition to being very inspirational, it really provides some missing pieces of the puzzle for me.

I will write a thorough summary of this later today, but for now I'll just say that this means a lot of changes to how we should run our projects. It effectively means the death of the requirements face as we know it. If we don't do this we will not be able to deliver our projects properly...

I'll leave you with that for now, and hopefully I'll be able to justify this statement in my post this evening.

JAOO 2005 conference has started

I've just finished my breakfast, and am waiting for the start of the pre-conference tutorials. Registration should open any minute, and I've spent the time getting up and running on the conference wireless zone.

Today's session should be good, it's with Robert C. Martin and Dave Thomas (not the pragmatic programmer, the other one...) on Scaling Agile Methods. Now if you follow this blog you can expect me to rant a lot about agile methods in the next couple of posts, so if you don't know anything about it please have a look at the Agile Manifesto. I really think everybody should know at least what's on there ;-)

If you look at that site and see the names of the people involved, you'll find Robert C. Martin in addition to other well known names such as Martin Fowler, Kent Beck, Alistair Cockburn, Ward Cunningham, Dave Thomas (the pragmatic one, actually) and Andrew Hunt.

JAOO 2005

It's time for JAOO 2005. There is a lot of interesting talks and sessions, and I want to experience as many of them as possible.

I will do my best to report on the different sessions that I go to and any interesting events that I vitness.

I will attend a tutorial with Robert Martin (aka. Uncle Bob) on agile methods, and definetly the track on Domain Specific Languages on Wednesday. I'm also signed up for a tutorial on JDO by the creator Craig Russel, which should be good. Other than that I'll play it by ear, and pick what catches my attention and time permits.

Interesting individuals @ JavaZone

One cool thing about going to a conference as a speaker (besides getting the chance to speak in front of people ;-) is that you get to meet a lot of interesting speakers. This year I've met both old friends of mine from school like Knut Magne Risvik, and more famous guys like Kevlin Henney and Tom Gilb.

Also I would like to say that the level of people speaking at JavaZone is generally very high, and for the record I'm now talking about the other guys, not me. The local guys have a lot of insight into the different subjects that they contribute.

The thing about people like Tom Gilb, Kevlin Henney, Eric Evans, Bruce Tate, Rod Johnson and our own Einar Landre is that they have a more fundamental understanding of the problems we face in software development. This allows them to think beyond the mere use of tools or methods, and see the "big picture". Now I'm decidedly nowhere near their caliber, but I find listening to their discussions both inspirational and educational.

Einar and I had a brief discussion with Tom Gilb about his talk on Agile methods and what is missing from them, namely metrics and improvment processes. His talk was unfortunately parallel with mine so I couldn't participate, but I got his slides, as well as a copy of his new book titled "Competitive Engineering" and I'm looking forward to reading them both.

Another thing that strikes me after having met these people is that they are generally extremly pleasant and forthcoming people. I'm just sorry I had to come home tonight and couldn't stay for the second day. Well life's full of choices and I really need to spend a little time with my family now. So JavaZone 2005 is over for me, we'll just have to wait for the videotaped sessions and presentations to become available.

My impressions so far

The focus in the sessions I've been to so far (Rod and Bruce) has been about the future of Java development. They both talk about new frameworks stretching the Java language and lightweight development. Personally I think this is crucially important.

We need to always consider the right tool for the right job, irrespective of whether it is open source or not, and get rid of our ridgid development practices. We are doing this now, by introducing things like Spring into our development and using Domain-Driven design. But we still have a long way to go...

Our primary challenge is to tailor our adoption of agile methods such as TDD and continuous integration with the PROMIT project model. We need to provide the same level of control that we had before, only do it better and more lightweight. This is a heavy challenge.

Our second most important challenge is to get rid of as much of our in-house developed framework as possible. Our JEF framework is mostly for GUI, and since things are happening here, we need to watch this development and be a part of it. In this process we need to look at how we can do things just a little smarter than we do today. I think aspect oriented programming will have to be a part of this.

If we are able to solve these to primary challenges we will have come a long way.

Bruce Tate - Lightweight Development Strategies

How do you develop things in a lightweight way.
Java and java development has to get simpler, more reliable and more efficient or it will die.
Java bloat can be good. We need the new frameworks, but we need to be able to remove the deprecated stuff. The success of java and the shrikwrapped product acquisition model is forcing bloat. We need a new software model like subscription.
Don't sell processes, sell principles:

• Small, smart teams
• Good feedback from customers
• Quality
• Simple requirements management
• Simplicity
• Refactoring
• Automated unit testing

Management fears agile development because they think it lacks dicipline. Agile is automated testing, continuous integration, refactoring and use of customer feedback. Make sure that you balance out the formal things you take out...

POJO programming model is stretching java in that you add services through the use of Spring and other framework and the dependency injection principle.

AOP are good for removing cross-cutting stuff like transactions and logging from the object. Spring provides AOP with "training wheels".

Strategies for injecting behavior:

• Code generation (falling out of fashion)
• Byte code generation (runtime - dynamic proxies, compile time - injection)
• Source level metadata (annotations)

Java limitations is mainly that it is a rigid language, how to express configuration? Using Java or XML. What about metaprogramming in Java.

Callbacks can be used to simulate closures in places where they provide benefit, like JDBC, transactions. Continuation in Java with things like WebFlow, Rife, Cocoon2 or Lakeshore. More on these topics in the Beyond Java talk tomorrow.

We need code blocks (closures) and continuations in Java.

Spring will replace EJB, annotations and AOP will get us into trouble in some points, but in the end will survive, java metaprogramming will increase, REST wil slowly subsume WebServices (except maybe SOAP), one dynamic language explodes (Ruby?)

Rod Johnson - Trends in J2EE

Rod Johnson is talking about the current state and the future of J2EE. Main message is that J2EE is in better health than we thought a couple of years ago. Diversity of open source is a strength, not a weakness.
Focus on creating a real object model (reference to DDD and Eric Evans) instead of "fake" objects like transfer objects. The J2EE space is starting to get back to real object oriented designs, but it has taken some years.
A real design pattern is not technology specific.
Agile J2EE :

• Acceptance of change
• Simplicity
• Testability
• Rejection of the waterfall

Framework oriented development:
The in-house framework is dead because there are more and more available 3rd party frameworks. Problem is that many of the best developers have been tied up in technical work when you need to use them on the problems of the business domain. Should use as little effort as possible on in-house framework.

Moving back towards a POJO programming model. No API means stability and backward compatibility. Aspect oriented programming and open source frameworks promote this. AspectJ 5.0 will be the definitive AOP solution moving forward.

Aspects will be integrated in Spring.

That's it for the short referendum. Will post an evaluation later.

JavaZone 2005 has started

I'm sitting here to listen to Rod Johnson talking about Trends in J2EE. This is the opening session for me.

Great book on agent systems development

I've been reading this great book on developing software agent systsms. It's called "Developing Intelligent Agent Systems: A practical guide", and is written by Lin Padgham and Michael Winikoff of the RMIT university in Melbourne Australia.

This book details a methodology called Prometheus that organizes the design and development of agent systems. What strikes me is that this the elements of this methodology can easily be transferred to other types of software development, especially enterprise applications.

The starting point and central theme of the Prometheus methodology is to identify System goals from Scenarios, and then refining these goals into functions or capabilities. This is a simplified description, of course, but this maps to any fairly complex software development where you need to keep track of the overall functionality of the system in a structured form.

Apart from this, the book has a brilliant introduction to software agents, so it's well worth a read for that alone.

Language as the enabler for thinking

Yesterday I was involved in a very interesting discussion about language. We're currently doing a workshop with one of the guys from Eric Evans company Domain Language on implementing the Domain Driven Design methodology in our organization.

A couple of us went out to dinner during which the discussion naturally touched upon language and thinking. Part of the reason for implementing your code using the language of the domain, which is the core of DDD, is that it enables you to think more clearly about the problems in this domain. As it turns out, research with children points to the same thing, that their ability to think and remember things are related to their language skills.

I think this is an interessting input into discussions about using Domain Driven Design, and not to mention the new wave of discussions about Domain Specific Languages. There is going to be a separate track on DSLs at JAOO in Århus later this month, so I'm looking forward to following that .

JavaZone 2005 - Oslo

I'll be attending JavaZone this year. I will give a talk about Software Agents and how they can be used to build more dynamic enterprise applications. If you're going to JavaZone this year and are interested in the subject please check out the presentation outline.

I hope a lot of you drop in and listen to this talk, as I feel this is an important topic for current and next generation enterprise applications. Software agents is in itself a very interesting subject, so I don't think you'll be disappointed.

Java 5 Generics

I must say that I was a bit surprised when I read about Bruce Eckel and his problems in getting the grip on Java Generics in his blog, where he's saying that even he finds Java 5 very complex and has spent a lot of time writing the new chapters on Concurrency and Generics for Thinking in Java 4th ed.

I'd heard so many good things about this before and so I decided to check this out for myself. I readily admit that the prospect of getting rid of these incessant casts in iterators gets me a little excited. But when I started out using Java 5 on this little home project of mine, I kept getting dragged of into side issues with this thing. Maybe its my design, or the fact that I'm just starting out with generics, but there are so many little things to watch out for that it is easy to get lost in the technical issues of it all.

I'll admit that I haven't really spent enough time to form a solid opinion on this, but I don't feel like it is coming together as well as I'd hoped. Will report more once I've had some further experiences.

My first entry

This is my first blog entry. I'm a late adopter of this form of communication but I've been following the scene for the last year or two, and have decided that I want to be a part of it.

My main interests at the moment include Agile methods with special focus on Domain Driven Design, Event Oriented Architecture, software agents in addition to my love for programming (mainly Java). Recently I've started to learn Ruby, and I must say that I like what I see. This is cool.

Well, I guess I'll be putting in my two cents from time to time about these subjects, so look out. I also plan to use this space to report my intake from any conferences I might attend.