I am nearing completion on a large-scale ORACLE upgrade. The project goes live in just a few days. One of my last requests to the technical team was simple and low-tech. I asked that they each keep a journal that starts when we begin the four-day to live process and ends after the production system stabilizes. I had introduced this concept to a smaller subset of the team the day before. Based on their reactions, I figured a more detailed explanation was required.
During the stress of a time constrained go-live, people make many mistakes. Investigating a mistake during a go-live is stressful. When you add stepping backward through a process and getting the chronology right, the stress level shoots up even higher. A work journal is the tool for that. It captures the little things that might not make it into the online tools. Additionally, when it is time for a postmortem, having a journal to refer to is often the difference between guessing and knowing.
The requirements are simple:
- Use a notebook dedicated to the project
- Write down each step
- Make sure to note problems
- Capture details, like time, impact, scope, and the people who worked the problem
- Start each day with a new page
- Write with the awareness that somebody else will read your work
Use a notebook dedicated to the project – The reason I ask each person to start a new dedicated notebook is simple. I want the notebook for the post mortem. Consider it like a school writing assignment. The notebook is the written record of you involvement in a project at its simplest level. Having one enables problem solving and the identification of ineffective processes. If disaster strikes, these notebooks often hold the key to unraveling the mystery of a failure.
Write down each step – Many of our tools required that we document our steps as we take them. This rule is different. Consider it a sequential record of time spent working on a project, but not a detailed explanation. A typical entry looks like this:
- 6/29/2011 1:51 PM: Started working ticket 901120.
- 6/29/2011 2:11 PM: Completed ticket 901120. No problems encountered.
- 6/29/2011 2:12 PM: Took 20-minute break.
Make sure to note problems – This may seem obvious, but keeping notes when a problem occurs is not second nature. A problem is anything that does not occur as planned. In this case, a complex upgrade, each step in the upgrade process has a baseline time estimate. If a step in the upgrade was schedule to run at 40 minutes and took 60 instead, that is a PROBLEM; note it in the journal with details.
- 6/29/2011 1:58 PM – step 14a was scheduled to run at 40 minutes. Actual run time was 60 minutes. The job ran in conflict with another job (No. 34356) resulting in a slow process. I contacted Martin about the delay and reported the problem to the PM. The delay will affect the overall timeline. User tests scheduled for day 3 are rescheduled from 3:00pm to approximately 3:30 pm.
Capture details, like time, impact, scope, and the people who worked the problem – This point should be obvious after reading the previous item. If you have a problem, capture enough details to jog your memory. You will need to be able to speak to this incident in 4 to 10 days. If you don’t capture details, you will not be able to remember happened. If you cannot remember what happened, it will be difficult to solve the problems to its root cause.
Start each day with a new page – Let us call this one a personal preference. I had a problem on an important project last year. I collected the journals from each team member and set the appropriate page on a big work table. I read all events before and after the problem for each person on the team (seven in all). I eventually isolated the problem and was able to do a small course correction to get things back on track. It was much easier to do this when each day was represented as a single day.
Write with the awareness that somebody else will read your work – This small piece of advice is from the emotional intelligence section of my project management toolkit. If you write with the audience in mind, you are much less likely to assign blame to a problem. A notebook filled with accusations and finger pointing will engender mistrust and undermine teamwork. Keep your entries focused on process failures.
It is funny how things work out. While I was writing this post, we had a problem on an N4000 server we are testing. It failed. The system operator came into my office, and with my team present, attempted to step backward through time describing the major events leading up to the failure on the server. These included hardware changes, configuration changes, data restores, a hard restart, plus two contractors working on the server. All of this occurred in the last 5 days. We were unable to reconstruct the chain of events. Too much had happened in the previous week. It was a miniature disaster.
I reiterated the need for a project change log, but in this case as a crutch for an imperfect memory. With a daily schedule of hundreds of discrete tasks, remembering six items in the proper time and sequence is hard for anybody. When the number approaches 15, it becomes impossible. A simple change log would have prevented this minor mishap, or at least aided in its solution.
I’ve used personal project journals for 15 years on projects big and small. With the advent of collaboration tools like Traction TeamPage, it is possible to move a project journal online. I’ve found this helpful at times, but just as often it’s a frustrating experience for all involved. It largely depends on the comfort level of the team members. Putting your journal online is more time consuming than jotting a note in a notebook. Yet the whole team benefits from access to update-to-date information. I don’t force this issue, but I strongly encourage participation.
There is also the question of utility. How important is livebloging from project team members? It’s not important if there are no problems. It becomes important when something goes wrong. It becomes vital when the team is spread across the globe. In fact, once a project moves out of the bullpen, the need for quick access to project journals increases in direct proportion to number of times zones separating team members. The farther the team is dispersed, the greater the need for immediate access to good information.
The problem that we need to solve is finding an easy way to make small entries that can be gathered and viewed by project; something that replicates the two line entries mentioned above. Twitter comes to mind as a possible solution, so does Yammer. Both come with baggage in the form of outside scrutiny. Traction TeamPage offers something called “Status” updates. These are short entries, like a tweet, that are entered without the normal wiki article structure. It is still an article, so article building techniques can be used. For example, linking to named articles work; as do using tags. It is possible to view a feed by the people who you are following, or by an individual, or by a tag. You can also create a section (or custom search) that gathers status posts using a unique tag (or even a hashtag). This seems like a viable solution for moving a personal journal online. I’ll have to give it a try on my next go live.