PEST Boston II - 18 July 2007

PEST BOSTON II - Brilliant :)

So here we are in Boston, and its PEST II at the Naked Fish restaurant.

Everyone who came last time was here, and we had some new friends too in the form of colleagues from HP Bangalore and Monstor.com.

I was very excited about coming to this PEST. Partly because I was looking forward to seeing everyone again - last time was a lot of fun, and partly because I wanted to tell everyone about our new scholarship scheme which is just starting up at Boston University. Our first outside of the UK, which will start in September this year, and have interns with us in summer '08. Its a very personal passion of mine to invest in the future of our industry, and its wonderful to see the scholarship schemes starting all over the world.

Although we still havent cracked the venue side of things - it was a bit noisy even in this seperated out area - everyone contributed well and we had some great discussions right from the start.

With such a mix of hardware/software experience along with the international flavours from the background of those attending US (7 people), India (2), UK (1), Iranian (1) we had some amazingly different view points as well as a lot of synergy.

The session started with an experience report from our own Don St Pierre, who was taking part in PEST as his last official act with TCL USA. He's off to join EMC from next week, and it was great timing that he could do this before he left.

Dons experience report focussed mainly around the experiences he documented in his white paper last October - about the need for seperation between QA and development reporting lines, and we were lucky to have Steve with us who had worked with Don on the same project and could add his ideas and recollections too.

We drew on these experiences to look at how we report from testing, and particularly how a set of metrics such as DRE can be used to create an 'apples with apples' comparison across projects. Across the group there was also a lot of experience of how metrics and measurements can hurt you as much as help - with the 'what gets measured gets done' approach that we see a lot with teams.

We also recognised that you can generate 'behaviours that exactly meet the metrics, rather than make sense' - things like if you measure a team on numbers of bugs raised then you get every little typo raised as a seperate entry and lots of time wasted.

Now as you can see I remembered my camera so we have a couple of pictures which is great!
(Watch out for this lady at the front of the table - shes a trouble maker ;) )

Don did some great prep work for this session and gave out copies of his white paper and some definitions of DRE. We also shared some best practices from TCL in terms of using the DRE metrics at gateways and to help create measurement of process maturity.

Some other things that we learnt between us across the session included:

• There is a job website called Dice which costs only $60 per posting, compared to Monster.com which is circa $360
• QA can often be mistakenly seen as the root cause of defects
• Root cause analysis on defects is very important in a continuous improvement environment. How can we set up our defect tracking to then generate suitable reports to do this? Perhaps a topic for next time.
• Common performance issues are missed when environments do not represent live in a quantifiable / scalable way
• Upper management can often try to micro manage QA - and when they do they look at things they understand rather than (perhaps) the things that are actually important
• Date and budget driven objectives that dont include quality - and particularly post live quality metrics generate very unhelpful behaviours on projects
• Bringing customer found bugs into the post implementation review and post live defect tracking is very important in a continuous learning environment
• Release specific scorecards are very useful - particularly when they track DRE
• Public vs Private metrics - what can you track and what can/should you publish. A difficult balance
• Reporting to senior managers, VPs and Presidents means finding things that both interest them and help us - you cant just talk quality, you need to talk value.

A lot of stuff came out from these discussions, and I'll be very pleased when our PEST website is up and running so that we can all share and discuss things online - and keep much better records. Hopefully that will be in September, just in time for the next PEST in Boston - week of 3rd September.

Thats about all for now, except to say:

• Im hoping to have a private room, with a round table for the next PEST Boston - probably at the naked fish restaurant again
• Steve Govednik (monster.com) and Dan Downing(Mentora) have voluntered for giving experience reports/challenges to the group next time
• Nadereh (mathworks) and Nancy (HP) have agreed to do stuff for the session after/ be alternates for the session in September
• Im going to look to bring some of the scholarship essays along for the group to have a look at - some of them have been really brilliant
• Don will be coming along to the next PEST, and has promised Matt hes going to bring a few people from EMC too!

Big thanks to everyone who came along this time, and who made it such a great event. Look forward to seeing you next time.
Stew

The point of PEST: None of us is as smart as all of us!