Passionate Test – The Expedition of Legion of Smoke

This entry is part 4 of 4 in the series TMQ - Less Words, More Facts

original link: http://tmq.qq.com/2016/07/the-passion-test-smoke-regiment-expedition/

I. The Origin of Smoke

The test team has adopted smoke test for a long time. After we restructured in FTs(feature teams), smoke test is managed by each FT.
In the background of fast iteration of versions, it is hard to parallelize test of multiple versions and to release in time using the orthodox process of “ready for test->in test->regression test”.
Thus we decided to try smoke test to alleviate the testing pressure as well as enhance the testing quality.
One day I said: “I will treat the guy a meal if he finds more than 10 bugs in a smoke”, then the smoke started.

(N.g., a feature team is grouped with roles responsible for the same feature module, which includes developers, product managers, testers, ops, designers etc.)

II. Everyday Scenes

1. There are 23 more “testers” joined our team!

Every day on the 10th floor of Nantong building, we can see a bunch of developers, ops, and product manager turned into testers, tapped on their phones with smile, and recorded bugs one by one.

“Another crash, it’s developed by A”

A: “gimme the phone, I’ll collect the crash log”, then A grabbed the phone and rushed to his computer……

Before the smoke test finished, A came back: “solved, it is because of of#$%^&&*”. People looked up to him while thinking “I’ll give you another one”……

2. The minimal adaptation base

Dev A: “A UI component misplaces in my phone, and the text is overlapped!”

Tester B: “This plugin can not be turned on in my phone”

Product manager C: “Why everything is normal in mine”

Smoke organizer: “Record those adaptation issues.” The rich product manager C pulled out several phones from his pocket, “try them”……

3. Bugs explosion even before the product is ready for test

After half an hour, there was a full page of bugs recorded in the laptop on the pool table. Tester clicked on the “import” button, the corresponding developers received the bug notification.

Developer A: “I will code this portion of logic, so I can fix all the bugs along the way. Good to know them beforehand!”

Developer B: “I found several bugs of my own, will fix them before anyone noticed.”

Tester C: “The developers are going to fix the bugs before the test, I will worry so much less.”

Developer D: “So many bugs……”

4. Everyone is product manager

After a lunch, the product manager felt pleased when he saw developer D was busy, as he always does until D looked up to him:”the bugs are all reassigned to you”

“What…?” The product manager rushed to his computer, opened the 10 unread tickets. He found all of them are UX issues.

5. A sad history

Pitfall 1, (smoke) test release

Smoke organizer: “We are going to have a smoke test today, could you release a test version please?”

Developer A: “Sure thing. I’ll do it now”

After 15 minutes, “How is it going?”

Developer A”The release platform has problems, I have to solve it first”

After 15 minutes, “How is it going?”

“XXX just submitted some code, I will release it again”

……

Pitfall 2, I am not a born leader 🙁

Smoke organizer: “Let’s install the App and start smoke”

Developer A: “OK, after I fix this issue”

Test B: “OK, after I finish this test case”

Product C: “Sure, could you install it for me please?”

After 10 minutes, everyone is doing their own things on their own seat……

Pitfall 3, I will take the Karma I started

Initially, we used an A4 to record bugs. And we testers have to record tens of bugs manually like a file clerk. Moreover, we have to do the regression test for all the bugs generated by us in the normal test as well as smoke test.

6. Countermeasures

Countermeasure 1

We rotate the developer responsible for the test release. And we start the release process 30 minutes before the smoke.

Countermeasure 2

We rotate the smoke organizer, so everyone can understand the difficulty of the organizer and not fixate only on his own matter.

Countermeasure 3

We use a template to record bug on a laptop, which can be imported to TAPD (translator: the Tencent internal version of JIRA) with one click. Moreover, the bug reporter is responsible for doing the regression test himself.

Then we testers are really happy.

After a few iterations, smoke test is now efficient and pleasant. I will list the tips and reflection during this process.

III. Benefits

a) UX flaws: Dogfooding and show case is a bit too late. An early stage smoke test can expose lots of UX issues.

b) Adaptation issues: With more people involved, more kinds device can be covered.

c) Bug shooting: A lot of bugs can be found at pre-test stage.

d) More perspectives: With various roles involved, we can get suggestions of issue fixing as well as optimization from different dimensions.

IV. Our Approach

Like “accurate test”, smoke is not the silver bullet and has inevitable drawbacks. For instances: the validity of the bug discovered,  repeated bugs, the time cost of a developer to deal with a large amount of bugs, etc.

As aforementioned, we faced some problem in practice indeed. However, honed with several iterations, every FT has recognized the value of smoke test, and agreed that the advantages of smoke test outweigh its disadvantages. The whole team is becoming more and more active in smoke test, and developers sometimes propose a round of smoke themselves.

I list some of the statistic data and the improvement process of smoke test in Tencent Mobile Manager 6.2 for your reference:

1. Bug valid rate

Valid rate: 64%

Advice: Better wrong than missed

Explanation: Despite the relatively low valid rate, we still encourage people to report new bugs. Developers and product managers can evaluate them.

2. Repeated bugs:

Explanation: The figure below gives the rejected bugs in smoke test, repeat rate is relatively high.

Advice: Generally, we consider the more times a bug repeats, the more critical it is since more people noticed it.

3. Optimization approach

It is critical that all roles put their thoughts in the practice of optimization.

Smoke guide:

1. Testers list the testing routes for important features.

2. Developers mark the routes that deserve more notice in accordance with the newly submitted code.

3. Product managers highlight the features that reflect the spirit of the product.

Bug record:

1. mark the issue type (bug, UX, logic, requirement, etc.)

2. format the content for the import

3. record bug right after the test.

The voice of the crowd:

“One thing. I noticed that recent issues are all bug. Just wanner remind that we can focus more on UX issues. And please use the App from the point of view of a common user”

“Another one. We have recorded a large volume of issues. Do you guys think it is a good idea to categorize them with marking their type (bug, UX, logic)?”

“Better ask developers to highlight the features that deserve more notice”

“Yes. Every time when you commit code to SVN, we need to think the test route in the tester’s point of view”

V. Smoke In Practice

Test preparation:

1. device: a laptop for bug recording

2. time: 5:30 – 6:00 afternoon

Developers:

1. rotate: each time we appoint a developer to release the test version

2. archive: archive should be ready in RTX (translator: the internal slack or hipchat of Tencent) before 5:30

3. collect: collect the testing routes (provided by testers and developers as aforementioned)

4. rendezvous: at 5:30

5. guide: developers highlight the important testing routes

Testers:

1. collect: collect all the bugs and batch import to TAPD

2. regression: print out the bugs (from previous rounds) that require regression test, and remind the corresponding person to conduct the regression

3. prioritize: with the assistance of developers, prioritize the bug reported

4. solved bugs: keep note of the solved bugs in regression

5. guide: testers highlight the important testing routes from the testers’ point of view

Products:

1. showcase: during the smoke test, product managers will answer the questions regarding UX. Reasonable feedbacks are recorded directly to TAPD.

How to record:

1. Title format: [version+module+smoke] [BUG/UX/logic/requirements] ***-reporter

bug [6.4 throughput App Smoke] [UX] The spinning-loading-bar requires optimization. It is not distinct and is ugly.

2. Import to TAPD

VI. Practical Tips

Today I’m the lead: rotate the developers to be the lead to enhance their proactivity and understanding

I’ll finish it today: record all the bug on the same day, so developer can react faster

Transfer to product manager in 1 second: UX issues are transferred to product manager instantly. The product manager then evaluate those issues and change the requirement if necessary

Better wrong than missed: regardless how the issue seems to be, big or trivial, record it

Faster regression: fast regression without leftover

Big and full: collect the phone model of everyone and provision devices that are missing

Foodie: prepare some snacks as reward

VII. Conclusion

In the background of agile development and fast iteration, smoke test can provide means for other roles to focus on the quality. It is an efficient supplementary mean to normal test as well. The optimization process requires all people’s involvement to produce good result in practice.