Boss Asked a Monitor System, I Did an APP

This entry is part 3 of 4 in the series TMQ - Less Words, More Facts

original link: http://tmq.qq.com/2017/07/app-2/

Recently I engaged in a new project that requires a backend API monitoring system. It is an Android project enlisting a chatbot and HTTP is considered not applicable as the communication protocol.

IMs such as Mobile QQ, Wechat use their own private binary protocol without published specification. So it is hard to simulate the network communication without direct support from their dev team.

Likewise, the new project adopts a proprietary protocol called “Harley”. Technically, I need to simulate the “Harley” protocol for the purpose of backend API monitoring.

Harley Protocol

Catering to the mobile environment, Harley provides fully optimized network API layer, real-time push notification, common resource downloading as well as incremental self-update to all sorts of mobile Apps.

Harley SDK is optimized for interactive traffic (high frequency and small volume). The business backend supports HTTPS, TAF or jce based protocols.

As Harley is mobile oriented, the official SDK supports only Android and iOS.

Traditional Method

Traditionally, a monitor system runs on PC, simulates the network request and verifies the response.

Applicability

  1. PC based system is convenient for standard protocols (e.g., HTTP, SMTP etc.) as there are various open source components
  2. On the other hand, it is hard to develop a system based on proprietary protocol. In our case, I have only Android and iOS SDK for Harley.

Pros

  1. PC based monitor system is stable
  2. PC is performant
  3. PC based system is extensible

Cons

  1. It requires a substantial development effort (translator: that is, re-program the whole SDK)
  2. The runtime is different from the real world environment, on which APP runs
  3. The code base of monitor system is segregated from that of the real product, and it is costly to synchronize them timely

Brood Something New…

In fact, the 1 and 3 issues are the major ones for the traditional PC based plan:

  1. develop effort: the development of a Harley simulation utility is so big that developers are reluctant to take the task
  2. maintenance effort: once backend change/upgrade the protocol fields, I need to adjust the simulation utility as well

So I decided to brain storm with myself:

  1. Why do we need a backend monitor in the first principle? Basically, backend API monitor is a periodically triggered automatic test to check the functionality of backend in real time. So we can notice the problem within 10 minutes.
  2. Why backend API monitor can not be substituted with UI automation test?: UI automation test takes too long. And the stability and coverage are too limited to real-time monitoring.
  3. Is mobile-end can only be used for UI automation? The performance and extensibility of mobile-end (Android and iOS) are equivalent to PCs.
  4. Backend API monitor enlists pure network interaction and does not involve any UI, is mobile-end still applicable?: I can’t see any reason why it is not as an APP of pure network interaction is enough.
  5. How about development effort? We can use the SDK directly. If the product uses MVP, we can reuse the code of Model and Presenter layer directly.
  6. How can the monitoring charts be displayed on a small phone? There are numerous ways to transfer data from a mobile phone to a PC. We can transfer the result to PC for display purpose.
  7. We need to execute periodical tasks in the long run, how about stability? No one has done this before so we do not know. If a real device is not stable, theoretically we can use a simulator.
  8. Is problems reported from mobile-end can be easily pinpointed? Of course. We use the product code directly so if there is any problem, most likely it is the problem of the source code itself!
  9. If the system is implemented in mobile-end, is it hard to maintain and upgrade? Whenever the product upgrade,  we copy the code directly:)

After the brain storm, I decided to give it a go.

Finalized Plan

By largely copying the real code from the target product, I accomplished an architecture for the monitoring system as follows:

Mobile-end layer

As recommended by Google, MVP is used in most Android development.

Model: Data layer. Different from MVC, in MVP, Model is resonsible for database r/w., network request/response etc. So I copy all the source code of this layer.

View: For UI presentation. Do not need.

Presenter: This is the logic layer bridging View and Model layers. In MVP, Model does not communicate directly to View. Instead, Presenter fetches the data from Model, processes the data and deliver the result to View for presentation. This way View and Model can be decoupled. I copy this layer selectively.

Theoretically, the copied source code = all Model + portion of Presenter.

“Theoretically” is far from enough. I need to categorize the specific source code:

Finally, copied source code = NetService (all)+ Manager (portion)+ Engine (portion)

App Implementation

1.Jce module:  Jce is a Tencent component used for defining the internal communication protocol, i.e., fields used for RPC. Please refer to the relevant open source projects of Tencent (translator: the Tencent version of Protocol Buffer)

For example, the request message is defined as follows:

2.The copied source code includes: Harley SDK init, packet assembly, network request, packet check.

Initialization: (from NetService module)

Network request: (from NetService & Engine)

Response check: (from Manager)

It involves sensitive information so is omitted here.

3.adb communication: we expose Activity (DO NOT expose Activity in ordinary APP for security reason)

Receive boot instruction from adb; start thread; send request

The relevant adb command:

4.log the result:

PC Layer Implementation

Easy as 10s lines of python.

1.Trigger task:

2.Capture log using adb:

Cautious:

Here, I use multi-processes instead of multi-thread for the following reasons: log capture is a blocking operation, if not terminated, it will keep on running. However, there is no thread abort instruction in python.

The return values, Poplog and f, represent the opened processes and files. They are used to terminate the process the close file respectively.

3.Log analysis:

1) check if the test is finished: “test_is_end”

2) check the result: it mainly analyzes the log, and check the critical fields. It involves sensitive information so is omitted here.

4. Report & alert

Generate report based on the data from step 3. Alert if any abnormal is noticed.

5. Scheduling

It is worth noting that the script can be used directly by jekins and other CI platforms.

 Result

  1. Efficiency: Each case (for one request), which involves network request + log printing + log capture + log analysis, finishes in seconds, which is sufficient for the purpose of backend API monitor.
  2. Stability: Initially a physical phone is used. After some incidents of unstable USB, problematic battery, a simulator is used instead.

Based on the test, the X86 Android 7.0 simulator is the most stable and performant version (host: i5 + 8GB + VT). The simulator is configured as follows:

To conclude, task accomplished!

Postscript

In this case, I summarized the methodology when encountering multiple options with different risks, pros and cons:

Critical points:

  1. The first thing we need to understand the END GOAL. For the API testing, the end is quality. Automation, backend API test is pure means, not ends.
  2. We need to investigate feasible as well as infeasible options, as an intuitively infeasible option might turn out to be the optimal option.
  3. We need to enumerate concrete reasons why an option is not feasible, we need to check repeatedly whether those reasons are true.
  4. With the lens of the END GOAL, we noticed that sometimes we can not find the optimized plan, simply because some else have done it before in a less efficient way; Sometimes we think a solution is not possible simply because no one has done it before. This is how a case becomes a stereotype. So we need to keep in mind that as technology advances, things are changing.
  5. Think when confronting a problem. Invite others in the brainstorm if necessary.
  6. Record the thinking for retrospection.

Post-postscript

In this monitor system implementation, the benefits and the prerequisites of the practice in used can be summarized as follows:

benefits

  1. Less code as most of the source code is copied directly
  2. With the deepened understanding of the source code, I can tell the bugs in source code directly when talking to developers.
  3. The client logic (in Model and Presenter) is verified while we are monitoring backend API. One stone, one Goliath, and one bird.

prerequisites

  1. The client source code complies to MVP.
  2. Testers can read, compile and debug the source code.

 

 

 

 

Things a Devop Needs to Know – Part I

This entry is part 1 of 4 in the series TMQ - Less Words, More Facts

This article has been republished to:

https://hackernoon.com/back-end-performance-those-metrics-we-care-about-ade678e87969