A profile of a performance engineer

A profile of a performance engineer

My name is Andrii, and for four years now, I have worked in performance testing and optimization. I started my career as a Java developer, but I soon moved to the dark side of non-functional testing. I dealt with various products, mainly e-commerce; worked with different stacks (from .NET and Java to Node.js and Python) and tools (from JMeter and Gatling to HP Load Runner). Now I am focused on performance testing of the back-end and optimizing the client part of the product at the AB Soft Company (Odesa).

There is a stereotype that performance testing is regular testing with more users involved; thus, the approaches do not differ. That's not quite right. In this article, I will ruin this performance testing stereotype and highlight what this specialization implies.

Disclaimer. The article contains IT jargon words. I did my best to minimize their count, but to preserve the consistency of how the readers perceive the author's ideas, I still seek the balance between a regular text and the one to be read by someone working in IT and using these words daily. The article makes no pretense of being unbiased. It presents my own view of the industry and thoughts about a performance engineer's job specifics.

Briefly on performance testing

As long as this article focuses on the job of a performance engineer, we can't avoid the conversation about performance as a testing type.

What is it?

Performance testing is a type of non-functional testing intended for checking the operation of the system and its components under a particular workload to determine the following characteristics:

●     speed of request processing;

●     rate of using dedicated resources;

●     stability;

●     ability to adjust to the change of workload level etc.

Quite a few types of performance tests are meant for determining one or more characteristics from the list above. Each has its peculiarities of planning, creating, and operating. If this topic is interesting for the audience, I can provide more details in the other article.

Why is performance testing necessary?

In the Web 2.0 era, the accelerated development of technologies and the market of portable devices make it possible for anyone to use content and receive services anywhere and anytime. Every year, the industry giants set new quality standards to provide end-users with what they want as soon as possible: instant registration and payment, one-click purchases and booking, 4K video streaming.

Performance characteristics influence the experience of using the system by an end-user. There is a reason why they are singled out as components of the product quality model (See ISO/IEC 25010:2011). The speed of providing a service can be a decisive factor for choosing between two service providers.

Performance and speed of system operation have become crucial for any business. Happy users bring the money, so seconds matter in the fight for them. It is where the story of the introduction of performance testing begins.

However, it takes more than just a wish to improve performance. It all begins with forming correct requirements and goals. Ask a business person, "Why do you need a performance testing?" and the most common answers will be sort of

● "To make it quick";

● "We want to know how many users our production can handle";

● "You are a performance engineer, so you are supposed to tell us what to do."

None of these options in such a format is worth spending time on as they are detached from reality and won't improve the product. However, they establish the areas where we can and should work to gain actual results to a certain extent.

Stereotypes

People in business and management with no technical background have little to no idea about the nature of performance and its testing. Unfortunately, the situation is not much better among the tech specialists from various project teams. There is a stereotype that performance testing is regular testing with more users involved; thus, the approaches to its fulfillment do not differ. In fact, the approaches are different, so are the tools and the entire process.

Image: Dmytro Yatsenko

Stereotype #1. Performance testing is similar to a DDoS attack.

Everything it takes is to launch as many floods as possible so that they chaotically load the servers of your system. Okay, we undertook such a test, and our system kicked up the bucket after 10 seconds. How beneficial are such results for you as a non-functional testing engineer, for the team, and business as a whole?

Every product has an end-user. A business's primary objective is to make money on this user and make them happy. Tests should emulate an actual workload on the system. "Actual" means a specific influence of each customer on a particular nod of your app. Irrelevant workload during the test will present irrelevant results. They, in their turn, can become the basis for a performance report, which has nothing to do with the system's present condition and makes no use, and something even leads to incorrect decisions.

The principle is "Do it right or don't do it at all", or you will waste your client's time and money.

Stereotype #2. Operating functional is the most important thing. It doesn’t matter how quickly a customer receives the result.

For many business domains, the speed of service provision is an integral part of the service itself. The most famous example is e-commerce, such as an internet shop, online gambling, trading, and other highly loaded systems.

Here we are back to the concept of the previous stereotype. User behavior modeling enables us to forecast the experience of using our product. It concerns business the most. These are not milliseconds that give profit but users and meeting of their needs that monetize.

The specific closed systems sometimes ignore the speed of operation. These systems feature a small number of active users, or they can be applied within one company and used by the employees only. The company staff is okay with waiting for its internal CRM downloading and processing a transaction as it is part of their job, so they are ultimately tolerant of waiting time.

The commandment "If it ain't broke, don't fix it." is more applicable to the functional space of the product. Speed always means there is room for optimization if it ensures significant improvement.

Stereotype #3. Performance testing takes several hours before deployment

In fact, performance testing is a long-lasting and resource-consuming process that is sometimes difficult to squeeze into the time frames established by various "flexible" development methods.

One cannot always determine how much time an entire cycle of testing is going to take. I mean all the phases: from collecting the requirements and writing tests to the result analysis and drafting a report.

The results of performance tests are subject to verification and confirmation. They are based on the data, the quality and volume of which are sometimes statistically insufficient to make any conclusions concerning the conformity with the specifications. If the test exposes a bottleneck, it may be more time-consuming than initially estimated.

Under perfect conditions, performance testing should be carried out on the stable product version after testing the code for functional defects. Otherwise, an overlooked defect can ruin the performance test results. If there is no chance to re-run these tests, it is fair to say that all the time spent on this round of testing is wasted. This is why performance processes should be smoothly integrated into the development process and product testing.

Stereotype #4. Most issues with speed can be resolved through scaling

Why spending time on code optimization, cash configuration, connection pools, etc.? It's much easier to double the number of servers and boost RAM capacity. However, the most apparent solution is not always the right one. The more extensive infrastructure we have, the more the customer pays for it. And this is not something they are willingly spending money on.

Besides, scaling does not always solve the problem. It is no secret that large enterprise projects, which have been present in the market for more than one decade, still operate entirely or partially based on old technologies and architectural solutions that are now outdated. These architectural constraints can bring to naught all the efforts to improve the system through scaling.

The main idea is to achieve the best condition of the system speed with optimal employment of resources. Therefore, the most visionary and efficient solution is to allocate resources for full-fledged performance testing for product adjustment and optimization.

Stereotype #5. Performance testing does not require any special skills, so practically any team member can perform it.

Not so rarely, at QA/DEV conferences, when I tell people what I do, I hear, “Oh, I also carried out that performance testing in my project. My manager/product owner wants to carry it out, but we were short of resources. I installed Apache Jmeter and flooded our QA from my laptop".

It is a typical scenario when someone other than a performance team tries to find a solution to the tasks set by the business. I offered some examples above.

The range of tasks for a performance engineer is vast, so they should master the skills of many other professions.

Who is a performance test engineer?

I have outlined an abstract profile of a person who can be identified as a performance test engineer and essential in this field. In no way do I make a pretense of being unbiased on this matter - this is just a summary of my personal observations and experience. You can agree with me or have a different opinion which I'll be happy to read in the comments.

Terms, titles, and interviews

Various sources of theoretical information do not always correspond with one another, as each was created autonomously. It results in different terms that name the same type of tests (for instance, longevity, endurance, CHO-tests). In performance testing, it is a separate dimension for holy wars. Along with the differences in terminology, there is a mixed bag of approaches to creating workload models, its generation, and analysis of the obtained data. Such a mixture of someone's own and someone else's experience can result in incorrect processes. The worst thing about it is that leadership can be oblivious about it, and so is a performance engineer, though they have built these processes.

A similar situation emerges with defining who a performance engineer is. It comes from a great variety of activities and responsibilities this position integrates. These are the linear and top management that often lack performance literacy. Many have a general idea about the developers' work: they code, or QA specialists: they look for defects in the code. It is more complex in performance, so a performance engineer is also responsible for increasing awareness about specific features of his work and performance as a whole.

To my mind, the most accurate term that communicates the scope of this profession is "Performance Engineer." "Engineer" is the best description of a specialist as an autonomous work unit in any industry. A tester writes and tests; an analyst collects data. In fact, these are only a few of the tasks an engineer should fulfill in the entire performance testing cycle.

Amid the promotion of performance testing and increasing demand for such expertise, the market sees many vacancies, featuring various titles and content: Performance Tester, Performance Analyst, Performance QA Engineer, Load Engineer, etc. As a rule, the vacancy description reveals whether the company has any idea about performance testing. Here is a very generalized check-list of red flags in a performance vacancy:

● The general information about performance is copied from the English-language sources on the issue;

● The list of all possible tools for generation of workload and monitoring;

● Lack of project details.

If the vacancy contains only these aspects, the company is likely to have a very vague idea of performance. It means they are going to hire you as the first potential expert in the area. It poses a question: who is supposed to interview you to verify your competence? You will talk to specialists from different OPS/DEV/QA teams on processes and specifics of solving a particular performance-related problem in the savvy company. At worst, it is going to be a conversation with a member of the testing automation team on bash, git, and database queries, fundamentals of ${your favorite programming language} and other essential things. It isn't easy to evaluate such an interview objectively, and either party will be unlikely to make any use of it. It may eventually turn out that the introduction of performance processes within a specific project is unnecessary.

Integration into the project

Performance testing is only necessary when the system is expected to receive a sufficient workload level during its operating cycle. In other cases, performance processes will become just a waste of time and money. So, a performance engineer usually works with major enterprise projects with distributed architecture, two or more products, etc. Seemingly, a large project means a large staff of performance engineers. Unfortunately, it's not always the case. The higher engineers' qualification is, the fewer specialists the project requires. So, a regular number of engineers per project is one or two people (it is not an absolute number and may vary from one case to another). There are different ways to integrate a performance engineer into the project.

On-call engineer

An engineer is not tied to a specific project. A dedicated unit of specialists works within the company (a.k.a Center of Excellence). Like a SWAT team, it is deployed for one project or another to create performance processes and provide relevant expertise. It is a common practice for outsourcing companies with many projects, where it is cost-inefficient to assign an experienced specialist for one project. In Ukraine, there is a company that implemented such an approach.

Remote engineer

An engineer or a group is integrated into a project. The specialist dives into the development project and has faster and more efficient communication with the team members for testing.

Full integration into the organization

It is the best but almost impossible option. An engineer is integrated into the development team but is still in charge of incorporating performance processes at the organizational level. For it to happen, business and management should be interested in boosting the performance from the early stages of development and at the design phase. It also includes outlining functional and performance specifications for the development of the product performance. It asks for the promotion of performance culture at the level of the organization. Leadership, developers, QA specialists, and other staff should be interested in such an approach and consider performance when solving their routine tasks.

What we see most commonly is the second option of integration of performance engineers into the project. During the system research, writing, testing, and detecting performance defects, an engineer works in close collaboration with the members of different teams. Shortage of necessary information is a standard issue for a performance engineer. In the distributed teams, people work remotely and live in different time zones. Meanwhile, Wiki has no information at all, or it is outdated. The better the communication channels are adjusted, the more time the engineer can spend on performance testing and analysis and do it faster.

Profile

This article has often argued that a performance engineer holds many skills that specialists from different areas have. Further, I will describe what I believe to be the critical competencies of such a specialist.

Business analysis

Most literature on performance suggests starting with business analysts who should provide ready users' profiles and detailed usage statistics for the product. However, the necessary information is often non-existent, or it is in a non-usable condition. You have to collect this information in bits and pieces and complement it with your gathered empirical data. To form the requirement for your product, the available data for other products in the specific business domain can be instrumental.

The more output data about the product and its application details you can collect, the more accurate the workload model will be, thus the testing results.

UX-analysis

As I have mentioned above, we speak about non-functional testing here. It is inextricably linked with the end-user experience (UX). Of course, proper product functioning is a top priority, but other characteristics are also important for selling it to a user: reliability, data security, usable interface, speed, etc.

It is worth mentioning that performance in technical terms and from the user's point of view are two different things. When we speak about the server part, we refer to latency, response times, and throughput. While for the user who interacts with the service through the client, these concepts mean nothing. What matters here is not how "speedy" our product is but how speedy our user perceives it.

So the ability to perceive the system from the user's perspective is necessary to prioritize properly when shaping a performance goal and optimizing not for optimization but for improving the product's user experience.

System analysis

Performance testing has nothing to do with black-box testing. If the information about the system environment and structure is not available for some reason, testing results will yield little information.  The performance testing strategy should include details about system structure and logical, physical, network, and program architecture of the production environments and dedicated environment for performance testing. Otherwise, it is impossible to extrapolate the obtained results even approximately and make any conclusions about system conformity with its specifications.

Not so rare, a performance engineer is one of very few project members who have a comprehensive idea about the product as a holistic body. Without such a forecast, it is impossible to spot any bottlenecks. It distinguishes a performance engineer from other team players. They usually start a performance career, coming from another specialty, so they may already have a shifted focus toward a particular perspective. For instance, the engineers who used to carry out automation of client part testing now look at the system from the client viewpoint; the developers analyze the system from the stand of the core and code exploited on the server. It is not about a specific specialist bias but about the system perception itself as it is shaped under the influence of this or the other specialization. Only by finding a certain balance can one trace a problem. So, in my opinion, the ability to see an entire system and its architecture from different perspectives is one of the crucial skills of a performance engineer.

Data analysis

During the workload, systems emulate a significant number of their users. These are always synthetic data, yet they should be as close to the real ones as possible. It concerns each specific user and their behavior within the system, as a real user, just like a random number generator, is absolutely unpredictable. Even if the model is ultimately close to reality, it is unable to forecast all possible results.

Given specifics of the system and tests themselves, the test logs volume is measured in gigabytes or even terabytes. After each testing round, it is necessary to verify these data for their statistical validity, calculate and typify errors, if any, rank, analyze, and - what counts most - present them clearly and understandably for all the parties involved.

It is essential to highlight that among all the activities of the performance engineer during each testing cycle, analyses of the data obtained takes the largest share of time. After each testing round, the reports drafted for the team's management and technical part will differ significantly. The former will reflect the general system status, focusing on consistency with the set goals; the latter will provide the complete analysis of the exposed issues and all possible data for their isolation and elimination.

Communication

You can say, “All team members communicate with one another during working processes - no big deal”. But here, I use the concept of communication in a broader sense. I don’t mean daily-, demo-, and other SCRUM meetings. A performance team of the project can comprise one engineer who needs to:

●    collect all necessary data and queries from business and top management;

●    Collect information about the system structure;

●    Agree on the dedicated environment (relevant to production environment) for testing;

●    Coordinate all the teams involved in the testing process;

●     Collaborate with DEV/QA/OPS team when analyzing exposed errors;

●     Present the data obtained and suggested hypothesis to the product owner;

●     Carry out training sessions for all team members interested in improving their performance literacy.

It is irrelevant how efficient performance testing is if the engineer cannot present its results properly to the customer. Without it, all the time and money are wasted, and the process turns into testing for testing.

Programming

Creating accurate performance tests is far from the simple recording of a user's actions through a recorder which is now a part of almost every load tool. Writing the test that will generate relevant workload on the system requires writing a code with an added logic and reading and understanding the code created by developers.

Each project has its peculiarities, while load tools cover the most common scenarios. There is often a need for additional tools to automate the process (parsing specific logs, generating large volumes of testing data, preliminary environment settings, collecting custom metrics from remote instances) or to expand tool functional (implementation of a specific protocol).

A performance engineer does not have to have a command of programming languages at the Senior developer level. Still, at least, he should read and write code to create his own framework for automating performance testing processes.

There are many more roles a performance engineer can play. But it seems those mentioned above are sufficient to shape a general idea about this profession.

Instead of conclusion

As you could see, performance testing is a much more comprehensive QA area than it may seem at first glance. The profession of a performance engineer is complex but exciting and multi-faceted. This profession forms a comprehensive view of the product, allowing the development of many competencies without limiting the choice of tools to complete the set goals. It is the reason why I have chosen this path for myself.