The value, challenges and best practices of automotive testing

The value of testing is often underestimated, not understood, or mostly an afterthought, but it can be the difference between success and failure.

We’re all focused on innovations and developing the best and most sophisticated features for our end users. But we must ask ourselves — what would these innovations and great features be like if they didn’t work as expected? Not usable, not fast enough, not available, not safe — what impact would neglecting quality present? One well-known consequence is OEM mass recall — this can significantly impact people’s lives and the company reputation, plus there’s the financial damage, and the internal impact of diminished team spirit, project escalation, team dissatisfaction, productivity reduction and talent loss. In short, testing is necessary to get the desired results of innovation.

Principles of testing

So testing is necessary, but how should we do it? There are seven principles considered the foundation of testing — principles that the organization must understand and live by — principles that each tester must learn, understand and apply from a very early stage in their career.

Testing shows the presence of defects, not their absence

• In other words, it’s not possible to deliver defect-free software/products 

Exhaustive testing is impossible

• Planning and prioritizing test activities is a must and can’t be dismissed

Early testing saves time and money

• Testing starts with concept creation — long before even a single line of code is written

Absence-of-errors is a fallacy

• Testing activities include both verification and validation. Making sure we’re doing the right thing and in the right way. Delivering working software/products is useless if it doesn’t meet the customer’s needs

Testing is context dependent

• There is no test plan or objective that can be applied to all projects; therefore, we analyze and understand each project’s requirements so that we can create a suitable and dedicated test approach/plan

Pesticide paradox

• In most cases, repeating the same test cases again and again won’t deliver any added value, therefore we should regularly update and evaluate test cases and their relevance. Additionally, adding new and different test cases will help find more defects

Defect clustering

• Defects are often not distributed evenly throughout a software product (a small number of modules contain most of the defects). If the same tests are repeated without any improvement/update, eventually the same test cases will no longer find new defects

Where do we start?

Consider asking a few simple questions based on a current or previous project/program:

• Why should we invest in testing activities?
• How much testing is needed? 
• What are the objectives? 
• How do test plan activities relate to those objectives? 
• How will we measure success?

Simple questions, but not necessarily with simple answers. In fact, these questions don’t have any answers in most organizations, at least not answers which are clearly articulated and uniformly understood by project or program participants and stakeholders.

This absence of clear purpose, destination, objectives and success criteria is a significant problem. Therefore, the test manager must define test objectives for the project or program. These objectives must be aligned with stakeholders, documented in the test plan and considered in the testing definition of ‘done’.

Challenges to watch out for

Late involvement of testing expertise that leads to unplanned costs and potential loss of reputation

Early testing saves time and money, so test experts should be involved in the very early stages of a project. These experts can help in the planning of projects, especially when it comes to defining the testability, test effort estimation and identifying the required resources and skills.

Lack of test planning leads to problems, issues and escalations

Exhaustive testing is not possible, therefore test planning is a very important step to start with.

Underestimation of testing roles and responsibilities

Define clear testing roles and responsibilities within the organization — this will prevent testers from becoming overworked or overloaded. It also contributes to defining a career path for the tester community, giving testers an outlook of career steps and development plans.

Miscommunication due to a different understanding of testing terms and definitions

Terminologies and definitions in the testing domain are an important part of understanding customer needs and expectations (and being on the same page as management and colleagues). The best way to tackle this challenge is to define a test mission for the company and create a glossary — a good starting point is the ISTQB and ISO 29119 for software testing.

Best practices

Deliver what was sold/promised to the customer

Talking from the test perspective, you should never sell the customer the idea of delivering a defect-free software/product as this is simply not possible. Defining metrics helps in figuring out if you’re heading in the right direction, whether the objectives of testing have been met and to what extent. Using these metrics will continuously help in improving testing efficiency and effectiveness.

Enhance testing technical expertise

As technology evolves, it’s important to keep our technical skills up to date so you’re always ready to embrace change. Supporting the team and giving them a chance to develop their technical ability is a key aspect of nurturing employees.

Divergence

Being better and different from competitors is very important for the business and its growth. Having a key differentiator when it comes to employee value proposition is also crucial to keep, grow and scale the business.

Cornerstones of achievement

• Having the required flexibility and agility in testing strategy and approach will help in better serving the customer’s needs and expectations 
• Aligning with software testing standards such as the ISO/IEC/IEEE 29119 — Software testing will help to ensure a common understanding, define a unified way of test approaches and improve testing in general 
• Educating testers by following educational standards such as ISTQB will contribute to having the tester community on the same page with a solid and unified understanding of testing methodologies and principles 
• Creating a quality mindset within the organization, ensures quality in not just a tester problem/task. Quality is a mindset and every team member contributes to providing the best possible quality to the customer. Implementing a shift-left approach and creating built-in quality is becoming imperative. To achieve all of this, sponsorship and commitment from the top management are essential 

The Luxoft system test and validation team have testing experts across technology domains such as IVI testing, sensor-system testing, driver monitoring systems, data services and connected vehicle system testing. Contact us to learn more about how we can advise and support you.