The value, challenges and best practices of automotive testing
The value of testing is often underestimated, not understood, or mostly an afterthought, but it can be the difference between success and failure.
We’re all focused on innovations and developing the best and most sophisticated features for our end users. But we must ask ourselves — what would these innovations and great features be like if they didn’t work as expected? Not usable, not fast enough, not available, not safe — what impact would neglecting quality present? One well-known consequence is OEM mass recall — this can significantly impact people’s lives and the company reputation, plus there’s the financial damage, and the internal impact of diminished team spirit, project escalation, team dissatisfaction, productivity reduction and talent loss. In short, testing is necessary to get the desired results of innovation.
Principles of testing
So testing is necessary, but how should we do it? There are seven principles considered the foundation of testing — principles that the organization must understand and live by — principles that each tester must learn, understand and apply from a very early stage in their career.
Testing shows the presence of defects, not their absence
In other words, it’s not possible to deliver defect-free software/products
Exhaustive testing is impossible
Planning and prioritizing test activities is a must and can’t be dismissed
Early testing saves time and money
Testing starts with concept creation — long before even a single line of code is written
Absence-of-errors is a fallacy
Testing activities include both verification and validation. Making sure we’re doing the right thing and in the right way. Delivering working software/products is useless if it doesn’t meet the customer’s needs
Testing is context dependent
There is no test plan or objective that can be applied to all projects; therefore, we analyze and understand each project’s requirements so that we can create a suitable and dedicated test approach/plan
In most cases, repeating the same test cases again and again won’t deliver any added value, therefore we should regularly update and evaluate test cases and their relevance. Additionally, adding new and different test cases will help find more defects
Defects are often not distributed evenly throughout a software product (a small number of modules contain most of the defects). If the same tests are repeated without any improvement/update, eventually the same test cases will no longer find new defects
Where do we start?
Consider asking a few simple questions based on a current or previous project/program:
Why should we invest in testing activities?
How much testing is needed?
What are the objectives?
How do test plan activities relate to those objectives?
How will we measure success?
Simple questions, but not necessarily with simple answers. In fact, these questions don’t have any answers in most organizations, at least not answers which are clearly articulated and uniformly understood by project or program participants and stakeholders.
This absence of clear purpose, destination, objectives and success criteria is a significant problem. Therefore, the test manager must define test objectives for the project or program. These objectives must be aligned with stakeholders, documented in the test plan and considered in the testing definition of ‘done’.
Challenges to watch out for
Late involvement of testing expertise that leads to unplanned costs and potential loss of reputation
Early testing saves time and money, so test experts should be involved in the very early stages of a project. These experts can help in the planning of projects, especially when it comes to defining the testability, test effort estimation and identifying the required resources and skills.
Lack of test planning leads to problems, issues and escalations
Exhaustive testing is not possible, therefore test planning is a very important step to start with.
Underestimation of testing roles and responsibilities
Define clear testing roles and responsibilities within the organization — this will prevent testers from becoming overworked or overloaded. It also contributes to defining a career path for the tester community, giving testers an outlook of career steps and development plans.
Miscommunication due to a different understanding of testing terms and definitions
Terminologies and definitions in the testing domain are an important part of understanding customer needs and expectations (and being on the same page as management and colleagues). The best way to tackle this challenge is to define a test mission for the company and create a glossary — a good starting point is the ISTQB and ISO 29119 for software testing.
Deliver what was sold/promised to the customer
Talking from the test perspective, you should never sell the customer the idea of delivering a defect-free software/product as this is simply not possible. Defining metrics helps in figuring out if you’re heading in the right direction, whether the objectives of testing have been met and to what extent. Using these metrics will continuously help in improving testing efficiency and effectiveness.
Enhance testing technical expertise
As technology evolves, it’s important to keep our technical skills up to date so you’re always ready to embrace change. Supporting the team and giving them a chance to develop their technical ability is a key aspect of nurturing employees.
Being better and different from competitors is very important for the business and its growth. Having a key differentiator when it comes to employee value proposition is also crucial to keep, grow and scale the business.
Cornerstones of achievement
Having the required flexibility and agility in testing strategy and approach will help in better serving the customer’s needs and expectations
Aligning with software testing standards such as the ISO/IEC/IEEE 29119 — Software testing will help to ensure a common understanding, define a unified way of test approaches and improve testing in general
Educating testers by following educational standards such as ISTQB will contribute to having the tester community on the same page with a solid and unified understanding of testing methodologies and principles
Creating a quality mindset within the organization, ensures quality in not just a tester problem/task. Quality is a mindset and every team member contributes to providing the best possible quality to the customer. Implementing a shift-left approach and creating built-in quality is becoming imperative. To achieve all of this, sponsorship and commitment from the top management are essential
The Luxoft system test and validation team have testing experts across technology domains such as IVI testing, sensor-system testing, driver monitoring systems, data services and connected vehicle system testing. Contact us to learn more about how we can advise and support you.
Head of Connected Vehicle Testing
Rabih is a certified ISTQB advanced test manager and test expert with more than 16 years of experience in software testing, focusing on test processes, and strategic and operational test management. He leads the connected vehicle system testing offering; a service that helps clients provide their end-users with a high-quality connected-car experience. This service includes telematics, diagnostics, vehicle network, over-the-air upgrades and cybersecurity testing.