2 Contents Introduction 04 06 Executive Summary 14 Current Trends in Quality Assurance & Testing 16 Key trends in IT Artificial intelligence 22 Test automation 26 31 The quality assurance organization Test data and environments management 36 40 Efficiency and cost containment in quality assurance Sector Analysis 44 Automotive 46 Consumer Products, Retail, and Distribution 49 Energy and Utilities 51 53 Financial Services 56 Healthcare and Life Sciences High-Tech 58 Government and Public Sector 60 Telecom, Media, and Entertainment 63 About the Study 66 About the Sponsors 70 Previous Editions World Quality Report 2009 2010–11 2011–12 First Edition Third Edition Second Edition 2013–14 2012–13 2014–15 Fifth Edition Sixth Edition Forth Edition WORLD QUALITY 2 0 16 -17 REPORT EIGHTH EDITION 2016–17 2015–16 2017–18 NInth Edition Eighth Edition Seventh Edition Quality World Report 2017–18 Ninth Edition
3 Regional and country reports are available online from www.worldqualityreport.com or from Capgemini, Sogeti, and Micro Focus local offices covering: North America, United Kingdom, France, Germany, Benelux, Southern Europe (covering Italy, Spain, and Portugal) Eastern Europe (covering Poland, Hungary, and the Czech Republic), China, Australia and New Zealand, the Middle East, and the Nordic Region. World Quality Report 2018–19 Tenth Edition
4 World Quality Report 2018 –19 4 Introduction elcome to the tenth edition of the World Quality Report (WQR) by Capgemini, Sogeti, and Micro Focus—a comprehensive and balanced overview of the key trends shaping W quality assurance (QA) and testing today. This year’s WQR is based on a survey of 1700 executives from across 10 different sectors and 32 countries. If you are one of them, I thank you for your time and inputs. I would also like to thank our subject matter experts, lead authors for their invaluable insights and analysis, as well as the team that worked on this report. Brad Little At a very broad level, the report shows just how far QA and testing has come, from being seen mostly as a supporting Executive Vice President, function, to one that is critical to business outcomes and customer satisfaction. This has led to the increasing Global Head of importance of the ‘quality at speed’ paradigm, and is a key Application Services, factor driving many of the other trends we see in this report. Capgemini Trends in focus include the growth in adoption of agile and DevOps, the spread of digital transformation, the move away from Test Centers of Excellence (TCOEs) and the continuing low levels of basic automation that have become a critical bottleneck to the further evolution of QA and testing. In addition, the report also points to regulatory and infrastructural challenges like handling test data, provisioning environments, and finding the right talent required for testing. We also see the criticality of adopting greater levels of automation to help solve some of these challenges. Finally, we see emerging trends such as the Internet of Things (IoT), blockchain and the convergence of analytics, artificial intelligence (AI) and machine learning (ML) transforming the QA and testing function in the future. The report contains an in-depth look into the above topics through six theme chapters. This is followed by eight sector chapters, which provide an additional analysis of the important QA and testing trends in these sectors. To understand the implications of these developments for your organization, I would encourage you to get in touch with Capgemini and Sogeti experts listed on the back-cover. I hope you enjoy reading this report and that it gives you some serious food for thought. Happy reading!
5 Introduction 5 n a digital economy, enterprises must compete through faster innovation and the delivery of high quality software that satisfies their end-user’s needs. At the same time, organizations must reconcile the competing concerns of I enhanced quality and security, improved responsiveness to business demands, and optimizing budgets and resources. Over the past decade or so, organizations have adopted agile and DevOps methodologies to achieve these goals, and are now scaling beyond just one or two teams to the entire enterprise. This is true of organizations worldwide, and across every vertical. On the journey to scale, organizations face new challenges, such as end-to-end automation, and completing their testing quickly. This is due to a number of factors, such as not having enough time to test, or not having a secure, integrated, and liot Raffi Marga automated build and test environment that supports open source and third-party tools. Senior Vice President and The results from this tenth edition of the World Quality General Manager, Report are conclusive: all but one percent of organizations Application Delivery Management, are now using DevOps practices in their organization. Micro Focus Their focus is no longer on whether to move to DevOps; rather how to refine their DevOps approach and continuously improve. This may entail greater automation into continuous integration and deployment pipelines, accelerating collaboration and faster feedback, uncovering and eliminating bottlenecks, and ensuring uncompromising stability, reliability and security in production. The World Quality Report is an in-depth analysis of responses from executives in medium to large organizations, and will help you to understand their challenges, how they are overcoming them, and current and upcoming trends in software quality. I recommend that you read the report and the analyses, and in particular, the recommendations and key takeaways which you can use to take your digital transformation to the next level. With valuable lessons learned through our own DevOps transformation, we at Micro Focus are in a unique position to help our customers on their DevOps transformation, whether they are implementing solutions around cutting edge technologies such as the Internet of Things or blockchain; legacy technologies such as Cobol and mainframe computers; or both, a common scenario among our larger and more established customers. Finally, I’d like to thank our friends and partners at Capgemini and Sogeti, and everyone who helped in the research by participating in this edition of the World Quality Report.
6 World Quality Report 2018 –19 6 Executive summary
7 Executive summary 7 Mark Buenen Vice President, Global Leader, Digital Assurance and Testing Practice, Sogeti Ajay Walgude Vice President, Financial Services, Testing Leader, Capgemini to our respondents, the top objective of their QA and testing The role of quality assurance (QA) and testing has changed strategy was “ensure end-user satisfaction,” (with a weighted from mere defect finding to one of being an enabler of average score of 5.85). The expectation that QA and testing customer satisfaction and business outcomes. This is a should contribute to end-user satisfaction has grown over fundamental change and its impact can be seen in almost the years and would have been completely unthinkable every chapter of this year’s World Quality Report (WQR), ten years ago. Today, however, the increasing customer- which is based on a survey of 1,700 CIOs and other senior centricity driving major IT trends such as digitalization of technology professionals from across 10 different sectors business and the adoption of agile, DevOps, and cloud, is and 32 countries. also shifting the objectives and expectations for QA. The most telling result which demonstrates this shift in outlook, is related to QA and testing objectives. According Important aspects of IT strategy, according to the WQR survey sample Figure 1 2018 2016 2017 65% 47% **New statement** 44% 42% 42% 42% 38% 38% 38% 37% 37% 36% 36% 36% 36% 35% 34% 33% 32% 31% 30% 30% 28% 23% 19% 17% To imple- Faster Cloud Enhance Enhance Cost Implement Higher Higher Increased time transfor- ment security third- customer responsiv- optimiz- output quality of to market mation agile or party SaaS eness ation experience software DevOps to business of IT solutions demands
8 2018 –19 World Quality Report 8 AI will enable enterprises to transform testing into a fully Key trends driving IT today self-generating, self-running, and self-adapting activity. Secondly, AI developments require a new specific approach The focus on customer-centricity drives digital transformation to validation and verification. The use of AI in testing is still in and requires IT systems which can deliver attributes such as the initial stages, with organizations applying smart analytics speed, convenience and security all of which add up to a good to drive critical decisions that help optimize test activities and customer experience. The importance of such attributes do early quality predictions. According to our survey, 57% of was clearly demonstrated in our survey results. When asked respondents said they had projects involving the use of AI for about the objectives of their IT strategy, respondents gave QA and testing, already in place or planned for the next 12 the highest weighting to objectives such as “enhancing months. 45% of our respondents said they were using AI for security” (average weight of 5.97), followed by “enhancing intelligent automation in testing. In addition, 36% said they customer experience” (average weight of 5.86), and ‘higher were using AI for predictive analytics in testing and 35% said responsiveness to business demands’ (average weight they were using AI for descriptive analytics in testing. of 5.73). The testing of AI products is still emerging as a requirement. We also see encouraging signs of adoption for technologies At present, there is no coherent, widely accepted approach, such as IoT and blockchain. The number of organizations that guideline or solution for such testing and hence, organizations work with IoT in some form has increased from 83% last year are experimenting with various approaches. This also came to 97% this year. There is also a lot of focus on blockchain, through in our survey, where 57% of respondents said that with 60% of our respondents saying that they are already they were experimenting with new testing approaches for using blockchain in their portfolio or planning to use it in the the testing of AI and ML elements. coming year. There are still many challenges when it comes to AI in IT. For instance, a full 55% of our respondents said that the challenge ccording to our survey, 57% of they faced when setting up AI projects was in identifying A respondents said they had projects where business might apply AI. The use of AI in testing is also likely to require newer skills and create newer roles such as AI involving the use of AI for QA and testing, QA strategists, data scientists, and AI test experts in QA and already in place or planned for the next testing teams. 12 months. Irrespective of all these challenges, we believe that AI is going to be one of the biggest trends in QA and testing for the next two to three years and organizations will need to develop Today, an average 76% of applications across organizations a strategy around it. They will first need to reach a certain are based in the cloud. The adoption of cloud and the level of maturity in automation, then go about implementing growth in connectivity and open architectures have led to analytics and then work towards creating self-learning, self- an increased focus on security. As mentioned above, the aware systems that can be applied to testing. top objective for IT strategy today is “enhancing security.” Despite this, organizations still do not have a clear strategy for security validation, with concerns around security being Agile, DevOps and TCOEs driven more out of fear than the growth enabling abilities of The adoption of agile and DevOps is driven by the dictum well-thought out security strategy. of “Quality at Speed” and has reached a critical mass today Another key development is that while IT trends such as cloud, with 99% of our respondents reporting that they were using blockchain and IoT have not caused a major shift of testing DevOps in at least some of their projects. Unfortunately, approaches, they all require specific technical expertise. in some places this adoption of agile and DevOps has led Working with cloud requires cloud and infrastructure to a focus on speed even at the cost of quality. Our survey expertise, blockchain calls for blockchain architecture also reveals that organizations are customizing Agile and knowledge and IoT requires engineering expertise in the QA combining it with waterfall to develop hybrid frameworks and testing teams. The need for new skills as well as newer that are a best fit to their organizational, regulatory, cultural, roles in QA and testing teams is a recurring theme which and business requirements. comes up in a number of different chapters in this report. The transition to agile and DevOps has also led to a fragmentation of the QA and testing department. QA is The role of artificial intelligence in now delegated to individual project teams and thus quality today is more dependent on the skills and preferences QA and testing of the particular project or Scrum team members. This decentralization of QA makes it more difficult to leverage The growth of artificial intelligence (AI) has two technologies, best practices, and test scenarios across teams complementary influences on QA and testing. First of all,
9 Executive summary 9 Currently, the lack of test environments and data is the and can create inefficiencies or lead to sub-optimal choices number-one challenge our respondents face in applying and instances of individual teams reinventing the wheel. testing to agile development. It is also, the number-two To avoid this, many organizations are retaining a thin layer challenge for increasing the level of test automation. Thus, of centralization known as the community of QA practice data and environments are probably the second biggest or Test Excellence Center as it helps them strike an optimal bottleneck (after automation) toward greater QA and testing balance between flexible, self-running teams and the ability maturity. We also see some positive movements which will to industrialize best practices and resources. help ease this challenge in the coming years, such as the growing utilization of containerized test environments, the rise of the API economy, the use of bots for zero-touch Automation testing, the growth of open data projects, and the increasing Automation of QA and test activities has been around for number of smart solutions for better test data sampling. more than 15 years now. Over the past few years, test automation has extended from automation of test execution to automation of test case generation using model-based ccording to this year’s WQR, the average testing tools. The objective of automation has also changed as there is less focus on shortening of testing times and more A spend on QA and testing is 26%, the on better coverage and effective use of test cases. This, again, same figure as last year. So, testing spends is related to the dictum of “Quality at Speed.” have come down and also seem to have However, the level of automation of test activities is still very stabilized. low (between 14–18% for different activities). This low level of automation is the number-one bottleneck for maturing testing in enterprises. The main reasons for these disappointing levels of test automation are also revealed by our survey: 61% of our respondents have difficulties automating their Cost QA and testing processes, as their applications change too We see two divergent trends when it comes to cost and much with every release, thus indicating that they struggle efficiency of test activities. On the one hand, the waterfall- to build a robust, yet adaptable, test automation solution. In based test approaches of core IT and legacy systems addition, 48% have challenges with predictable and reusable have seen significant reductions in cost due to extensive test data and test environments, which makes it challenging automation and outsourcing. On the other hand, trends to automate testing. And finally, 46% of the enterprises also like digital transformation, the move to the cloud and pointed to a lack of skilled and experienced test automation the adoption of agile and DevOps, as well as the use of resources. Moving forward, organizations will need to move automation and analytics in QA and testing have all led to toward higher levels of end-to-end testing automation. The a wave of expenditures on new infrastructure, tooling and test automation solutions must be enhanced with smart, reorganization, and restructuring of workflows. cognitive solutions that will enable the self-running and self- adaptive test platforms mentioned earlier. These led to a spike in QA and testing budgets that we saw in 2015 (when QA and testing accounted for 35% of the IT budget) and 2016 (QA and testing accounted for 31% of the IT Environments and data budget). According to this year’s WQR, the average spend on Over the last few years, there have been a host of QA and testing is 26%, the same figure as last year. So, testing developments in IT that have transformed the QA and spends have come down and also seem to have stabilized. testing function, but the changes in test data and test However, with another wave of investments expected to environment management required to enable this are falling take place in the virtualization of test environments, test data behind. In test environments, we see a continued reliance on management, test automation, and the use of analytics across permanent environments, with 31% of testing occurring in the testing lifecycle, this proportion might again increase to such environments. Test data management and provisioning around 30% over the next two to three years, followed by too are not maturing, as the number of enterprises that cite another period of stabilization and increased efficiencies. challenges in these areas is not dropping at all (50–60%). The number of organizations leveraging modern test data provisioning technologies is low, indicated by the fact that 58% of our respondents still rely on manually generated test data. According to our survey, 66% of our respondents said they use spreadsheets to manually generate new test data and 62% are using copies of production data to perform testing.
10 2018 –19 World Quality Report 10 World Quality Report findings Our survey reveals a lot of excitement and experimentation The first time ever that “end-user around the use of AI, ML, and analytics technologies. On satisfaction” is the top objective of QA average, organizations are devoting 22% of their IT budget and testing strategy to AI projects and 57% of our respondents said they had AI projects for quality assurance, already in place or planned for Expecting QA and testing to directly contribute to “ensuring the next 12 months. end-user satisfaction” is not an obvious or intuitive A new trend in evidence this year is the use of analytics to expectation. However, this year, it came out as the top drive optimization prior to automating test sets. According to objective of QA and testing strategy. Forty two percent of our survey, 54% respondents said they were using analytics the respondents indicated that this was an important from project data to optimize test sets, while 45% stated objective, up from 34% last year. In addition, “enhancing they were using data from operations for optimization, and customer experience” emerged as the second most another 34% said they were using code coverage analytics. In important aspect of IT strategy, with 42% of our addition, as many as 45% of our respondents said that they respondents indicating it as such. It was only superseded by were using analytics and AI for intelligent automation of QA “enhancing security,” which had 47% of respondents processes, 36% for predictive analytics and 33% for creating indicating it as important. self-learning, cognitive platforms. ur survey also revealed the increasing use of Our survey also revealed the increasing use of bots for bots for test activities. As many as 79% O test activities. As many as 79% respondents said they were currently using or planning to use bots for the set-up of test respondents said they were currently using or environments, while another 78% were using or considering planning to use bots for the set-up of test using bots for lifecycle test automation, and 77% were using environments, while another 78% were using or or considering using bots for test data generation. Trends considering using bots for lifecycle test such as the use of AI, ML, and analytics are likely to come automation, and 77% were using or considering together to help create self-learning, self-aware systems using bots for test data generation. that will exponentially increase the benefits delivered by automation and increase the efficiency of QA and testing. These trends are remarkable, for not only do they recognize Low levels of automation and challenges the impact that QA and testing and IT can have on the end- customer’s experience, but they also give it the greatest with test data and test environments importance. This means QA and testing processes, or IT holding back QA and testing efficiencies systems that support speed, quality, and convenience (in other words, quality at speed). Over the next two to three Agile and DevOps adoption continues to grow driven by years, we expect these objectives to become powerful drivers the “need for speed,” agility, and flexibility. According for digitalization and the greater adoption of cloud, agile to our survey, 99% of respondents said they were using and DevOps. DevOps in at least some part of their business. Despite this growth in adoption, organizations are still not able to The convergence of AI, ML, and analytics and their use tap the full benefits promised by these approaches mainly in carrying out smarter automation will be the biggest due to low levels of automation and challenges with test data disruptive force which will transform QA and testing over the and test environments. next two to three years.
11 Executive summary 11 elements such as AI and ML in QA and testing. Apart from this This came through clearly in our survey. When asked about a few other things such as the adoption of containerized test the challenges in applying testing to agile development, the environments, the rise of the API economy will also help. biggest challenge came out to be “lack of appropriate test environment and data,” followed by “inability to apply test automation at appropriate levels.” The number of respondents The skills required for QA and quoting these challenges have also risen from 46% last year testing have changed to 53% today for “lack of appropriate test environment and Today, the way QA and testing activities are executed has data,” and from 41% last year to 50% this year for “inability changed. On the one hand, the adoption of new frameworks to apply test automation at appropriate levels.” Additionally, and technologies has broadened the number of skills when asked about the technical challenges in developing required for testing, while on the other, testing activities applications, respondents gave the highest weighting to “lack too have spilled over to other domains and functions such as of end to end automation from build to deployment,” with Development and Business Analysis. Thus, today, everyone 55% of them indicating this as a challenge. has a role to play when it comes to QA and testing. We need much more open, loosely formed teams with some people Interestingly, automation too is held back by challenges who can provide specialized, supporting expertise (for things related to test data and environments. So, when respondents like AI or analytics) and some people who possess the core were asked about the challenges in achieving their desired testing skills and mentality and can lead the testing effort. level of test automation, 48% of them reported challenges with test data availability and stability, making it the second- In the previous years WQR, we saw the need for Software highest ranked objective. These issues around test data and Developer Engineer in Test (SDET) profiles in testing teams. environments are the result of several factors, including the This year, with greater experimentation around technologies greater frequency of releases, the increased complexity of such as AI, analytics and IoT, there is a further need for much test data and environments, newer types of data, challenges more specialized skills in test teams. This also came through with integration and standardization of data coming from in our survey, where 42% of respondents said that the lack of different sources, and the regulatory requirements relating proper skills for QA and testing was an important technical to data brought about legislations such as the General Data challenge they were experiencing in developing applications. Protection Regulation (GDPR) and the International Financial Similarly, 36% of respondents said that with the advent of Reporting Standard 9 (IFRS 9). AI, they needed a greater understanding of AI’s implications on business processes among their QA and testing teams. These challenges around automation, test data, and Additionally, 30% of respondents stated that they need more environments, create a situation where organizations are functional automation expertise, 29% said they needed more unable to keep pace with the volume and frequency of of test environment, TDD (test-driven development) and testing required. Essentially, they slow down testing, thus BDD (behavior-driven development) expertise, and 28% have defeating one of the main objectives of adopting frameworks stated that they need predictive analytics skills. such as agile and DevOps. This also came through in our survey results, when 43% of respondents said that “too slow With the new trends and adoption of technologies seen this testing process” was a challenge when it came to developing year, it is clear that the skills challenge is not going to go applications today. away anytime soon. Unless organizations take active steps to retrain their employees and develop these skills, this The biggest factor that will help solve these challenges could soon emerge as a critical bottleneck holding back the and allow organizations to derive the full benefits of Agile progress of the QA and testing function. and DevOps is greater automation and the usage of smart
12 2018 –19 World Quality Report 12 Key Recommendations of infrastructure, and in general, lead to greater efficiencies. Increase the level of basic and smart Organizations must move towards “smart” test data and test automation but do so in a smart, test environment management, i.e. the creation of self- phased manner provisioning, self-monitoring and self-healing environments and data based on specific requirements. Leveraging such We believe that automation is the biggest bottleneck holding technologies, together with a more centralized approach back the evolution of QA and testing today. This is due, in to test data and environment provisioning will help ensure part, to automation’s key role as an enabler of successful the optimal level of risk coverage, speed up releases, agile and DevOps transformation. With increasing agile and cut infrastructure costs, and increase team productivity DevOps adoption (99% according to the 2018 survey), the and throughput. importance of automation for delivering “Quality at Speed” has also risen. It is also a result of new technologies such as AI, ML, and analytics, all of which hold significant promise in Build quality engineering skills terms of the benefits which automation can deliver. Finally, beyond SDETs our survey also reveals that levels of basic automation are In our last couple of reports, we have talked about the need still quite low (between 14%–18%), thus indicating significant for SDETs in testing. However, with trends like agile, DevOps, scope for growth. cloud, IoT, blockchain and AI on the one hand and the need For all these reasons, automation and especially smart test for a more automated and integrated QA approach on automation, is poised to bring about significant changes in the other, enterprises today need to focus on new quality the way QA and testing is done over the next two to three engineering skills. years and organizations need to have a strategy and roadmap in place if they want to reap its benefits. We recommend a Given these challenges, we recommend that organizations phased approach in three stages, i.e. first, the optimization take the following approach toward building up the required of testing, second, implementation of basic automation, QA skills: and third, the adoption of intelligent and self-adaptive test automation solutions to make automation “smarter.” The first priority is to attract/reskill towards agile test specialists who have functional automation skills and Implement a non-siloed approach for test 1 domain testing skills. We would recommend that environment and data provisioning automation be a must-have skill for everyone in the QA function today. Fifty-three percent of our respondents said they were facing challenges with test data and environments when applying testing to agile development. Such challenges negate all the The second priority is to attract/reskill Software 2 efficiencies gained through the adoption of cloud, agile and Development Engineers for Test skills (SDET). The DevOps by introducing delays in the provisioning of test data SDET must have advanced automation skills, white and environments. This is an area that needs urgent attention box testing capabilities, development skills, and the and we recommend taking a centralized approach to solving ability to build orchestration platforms. They may also these issues. Organizations must start thinking in terms of be required to possess basic algorithmic application lifecycle automation, i.e. automation of testing and the capabilities, and natural language processing skills in provisioning of test data and environments together, rather the event of it being an AI application. than in siloes. A certain level of centralization will result in better leveraging of best practices, tools, and techniques, lead to better re-use
13 Executive summary 13 The third priority is to ensure sufficient niche need to track and understand these spends as well as the QA skill sets such as security, non-functional returns on these investments. 3 testing, test environments and data Thus, we would recommend that organizations take this management skills issue of budgets and spends seriously and create a detailed and elaborate tracking mechanism to see how and where budgets are allocated and then track how and where they are spent. Such a mechanism would need to be quite granular, as The fourth priority is to attract/reskill advanced QA 4 project teams consisting of different skills sets and carrying experts with AI architecture skills to build” smart out a variety of tasks make it difficult to understand whether assets” that perform both repetitive and intelligent a certain spend went into testing or development tasks. These skills comprise of deep machine learning concepts, algorithms such as decision trees, classifiers Develop a testing approach for and neural networks, advanced statistics, and data optimization skills. AI solutions now According to our respondents, in 2018, an average 22% of their IT budgets are being allocated to AI projects. In addition, 64% of our respondents said they had AI projects in place or planned for the next 12 months for customer Of course, getting the kind of resources listed above is a processes, 62% said the same thing for internal processes, tough ask in today’s market, which is why we recommend and 57% said so for QA purposes. Thus, the adoption of AI is a greater focus by organizations on building up these skills clearly increasing. through workforce transformation programs comprising of internships, specialized trainings, and mandatory learning However, when asked about the measures they were taking and development plans. toward the testing of “intelligent applications” (including AI and ML elements), as many as 57% of the respondents said Improve tracking to optimize spends they were experimenting with new testing approach and 45% said they were investigating a suitable testing approach. The adoption of agile and DevOps in project teams has led to Given the rate at which AI is coming up, every organization a situation where QA and testing activities are being done by needs to start developing a QA and test strategy for their AI many, including developers as well as specified testing solutions now. Organizations must realize that the potential professionals. This makes it tough to accurately track, business and social impact of incorrect or faulty AI solutions understand, or optimize QA and testing spends. In addition, can be huge. The validation and verification of AI solutions the last few years have seen increased investments in during development, as well as automated continuous automation tools and cloud-based, virtualized, and quality monitoring must be part of this QA strategy for AI. containerized test environments, and there is a pressing We would like a system that could repair itself. It should be capable of automated code creation or code correction. — Chief Product Officer, Technology, Germany
14 Current Tre nd s i n Quality Assurance & Te s t i ng
15 Key trends in IT 16 The brave new world of QA and testing Artificial intelligence 22 A force multiplier for QA and testing Test automation 26 The single-biggest enabler of maturity in QA and testing The quality assurance organization 31 Greater adoption, newer challenges, and a move away from Test Centers of Excellence Test data and environments 36 management The big gest obstacles to progress in QA and testing Efficiency and cost containment 40 in quality assurance QA and testing costs showing signs of stabilization, though a wave of investments in new technologies is expected
16 2018 –19 World Quality Report 16 Key trends in IT The brave new world of QA and testing Prabakaran Karuppiah Associate Vice President, Shiva Agolla Antoine Aymer Digital Assurance, Sogeti, Capgemini Group CTO, VP - Testing Services, Capgemini Global Director, Sogeti Services Malcolm Isaacs Senior Director Vivek Jaykrishnan - Technology, Engineering Services Global ADM Solutions Marketing Business Line, Capgemini Manager, Micro Focus of ensuring end-user satisfaction as a key objective of the QA The quality assurance (QA) and testing function has evolved considerably over the last few years. We see six key trends and testing strategy. This year, it has come on top (cited by that will have an enormous impact on how QA and testing 42% of respondents and with the highest average rate of 5.85 evolves over the next two to three years. These trends are: out of 7). It has also consistently scored as one of the most digital transformation and the API economy, the internet of important elements of IT strategy over the last three years. things (IoT), artificial intelligence (AI), cloud, cybersecurity, For instance, in 2018 it was cited as an important objective and blockchain. In this chapter, we will be taking a more in- by 42% of respondents, surpassed only by the overarching depth look into each of these trends, with the exception of AI, concern to enhance security (cited by 47%). which has a separate chapter devoted to it. This focus on customer experience also drives the primacy of To understand these trends, it is important to know what objectives, such as time to market and is responsible for the drives them. In this regard, the most important development increasing adoption of agile and DevOps across the world. It is is the increasing customer centricity and the consequent also what drives firms’ attempts to capture, analyze, and utilize importance of end-user experience. IT today is not just about the masses of structured and unstructured data produced by ensuring system availability, functionality, and cost reduction. consumers today. Together, these developments shape each It is also expected to contribute to business goals. For the of these trends in different ways. third year in a row, our WQR survey confirms the importance Executive management objectives with QA and testing Figure 2 2018 2016 2017 42% 42% 41% 38% 41% 40% 39% 39% 38% 39% 38% 37% 37% 36% 35% 34% 35% 30% 29% 29% 26% 27% 28% 21% Reduce the Detect Ensure Implement Increase quality Protect the Increase the Contribute to overall application end-user software quality awareness corporate quality of business cycle times by satisfaction defects before checks early in among all image and software growth and go-live reducing waste the lifecycle disciplines branding or product business outcomes
17 Current Trends in Quality Assurance & Testing 17 Digital transformation and the API economy Digital transformation refers to the use of digital technology third principle is the appropriate leveraging of technology. to deliver services faster, cheaper, and better to consumers. This is where we see elements such as cloud, automation, IoT, This requires a change in almost every part of an organization’s microservices and analytics working together. ecosystem, including business models, organizational The WQR 2018 survey, reveals that digital transformation frameworks, technology, employee skill sets, and most creates higher demands on QA and testing approaches and importantly, its culture and mindset. All these elements need that a large proportion of enterprises have some serious to add up to a cohesive whole that drives business value and challenges. For instance, when asked about their greatest ultimately serves customers better. Getting this mix right is what allowed smaller firms such as Uber, Airbnb, or Netflix to challenges in testing mobile, web, and other types of front- bring disruptive change in their industry. office applications, 52% of our respondents pointed to “not enough time to test” as an issue, followed by 43% who said, Broadly, there are three key principles for achieving this “we don’t have the right tools to test” and 34% who said, successfully. The first is to re-think your business models and “we do not have the right testing process or method.” Such organizational frameworks. Usually, this means adopting challenges could arise either due to organizations sticking to business-aligned approaches such as agile and DevOps. The a traditional waterfall approach, an issue with skill sets, or a second principle is the use of data and its integrated approach lack of the cultural transformation required by frameworks to drive better customer service. In a world of ever-increasing such as agile and DevOps. data, this is the key to competitive advantage. Finally, the Challenges with testing mobile and multi-channel (mobile, wearables, social and traditional) apps Figure 3 2016 2018 2015 2014 2017 52% 52% 46% 46% 46% 45% 44% 44% 43% 42% 48% 47% 41% 40% 40% 38% 36% 35% 34% 33% 36% 30% 29% 34% 28% 28% 26% 26% 26% 28% 13% 8% 5% 4% 1% n't have We do We don't have n't have We do We do n't have We don't have n't have We do We don't the right enough time to the right testing in-house testing mobile testing the devices do mobile tools to test test process/method environment experts readily available testing
18 2018 –19 World Quality Report 18 Despite these challenges, we see most organizations moving Internet of things ahead on the path to digital transformation. Today, driven by the need to serve their customers, many companies The internet of things (IoT) promises masses of richly detailed have started using technologies like digital assistants, data on customer behavior and its adoption has been rising Bots, augmented reality, voice and face recognition, etc. across industries. According to our survey, the percentage The promise of these technologies also came through in of respondents that do not have IoT products has decreased our survey. For instance, 54% of our respondents said they sharply from 17% in 2017 to 3% in 2018. The data also shows foresaw themselves using robotics automation in the coming us that the adoption of IoT is increasing sharply in Energy and year. A key new development in this regard is the growth Utilities, High-tech, Public Sector, Transport, and Financial of microservices and the API economy. This refers to an Services. approach where bigger applications are broken down into Of course, IoT also introduces new challenges around the smaller, more manageable pieces called microservices. It effective usage of the vast amounts of user data it generates. simplifies development, testing, and release and allows for One such challenge is related to the proper testing of regular incremental improvements, lowered risk, greater IoT products. This came out in our survey as 34% of our flexibility, and improved time-to-market. respondents said, “our products have IoT functionality but do not have any specific test strategy, though we plan to However, it is also important to remember that this approach include such strategy in the near future,” while 22% stated has disadvantages. While splitting a bigger application into that “our products have IoT functionality but currently we do smaller containers might result in better time-to-market, it not have any specific test strategy.” Encouragingly, this year also creates the need for monitoring each of the individual has seen an increase in the number of conversations taking components. While it simplifies the testing of individual place around end-to-end test strategies for IoT deployment. components, it makes testing of the whole, integrated This implies greater awareness, if not actual implementation system tougher. of proper testing strategies for IoT products. Organizations with specific test strategies for testing products in an internet-of-things (IoT) environment Figure 4 6% 5% 4% 6% 5% 1% 1% 2% 3% 4% 11% 21% 16% 26% 17% 19% 18% 22% 25% 27% 28% 44% 32% 32% 29% 28% 26% 34% 36% 42% 34% % 4 3 46% 41% 43% 42% 48% 39% 48% 50% 3% 38% 3 36% + including Media Hi tech Public Sector/ Government Automotive Sciences Telecommunicat- Healthcare & Li fe hardware ions, Total Energy, Utilities Manufacturing Financial Services industry Consumer Products, Retail, Transportation We do not work with IoT products solutions or we do not have specific test strategies Our products have IoT functionality, but currently do not have any specific test strategy Our IoT products and solutions do not have any specific test strategy currently, but we plan to include one in the near future We have a fairly mature IoT test strategy
19 Current Trends in Quality Assurance & Testing 19 Cloud all applications are based in the cloud. Both private cloud Digital transformation, the adoption of agile and DevOps as and hybrid cloud are proving to be popular, with 22% of all well as the increased need for greater analytics capabilities applications running in a private cloud as opposed to 15% of have all been driving cloud adoption over the last few all applications which run in a hybrid cloud. years. According to this year’s survey, as many as 73% of Percentage of applications currently hosted in different types of cloud Figure 5 2016 2017 2018 Private cloud Non-clo ud- 30 ed bas % 27 % Public cloud 24 % 22 On-premises 23 % cloud % 19 Hybrid % cloud 18 19 20 % % % 17 18 18 % % % 15 15 % % This increase in cloud adoption comes with its own set of In addition to our survey, we see two interesting, recent trends. challenges. For instance, when asked about the testing of The first is the rise of the multi-cloud. To prevent vendor lock- cloud-based or third-party SaaS services, as many as 60% in, an increasing number of companies are hedging their bets of the respondents said they paid special attention to and hosting different applications with different cloud service security requirements and risks, while 52% reported paying providers. This is a growing trend and we expect this will soon special attention to performance requirements and risks. be the norm when it comes to the cloud. The second trend has Anecdotal evidence suggests that the integration of to do with edge computing. This is a nascent trend, but as IoT third-party services is another challenge faced by a growing and its associated technologies mature, more companies will number of organizations. be adopting a mixture of edge computing and cloud hosting in a manner that optimizes results. Every part of the production cycle will be affected by artificial intelligence and QA is not an exception. Logistics, Italy — VP Applications,
20 2018 –19 World Quality Report 20 Cybersecurity This coming together of several trends, such as cloud, IoT, This concern around security can be seen from our survey as well as digital transformation and customer-facing apps results as well. For instance, when asked about the objectives have all created the perfect storm for businesses. On the one of their IT strategy, respondents gave the highest weighting hand, organizations now have both the motivation (given the to enhancing security, with 47% of respondents naming increased focus on customer centricity), and the ability to it as an important aspect of their IT strategy. This concern track, store, and use customer data to serve their consumers regarding security seems to have risen since last year’s better. On the other hand, there are increasing concerns over survey, in which only 35% of respondents saw security as an data privacy and security. This situation is compounded by important objective. a lack of clarity around standards and security protocols for emerging technologies such as IoT. Important aspects of IT strategy, according to the WQR survey sample Figure 6 2018 2017 2016 65% 47% 44% 42% 42% 42% **New statement** 38% 38% 37% 38% 37% 36% 36% 36% 36% 35% 34% 33% 32% 31% 30% 30% 28% 23% 19% 17% Faster Cloud To imple- Enhance Increased Cost Higher Implement Higher Enhance time ment transfor- security output optimiz- customer responsiv- third- quality of mation agile or to market ation experience eness party SaaS software DevOps of IT to business solutions demands storage, retrieval, and sharing of customer data, is applicable When it comes to protecting customers’ data, the European to every organization that operates in the European Union. Union’s General Data Protection Regulation or GDPR, which Though it has already come into force, there is still a lot of came into effect on May 25 of this year, has been a milestone. confusion in the market about its impact on IT operations. The GDPR, which mostly consists of regulations concerning the
21 Current Trends in Quality Assurance & Testing 21 Blockchain the higher side but nonetheless, this statistic is indicative One of the hottest technology trends today is blockchain. of the high level of awareness and excitement around this While it was first put to use in the Financial Services industry technology. for cryptocurrencies, more and more industries today are experimenting with use cases to leverage this technology. Of course, with blockchain being a new technology, it has also given rise to concerns. When asked about the major The excitement around blockchain was also captured in our unknowns about blockchain in their portfolio, 52% of the survey results. When asked about their plans for blockchain, respondents pointed to security-related risks, followed by 66% of respondents said they were already using it in their 45% mentioning data-related risks, and 38% identifying risks portfolio or planned to do so in the coming year. Anecdotal of integration with the rest of the landscape. evidence suggests that this number might be slightly on Are you already using blockchain in your Major unknowns with blockchain in the portfolio portfolio today or planning to use it in the coming year? Figure 8 Figure 7 52% 2018 45% 38% YES 35% NO 27% 34% 66% Other Business Risks of Data- Security- integration risks technology related related risk (block-size, with rest of the risks risks landscape performance) Summary means that testing too must support business goals and start The IT industry is going through a period of rapid change, driven by increasing competition, newly emerging thinking beyond simple identification of defects or delivery of technologies, and new frameworks and modes of operation. services. Instead, testing has a positive contribution to make Although these trends have enabled newer capabilities, in terms of delivering a superior quality product on budget they have also increased the complexity and challenge with and ahead of schedule. regard to IT operations. Together, these changes have several Finally, it’s also important to remember that the new IT implications for QA and testing. model is not just about the latest technologies or improved Firstly, they have sped up the entire IT development, processes. Above all, it is a change in culture, attitude, and testing, and release process due to which testing speed and mindset, which will also require a change in the traditional collaboration have both become critical. This has led to a lot ways of working or delivering services. This means that of focus on “QA at speed” as well as on decentralizing testing people, processes, and technology will all have to go through to make testing operations faster and more collaborative. a period of change and improvement before we can fully realize the benefits promised by the new technologies Secondly, all these changes revolve around the need for and frameworks. greater customer centricity and business alignment. This
22 2018 –19 World Quality Report 22 Artificial intelligence A force multiplier for QA and testing Deepika Mamnani Gerhard Dupjan Tom Van De Ven Head of AI, Senior Director, Financial Services, Senior Test Consultant, Capgemini Quality Assurance, Sogeti Germany Sogeti High Tech Dhiraj Sinha Albert Tort Pugibet Stefan Gerstner Technical Director & Innovation Manager, Vice President, Digital Assurance Vice President, Financial Digital Assurance & Testing, Sogeti Spain and Testing, Sogeti Germany Services Testing, Capgemini QA tasks to the most productive professionals in a team in When it comes to artificial intelligence (AI), one must keep order to reduce cost and time to market. However, the ability two things in mind when interpreting this year’s WQR to quickly make progress in this area seems to hinge on an survey results. The first is that AI in testing refers both to the organization’s maturity in mining data. For AI-driven QA, this application of AI to quality assurance (QA) and testing, as well typically means data from application lifecycle management as to the testing of AI products. The second thing to be kept in (ALM) tools, such as test case coverage, execution and defect mind, is that AI is an emerging technology and the knowledge, data, and production data as well as code coverage and expertise, and maturity required to apply it to QA is still operational logs. lacking in many organizations. Clearly, there is enthusiasm for and excitement about AI technologies and solutions, but their Data is, thus, one of the biggest challenges when it comes to actual application in testing is still emerging. the adoption of AI in testing; not just the ability to work with it but also the quality and quantity of the data available. The Our survey results unambiguously reflect this excitement. other big challenge lies in identifying possible use cases for They also reveal that there is a lot of experimentation going AI in testing. In addition, the use of AI in testing creates roles on with AI technologies. The purpose of applying AI to QA that are very different from the typical tester profiles. These and testing is simple: to create a smarter and faster testing roles require a different mix of skills comprising testing, architecture that runs and adapts itself automatically to development, natural language processing, data science, application changes. A first essential step in this process is mathematics, algorithmic knowledge, and machine learning. applying smart analytics to critical decisions in the testing This is an aspect that we expect will become critical in process, such as which tests to run, how many tests to run, the future. etc. Today, smart analytical solutions can automatically Despite such challenges, AI is here to stay and its importance select and prioritize test cases, assist in the creation of test- will only grow in the coming years. case design, identify risk factors, and automatically assign Artificial intelligence: The oft-repeated (but often misunderstood) mantra Our survey results indicate that, across sectors, organizations or plans they had in place for the next twelve months, a are devoting an average of 22% of their IT budgets to AI full 64% said they had AI projects in place or planned for projects. Interestingly, this percentage is highest for the High- customer processes, followed by 62% for internal processes, Tech sector (25%), followed by organizations in the Financial and 57% for quality assurance purposes. While expert opinion Services (23%), and Energy and Utilities sectors (23%). As holds that these numbers reflect plans and aspirations mentioned earlier, this probably reflects the development of rather than implemented projects, the fact that a majority better AI use cases in these sectors. of organizations are at least talking about it, reflects the potential that businesses see in this technology. In addition, when respondents were asked about AI projects
23 Current Trends in Quality Assurance & Testing 23 Artificial intelligence projects or plans for the next 12 months Figure 9 NO YES 36% 43% 45% 45% 38% 55% 55% 57% 62% 64% In place or In place or planned In place or In place or planned In place or planned planned on on external on development planned on on quality customer processes processes assurance and production internal processes with how companies approach new technologies, such as AI. However, translating that potential into reality is not always easy. In their engagement with AI, most organizations are Many of them first build up their knowledge and expertise in still stuck at the level of data analytics rather than using AI the technology and then try to build a business case for it. technologies such as machine learning, neural networks, Apart from the above challenges, the survey results also fuzzy logic, robotics, or deep learning. They are also struggling pointed to “difficulty integrating AI with the existing with how to impact business outcomes using AI technologies. applications” (51% of respondents) and “(lack of) available This came through clearly in our survey results. For instance, AI knowledge in development” as two big issues. In addition, when asked about the challenges they faced or expected to expert opinion also points to problems with the quantity and face in implementing AI projects, a full 55% of respondents quality of data as one of the biggest obstacles holding back reported that they had “difficulties with identifying where the greater adoption of AI for QA and testing. business might actually apply AI.” Some of this also has to do Challenges encountered or expected when setting up an artificial intelligence project Figure 10 2018 55% 51% 46% 32% 38% Available Available AI Available Difficulty integrating AI Identifying AI knowledge knowledge with the existing structured/ where business in development in quality assurance applications unstructured data might actually apply AI
24 2018 –19 World Quality Report 24 AI: What’s it good for? intelligent automation, 36% for predictive analytics, and 35% So how are organizations using AI in QA and testing today? for descriptive analytics and self-learning cognitive platforms. The 2018 WQR survey results give us a few indications. These trends are directly related to some of the trends, Our survey results confirm that speed-to-test using intelligent such as predictive analytics, robotics automation, cognitive automation and optimization is where AI is playing a key role automation, and machine learning that were identified as in QA, with 45% of respondents stating that they use AI for emerging trends in this year’s survey. The use of analytics and artificial intelligence for optimizing quality assurance Figure 11 45% 11% 35% 36% 35% 33% 2018 Yes, for Yes, for No, our data Yes, for Yes, for Yes, for intelligent predictive quality does selflearning, descriptive cross application automation not allow this analytics cognitive dashboards analytics platforms they were investigating a new testing approach, and 35% The complexity of AI, together with its growing popularity said that they were using a new testing approach. From the across all business areas, leads us to the question of how results, it is clear that no standard processes have yet been enterprises are dealing with the testing and validation of AI developed and that different organizations are experimenting solutions. When respondents were asked how they tested in different ways. such intelligent applications, 57% said that they were experimenting with a new testing approach, 45% said that Changing roles in AI from the High-Tech sector and 42% from Energy and Utilities, Our survey yielded some interesting results when it came and Financial Services reported this as a challenge. to the skills required for both applying AI in the test process as well as for the testing of AI products. For instance, when This complacency regarding the skills required for working asked about the challenges they faced or expected to face with AI was also seen when respondents were asked about when setting up an AI project, only 38% of the respondents the extent to which AI changed the skills they needed from reported “the lack of available AI knowledge in testing” as their QA and testing professionals. As many as 51% of an issue, thus ranking this statement second last among respondents said that AI did not change the skills required for all options given. Interestingly, the sectors that displayed test data set-up expertise, 50% said there were no changes greater maturity in the usage of AI technologies tended to required in test strategy, and test design skills, and 49% said see this as more of a challenge. Thus, 43% of respondents there were no changes required in data science skills.
25 Current Trends in Quality Assurance & Testing 25 The extent to which AI changes the skills expected of QA and testing professionals Figure 12 31% 31% 28% 35% 34% 36% 32% 45% 49% 49% 48% 51% 50% 53% 20% 19% 18% 18% 16% 18% 18% Understanding of Non-functional Test data Data Development Software Test strategy and coding AI implications on Science set up and test development testing skills skills design skills skills engineer (performance, business expertise processes security) testing skills Skills are less relevant Skills are OK- no change required Skills are lacking To do this, they will need to have business knowledge as Experience however, suggests that AI changes the skills well as a broad understanding of data and techniques such required from QA professionals. The traditional tester is no as mathematical optimization, natural language processing, longer adequate, as working with AI requires professionals and robotics with a diverse range of competencies such as testing, mathematical optimization, neuro-linguistic programming, In the future, data scientists will also need to Data scientists: AI, business intelligence skills, and algorithmic knowledge. be a part of QA teams. These professionals will need to sift At present, finding this combination of skills is difficult and through data and use predictive analytics, mathematics, and experts suggest that challenges regarding the availability of statistics to build models. qualified professionals will increase in the future as more organizations start experimenting with AI. They will need to have the traditional AI test experts: testing skills as well as the ability to build machine learning Moving forward, as the practice of AI matures, we see three algorithms, mathematical models, and natural language new roles emerging in QA and testing: processing models. These professionals will need to AI QA strategists: understand the implications of AI for business processes. Summary Over the next two to three years, we will see an increase AI is an exciting new technology, with many possible business in the use of data and test automation to become more applications and considerable scope to evolve in the future. predictive when it comes to QA and testing. We are also likely Our survey results clearly demonstrate the excitement around to see an adoption of AI across the entire lifecycle, extending it as well as its challenges. As an emerging technology, there from the requirements side, through operations, to customer is still some lack of clarity around what business applications experience. All in all, the testing profession can look forward it can be used for as well as on how to use it. Such questions to some exciting developments ahead. are likely to become clearer as time goes by.
26 2018 –19 World Quality Report 26 Test automation The single-biggest enabler of maturity in QA and testing Brian Olsen Antoine Aymer Sridhar Throvagunta Senior Consultant, Sogeti Denmark Global Director, Sogeti Senior Director, Financial Services, Capgemini end-user satisfaction,” another 42% (34% in 2017) pointed Nothing can stop an idea whose time has come, and test to “detect software defects before go-live,” and 41% (29% automation today, is just such an idea. The time is exactly right, in 2017) mentioned “contribute to business growth and because nearly every major IT trend we see simply reinforces business outcomes” as important objectives. the need for greater automation. Take, for instance, the focus on time to market, the rising adoption of agile and devOps, the convergence of technologies, such as machine learning Additionally, 29% of the respondents said that “reducing (ML), artificial intelligence (AI), and analytics and in strategic overall application cycle times” was an important objective terms, the increasing alignment between IT and business. All for QA and testing. All of these are business goals and they these trends contribute to the ever-increasing importance of reiterate a theme we have been seeing for several years now automation; either because automation is a critical ingredient – the tighter coupling of IT with business. For QA, this means for these trends to succeed, or, because they have a multiplier an increased focus on the concept of “Quality at Speed” effect on the benefits that automation can offer. and its associated promises of avoiding human intervention wherever possible, reducing costs, increasing quality, and Our 2018 WQR survey, unambiguously demonstrates these achieving better time to market. And the way to achieve each trends. When respondents were asked about the objectives Automation. of these goals? of their QA and testing, 42% (28% in 2017) picked “ensure Executive management objectives with QA & testing Figure 13 2018 2016 2017 42% 42% 41% 38% 41% 40% 39% 39% 38% 39% 38% 37% 37% 36% 35% 34% 35% 30% 29% 29% 26% 27% 28% 21% Ensure Reduce the Detect Implement Increase quality Protect the Increase the Contribute to software end-user overall application quality awareness corporate quality of business satisfaction cycle times by defects before checks early in among all image and software growth and reducing waste go-live the lifecycle disciplines branding or product business outcomes
27 Current Trends in Quality Assurance & Testing 27 There are two additional factors driving the adoption of Finally, we are also seeing the adoption of analytics, AI, and ML in testing, a trend that started in earnest recently and is only automation. First is the adoption of agile and DevOps, which likely to gain momentum in the coming years. In our survey, seems to have reached a tipping point today, with 99% of as many as 45% of respondents said they used analytics our respondents saying they use DevOps for at least some and AI for intelligent automation of QA processes. This is an part of their business. This also increases the importance of important development, one which holds promise in terms of automation, since one cannot derive the full benefits of agile the benefits that automation will deliver in the future. and DevOps without automation. Automation today and tomorrow For many years, the focus of test automation was on test provisioning) along with traditional end-to-end automation. execution and functional testing, and automation used to Another emerging trend is that of automating API testing happen in silos. So, we had point solutions for automating and we are likely to see increased activity around this over the individual processes, such as test execution. Today however, next couple of years more and more components of the lifecycle are being So, what does automation look like today? According to our automated and there is a focus on automating the entire respondents, automation continued to be the most popular lifecycle. Take, for instance the increasing importance of for generating and executing functional test cases. On robotic process automation (RPA), which seeks to automate average, 18% of functional test cases were generated using manual business processes, thereby reducing costs and test generation tools, and 16% were executed using test improving speed and quality. Similarly, today’s ever-changing automation tools. Similarly, 16% of all security tests were applications and the increased complexity and number of executed using automation tools, and automation was also new releases, have also led to a rise in the importance of applied to the execution of 16% of all performance test cases. both model-based testing (for automated test cases and Quite encouragingly, 15% of all end-to-end business scenarios automated script generation based on requirements) and were also being executed using test automation tools. eco-system automation (for test data and test environment Indicative proportion of automation of each of the following activities Figure 14 2018 2017 **New statement** 16% 15% 16% 16% 14% 15% 18% 16% 16% 15% 16% 15% 16% % of % of functional % of security % of API % of % of test % of end tests that are test cases that performance data that is test-cases -to-end business functional test cases that are scenarios that are generated test cases that generated executed that are are executed automated with test by test data executed are executed with test automation tools with test with test with test generation automation automation tools tools automation tools tools tools automation, and 49% said cognitive automation. We have To understand future trends, we asked respondents about already touched upon model-based testing, a trend facilitated new automation techniques they could foresee, using in the by the increased availability of both commercial and custom- coming year and 61% percent said “model-based testing.” made, model-based tools and accelerators today. A further 59% said predictive analytics, 54% said robotic
28 2018 –19 World Quality Report 28 Projected business interest in automation techniques in the coming year Figure 15 61% 59% 54% 47% 48% 45% 42% 42% 41% 43% 42% 40% 40% 40% **New statement** 40% 38% 40% 39% 39% 36% 35% 34% 31% 30% 21% Cognitive Machine Model-based Predictive Robotics Test data Test Test design Self- automation learning testing (automated automation analysis automation environment automation remediation test cases design) virtualization 2016 2017 2018 suggestions in a non-intrusive way that does not hold up Organizations today, are trending towards lifecycle test development, this would also facilitate a greater adoption of automation, which means achieving an assembly-line-like agile and DevOps. test lifecycle. This is achieved by orchestrating automation in an ecosystem comprising proprietary, open-source, and However, before we reach this golden age, there are commercial tools. The next step is smart test automation, significant challenges that need to be overcome. which is achieved by bringing together solutions, such as model-based testing, optimization solutions, scriptless test automation, test data and environments management, The many obstacles to success service virtualization, nonfunctional engineering, etc., and When asked about the main challenges in achieving their orchestrating them to achieve lifecycle automation besides desired level of automation, 61% of respondents said they infusing intelligence through analytics and dashboards, RPA, had difficulties automating as their applications changed with and other non-intrusive proactive techniques to course- every release. This could be a direct result of the flexibility correct development. This trend will develop fully over provided by frameworks like agile and DevOps, which allow the next two to three years and our survey confirms that organizations to change their requirements or stories organizations are already experimenting with and adopting frequently. This often leads to too many changes with every many of the technologies discussed. release and puts additional pressure on testers as test cases This means we are very close to a future in which it will be generated earlier or previous automation work no longer possible to start analyzing log files (using predictive/model remains relevant. building capabilities) and gaining a perspective on a piece In addition to the above challenge, 48% of our respondents, of code even before it is ready to be tested (i.e., while the pointed to issues with test data and test environment developer is still working on it). Such automation will help availability and stability, while 46% said they lacked skilled and identify issues in the way the code is being developed and experienced test automation resources. These are issues that would even suggest course corrections to the developer, have existed for some time now and are likely to become even so they can avoid potential pitfalls that might arise. In other more problematic as an increasing number of organizations words, it would build quality into the product itself so that adopt automation. Fortunately, there is a plethora of tools less time and effort are spent on testing. By allowing testers and solutions which could help ease both of these challenges. to work in parallel with the development process and make
29 61% 48% 46% 42% 42% 42% 41% 41% 40% 40% 39% 39% 38% 37% 38% 36% 35% 34% 32% 29% 25% 24% We find it We find it Challenges Lack of We find it We don't find it We have Chal- We don't We have Current We started difficult to difficult to with the test skilled and difficult in have the difficult to poorly lenges have the too many auto- too late integrating automate data and experienced automating right integrate defined with right different mation with testing the because our environment test because we automation test project service automation auto- solution and test different applications availability automation use multiple tools automation require- virtual- testing mation does not automation automation change too resources develop - into a ments ization process/ tools support tools much with ment DevOps prohibiting method mobile every release lifecycles process us from testing deciding on the right test scenarios Current Trends in Quality Assurance & Testing 29 Main challenges in achieving desired level of test automation Figure 16 2017 2016 2018 61% 48% 46% 42% 42% 42% 41% 41% 40% **New statement** 40% 39% 39% 38% 38% 37% 36% 35% 34% 32% 29% 25% 24% We find it We find it Challenges Lack of We find it We don't find it We have Chal- We don't We have Current We started difficult to difficult to with the test skilled and difficult in have the difficult to poorly lenges have the too many auto- too late integrating data and automate experienced automating right integrate defined with right different mation with testing the environment because our test because we automation test project service automation auto- solution and test different applications availability automation use multiple tools automation require- virtual- testing mation does not automation automation change too resources develop - into a ments ization process/ tools support tools much with ment DevOps prohibiting method mobile every release lifecycles process us from testing deciding on the right test scenarios Notwithstanding all these challenges however, our WQR 2018 survey also reveals that there is a significant percentage of organizations that have started reaping the benefits of automation. Delivering on its promises As organizations gain maturity in the way they handle the 2018, 57% in 2017, and 40% in 2016). One can see an almost new IT paradigms, the benefits they gain from automation linear increase in the numbers for every benefit mentioned also increase. This is corroborated by our survey data. For in the survey. In other words, automation reduced costs and instance, when asked about the benefits they had derived time to market, allowed for better detection of defects, and from test automation, 2018 respondents gave the highest allowed for a better risk coverage. This maps perfectly to the 2017 2016 2018 weighting to “better test coverage” (68% in 2018, up from top objectives of QA and testing, namely “to reduce software 51% in 2017 and 40% in 2016), followed by “better control defects before go-live” and “to contribute to business growth and transparency of test activities,” (66% in 2018, 43% in 2017 and business outcomes.” and 38% in 2016) and “better reuse of test cases” (65% in
30 2018 –19 World Quality Report 30 17% Benefits realized through test automation Figure 17 2018 benefit 2017 benefit 2016 benefit 2016 2017 2018 – value % – value % – value % 68% Better test coverage 19% 19% 18% 51% 40% Better control and 66% transparency of 20% 19% 18% 43% test activities 38% 65% Better reuse 22% 19% 18% 57% of test cases 40% 64% Reduction of 54% 20% 20% 16% test cycle time 39% 64% Better detection 21% 19% 18% 60% of defects 42% 61% Reduction of 18% 19% 17% 53% test costs 39% benefits promised by automation could be due to a number of The highest-rated benefit of “better test coverage” also reasons. For instance, the top benefit that organizations were enables optimization of testing as organizations can control unable to realize with automation was the “reduction in test the amount of coverage and risk they are comfortable with. costs.” This could be due to the initial up-front investments All the benefits quoted are also quality-related and consistent that are required when setting up automation or it might be with the dictum of “quality at speed.” Despite these benefits, due to the costs of retraining or upskilling resources. we should note that a significant proportion of organizations still struggle to see the benefits of automation. According to While one can give multiple reasons for companies not being our survey, the sectors that are leading in terms of accessing able to convert automations promises into reality, most of these benefits are Telecommunications, Automotive, High- these reasons could possibly be grouped under the single Tech, and Transportation. heading of a lack of maturity in handling agile and DevOps and When asked about the (expected) benefits they had failed integrating them with automation. Given this fact, we believe to realize from automation, 39% of respondents pointed to that as organizations gain in expertise and maturity, they will a “reduction in test costs,” 36% indicated “better detection be able to leverage automation much more successfully in of defects,” and another 36% said they had not been able to the years to come. achieve a “reduction of test cycle time.” This failure to reap the Summary over the next few years. To be successful, organizations must Automation is set to grow smarter and become much more understand that automation is not only about replacing effective over the next two to three years, especially as manual testing and chasing some incremental cost savings. the convergence of technologies such as analytics, AI, and Instead, a focus on delivering quality at speed and supporting ML bring about a paradigm shift in the benefits that it can frameworks such as agile and DevOps to deliver much greater deliver. However, getting to that stage requires a lot of up- results and take QA and testing to the next level. front investment that organizations must be willing to make
31 Current Trends in Quality Assurance & Testing 31 The quality assurance organization Greater adoption, newer challenges and a move away from test centers of excellence Frederik Scheja Mandar Salunkhe Gitte Ottosen Senior Director, Financial Services, Test Architect, Principal Consultant, Digital Assurance and Capgemini Sogeti Sweden Testing, Capgemini Sogeti Denmark Deepika Mamnani Andrew Fullen Senior Director, Financial Services, Capgemini Solutions Director, Sogeti UK Driven by digitalization, increased competitive pressures, and still sticking to waterfall for other processes (typically back- the primacy of objectives such as time to market, the adoption end processes based on older, legacy systems). of both agile and DevOps has been growing over the past The increasing engagement with agile and DevOps has also few years. As an increasing number of organizations adopt led to a better and more wide-spread appreciation of the these frameworks, newer challenges, which were perhaps challenges associated with adopting these frameworks. One not so apparent earlier, have come into focus. At the same of the biggest emerging challenges is related to the skill sets time, these frameworks have also led to a move away from these new frameworks require. There are also significant Test Centers of Excellence (TCoEs), with most companies challenges involving test environments, data and tooling. decentralizing testing to the project level. These trends are not uniform and there is wide variation in Agile and DevOps: Continuing adoption and terms of both adoption and maturity across geographies varying levels of maturity and sectors. Our survey reveals that organizations across the Thirty percent of our respondents said that implementing world are customizing agile in different ways and combining agile and DevOps was an important aspect of their overall it with waterfall, to find an approach that best fits their needs IT strategy. Survey results also reveal that the adoption and capabilities. This kind of a hybrid approach often develops of DevOps has increased over the last few years. In 2015, naturally as a pure play agile framework does not work in 82% of our respondents said that they were using DevOps many scenarios due to its organizational, regulatory, cultural, principles in their organization. By 2017, this figure had and business requirements. For this reason, we see many grown to 88% and, according to this year’s survey, a full 99% organizations adopting agile and DevOps for a few processes of respondents said that they were using DevOps principles (typically the customer-facing, front-end applications) while in their organization. Proportion of projects using DevOps principles Figure 18 2015 2018 2017 2016 47% 42% 32% 32% 30% 30% 26% 21% 18% 17% 16% 15% 11% 12% 11% 12% 8% 9% 7% 3% 1% Fewer than 20% 20–50% of our 90–95% of our 70–90% of our 50–70% of our Not applicable projects use of our projects projects use projects use as we do not projects use DevOps use DevOps DevOps DevOps use DevOps DevOps principles principles principles principles principles principles
32 2018 –19 World Quality Report 32 without a test professional.” At the same time, 39% also What many organizations call agile is, in fact, a combination reported that “test activities were performed in a distributed of these frameworks and waterfall. This also came through team” and another 33% said that “test activities are mostly in this year’s survey. For instance, when asked about how performed by test professionals.” This almost bi-modal test activities were performed in their organization, 45% of distribution (between agile and non-agile ways of organizing respondents said that “test activities were performed by all testing responsibilities), shows that most organizations are team members, supported by a test professional,” while 43% opting for the hybrid model mentioned earlier. said that “test activities were performed by all team members, Testing methods adopted when agile is in use Figure 19 2018 2018 45% 45% 43% 43% 39% 39% 33% 33% Test activities Test activities Test activities are Test activities Test activities Test activities Test activities are Test activities are mostly are performed performed by all are performed are mostly are performed performed by all are performed performed by in a distributed team members, by all team performed by in a distributed team members, by all team test professionals team c without a speci fi members, supported test professionals team c fi without a speci members, supported test professional by a test professional test professional by a test professional marginalization of testing and operations, with most project This type of hybrid model is mostly adopted by companies teams being dominated by developers. This often results in that have a significant legacy component in their IT systems greater emphasis on speed over accuracy, which can defeat and it ends up complicating operations and raising the effort the very purpose of testing. Such firms will need to strike a and time spent on testing. According to the WQR survey, balance between these two often competing objectives, in respondents spent an average 25% of their overall project order to derive the full benefits of these approaches. effort on testing in an agile project. On the other hand, in the more mature regions, we see an It’s important to keep in mind, that there is considerable increasing number of companies shifting from waterfall to variation across regions when it comes to these trends. We agile and DevOps. These regions are marked by a greater see more of these hybrid models and waterfall methods adoption of continuous testing and integration testing and persisting in regions that are further behind on the maturity a lot of excitement and some early adoption of artificial curve. In these regions, the rise of DevOps and the importance intelligence (AI) and machine learning (ML) technologies. of business goals such as time-to-market has also led a DevOps – driving change using and 38% planning to use), “cloud-based development As already stated, only one percent of respondents indicated and test environments” (43% using and 40% planning to use), that they were not experimenting with or applying DevOps and the “continuous monitoring of apps in production” (41% in any form. According to the survey, the top DevOps using and 40% planning to use). processes being followed were “breaking down large efforts into smaller batches of work” (44% of respondents currently
33 Current Trends in Quality Assurance & Testing 33 Enterprise-wide adoption of certain DevOps practices Figure 20 Not using today but planning to use Currently using No plans/don't know We just break down large 44% 18% 38% orts into smaller batches of work ff e We use more cloud-based 43% 17% 40% dev and test environments We use continuous 40% 41% 19% monitoring of apps in production We automate the delivery 20% 40% 40% pipeline (automated deployment) with integrated test cases We use virtualization 20% 42% 38% of dev and test environments The increased adoption of DevOps has led to a premium being they used analytics from operations to determine or optimize test coverage. Of course, this use of analytics is an emerging put on the need for speed. Consequently, there is increased trend and we need to observe its impact on the overall testing use of both automation and engineering platforms, as well as approach, budget, and skillsets. Some of the new, emerging analytics. Over the past year, we have also seen an increase roles have been discussed in our AI chapter. in the number of discussions around the usage of predictive analytics to drive optimization prior to automating test Another new trend driven by the same considerations is the sets. For instance, when asked about the special approach usage of bots for automated testing. Our survey respondents they took to speed up and optimize testing in agile/DevOps stated that they were using or considering using bots for environments, 54% of respondents said that they were using lifecycle test automation (49%), set-up of test environments analytics on all available project data to optimize their test (48%), test data generation (46%), and integration of test sets. Forty-five percent of the respondents also stated that types as part of the DevOps pipeline (45%). Interest in and likelihood of using bots for test activities Figure 21 48% 49% 45% 46% 29% 31% 23% 31% 24% 31% 22% 21% We are using or considering We are using or considering We are using or considering We are using or considering for integrate to using bots to usingbots for using bots using bots automated test lifecycle test set up data our test generation automation types as part of DevOps environments pipeline No plans/don't know Not using today but planning to use Currently using
34 2018 –19 World Quality Report 34 Challenges – the dawn of realization Forty-two percent of respondents also reported a lack of Testing continues to be a challenge, with only 6% of professional test expertise in agile teams. Agile, DevOps, respondents stating that they had no difficulties in testing automation and artificial intelligence not only require newer with agile. When asked about what difficulties they did face, skill sets, but also make it necessary for QA professionals to 53% reported the lack of an appropriate test environment have multiple technical competencies. This is a significant issue and data (versus 46% in 2017), 50% reported an inability to from a staffing perspective since this mix of technical testing apply test automation at the appropriate levels (as opposed skills is not readily available in the market. As the skillset is to 41% in 2017) and 48% pointed to difficulties in slicing test moving from functional to SDET (Software Development activities for more than one location for distributed Agile Engineer in Test), organizations are faced with challenges of (as opposed to 42% in 2017). These results give us a sense reskilling the existing testing teams and attracting the right that increased adoption has led to greater realization of how testing talent to build future-ready testing teams. challenging these issues really are. Challenges currently faced in applying testing to agile developments Figure 22 2018 2017 2016 53% 50% 48% 46% 44% 45% 44% 44% 44% 43% 43% 43% 43% 43% 42% 42% 41% 41% 41% 40% 39% 39% 38% 6% 1% 1% Difficulty in No real Difficulty to Lack of a Lack of Difficulty in Early Inability to Lack of slicing test difficulties re-use and good testing professional identifying involvement apply test appropriate activities for with testing repeat tests approach test the right of testing automation test more than in agile across that fits with expertise in areas on team in at appropri- environment one location sprints/ the agile agile teams which test inception ate levels and data for distribut- iterations development should phase or ed agile method focus sprint planning 28% said they needed more predictive analytics skills. To In our survey, 30% of respondents stated that they needed tackle this challenge, an increasing number of organizations more functional automation expertise, 29% said they lacked are forced to adopt workforce transformation programs and test environment, virtualization, test-driven development aggressive hiring of these niche skills. (TDD), and business-driven development (BDD) skills, while
35 Current Trends in Quality Assurance & Testing 35 The extent to which agile and DevOps adoption changes the skills expected of QA and testing professionals Figure 23 Total 2018 Total 2017 45% 34% 21% Functional test automation expertise 30% 18% 52% 30% 45% 25% Test environment and virtualization expertise 29% 52% 19% 28% 46% 26% TDD (test-driven development) or BDD 29% 52% 19% (behaviour-driven development) 32% 45% 23% 52% 28% 20% Predictive analysis skills 34% 43% 23% 27% 51% 22% Development and coding skills 27% 53% 20% Non-functional testing skills (performance, security) 27% 53% 20% 26% 48% 26% 26% 52% 22% Software development engineer testing skills (SDET) 32% 42% 26% 26% 54% 20% Test strategy and test design skills 32% 42% 26% 20% 54% 26% Test data set-up expertise 44% 22% 34% Understanding of business processes 25% 53% 22% 33% 45% 22% 25% 56% 19% Production quality monitoring skills Skills are lacking andrequired more Skills are OK – no change needed Skills are less relevant Test Centers of Excellence: Nothing permanent but change or a Test Excellence Center, most are moving away from it. About four to five years ago, Test Centers of Excellence According to our survey, when asked about the percentage (TCoEs) were all the rage. They offered economies of scale of testing roles they had organized in a centralized QA and helped standardize tools, test environments, and organization, 21% of respondents indicated they had processes. However, the excitement about TCoEs has ebbed centralized test automation roles, 20% had centralized considerably over the last two years. The rise of agile and Software Development Engineer Tester roles, 19% had DevOps and the importance paid to speed and flexibility, centralized non-functional testing roles, another 19% had have led to most organizations placing testing responsibilities centralized business domain-based tester roles, and 18% had in individual projects. centralized specialized technology tester roles. While some organizations still retain a thin layer of centralization in the form of either a community of practice Summary These trends are likely to continue with the increasing In summary, we see a very mixed picture when it comes maturity of agile and DevOps practices and a greater focus to agile, DevOps and TCoEs. On the one hand, there is an on scaling these practices to the entire organization. We are overall increase in adoption of agile and DevOps along with also likely to see a greater adoption of cloud, automation, a growing realization of the kinds of challenges that need to analytics and artificial intelligence all driven by the adoption be tackled to derive the full benefits of these frameworks. We of DevOps. The testing skillset has undergone radical change also see variation in terms of maturity across regions as well in the past years and will continue to morph to cater to the as across sectors. When it comes to TCoEs, the trends are a increased technical testing needs. continuation of those seen last year, with more organizations shifting away from such centralized organization of testing.
36 2018–19 World Quality Report 36 Test data and environments management The biggest obstacles to progress in QA and testing Shivakumar Balasubramaniyan Eva Holmqvist Senior Test Specialist, Sogeti Sverige Vice President, Financial Services, Capgemini With multiple releases and shorter test cycles, the volume Change is coming thick and fast in the IT industry. The coming of work has increased, while rapid technological change has together of frameworks such us agile and DevOps, with rapidly forced the industry to grapple with the challenges of handling evolving technologies such as analytics, automation, cloud, more complex and newer types of data and environments. the internet of things (IoT), and artificial intelligence (AI), has Our 2018 WQR survey reveals the ways in which these transformed the quality assurance (QA) and testing function. developments have impacted the handling of test data and On top of that, new regulations, such as the GDPR and IFRS 9, environments. add to the complexity and uncertainty of this period of rapid change. Nowhere is the impact of these changes felt more than in test data and test environment management. Test data and environments today The industry uses both permanent and virtual test environments are permanent, and in the Healthcare and environments, which can be either cloud or non-cloud Lifesciences industries the corresponding figure is similar, at based. According to our 2018 survey, an average 69% of 34%. On the other hand, a mere 26% of all test environments all test environments used today are non-permanent test in the High-Tech sector and 28% in the transportation environments. This means that the reliance on traditional sector are permanent. The persistence of permanent test permanent test environments continues, with an average environments in the Public sector as well as the Healthcare, 31% of test environments being permanent. Respondents and Life Sciences sectors can be explained by the fact that also indicated that, on average, 19% of their tests occurred in these sectors have historically delivered services that were “cloud-based temporary test environments,” 17% occurred in critical for the safety and well-being of the public and had “virtualized test environments,” 17% occurred in “non-cloud- therefore to deal with masses of data that needed to be based temporary test environments,” and 16% occurred in secure. Due to security concerns as well as the criticality of “containerized (docker or similar)” test environments. the services they provide, these sectors have been slow to It’s important to note the considerable variation that exists embrace cloud and virtualization technologies, thus leading across different sectors when it comes to test environments. to a greater dependence on permanent test environments. For instance, in the Public sector, a full 35% of all test Indicative proportion of automation of each of the following activities Figure 24 2017 2018 18% 16% 16% 16% 16% 16% 16% 16% 15% 15% 15% 15% 14% **New statement** % of perfor- % of API % of end-to-end % of functional % of functional % of security % of test mance test test cases business test cases that test cases that tests that are data that is cases that are that are scenarios that are generated are executed executed with generated by executed with automated are executed with test with test test automation test data test automation with test generation tools automation tools tools tools automation tools tools
37 Current Trends in Quality Assurance & Testing 37 environments. All three types of testing have grown over An encouraging trend seen this year is the excitement around time. Performance testing in cloud environments grew from containerized test environments. Though our survey results 44% of respondents in 2016, to 56% in 2017, to 58% in 2018. show very little change since last year and only 16% of all Additionally, functional testing of cloud services grew from test environments are being containerized today, we expect 50% in 2016, to 53% in 2017, to 58% in 2018, while security to see the adoption of containerized test environments rise testing grew from 43% in 2016, to 55% in 2017, before over the coming years. This indicates a support for agile, dipping slightly to 53% in 2018. This reflects the increasing DevOps, and automation and is a sign of increasing QA focus on cybersecurity practices across organizations today. maturity since automated testing done in a permanent test environment previously used for manual testing usually When it comes to test data, 66% of our respondents said that generates false positives which can increase test times. Using they used spread sheets to manually generate new test data dockers allows organizations to set up specific environments for multiple iterations of testing. This is a three-percentage with the exact data they need and reduces false positives so point increase from 2017. Another 62% of respondents said that real defects can be detected faster. Thus, the possibility that they copied production data which they anonymized of greater efficiencies and cost reduction is turning out to be before testing, an increase of 19 percentage points since a key driver for dockers or containerized test environments 2017. Clearly, the maturity in test data provisioning is still and we expect even greater use of such environments in not changing in enterprises. Both trends can also partly be the future. explained by the increased number of test runs, which leave test teams with very little time to provision and generate This year’s survey also shows that performance testing, test data. This is just one of the many challenges facing the functional testing of cloud services, and security testing industry today. were the most popular forms of testing done in the cloud The ever-increasing challenges around data and environments Another aspect of this problem is that the industry is having The adoption of agile and DevOps has led to shorter to deal with newer types of data and is grappling with how to delivery cycles and more frequent releases for which data standardize the data. Take, for example, the rise of IoT and the and environments need to be provisioned. This pressure on growth in the number of connected homes, connected cars, QA departments is further compounded by the increased and connected health devices. How does one standardize the complexity of test data and environments today. In addition, data from different connected systems? Or, in general, how dealing with digital, front-end applications means dealing with does one provision test data when different teams request it high-volume, high-velocity data that can be hard to create in different formats without increasing test cycle times? from scratch and, from a data perspective, most organizations in this space are still maturing. While the application of new Both application modernization as well as the move from AI and ML-based technologies can help solve some of these legacy to cloud and distributed systems, create the need for issues, such applications typically have a voracious need for newer types of data. Similarly, when you talk of AI adoption, data and a key pre-requisite that the data be fresh. take for example connected cars or driverless vehicles, you are essentially talking about a way of doing things for which From an execution perspective, automation too is a huge no precedent or use cases exist. How can we create the data challenge. An important reason for this is that DevOps tool or the environments today to test such use cases? The answer chain automation still hasn’t become pervasive. Therefore, is that such data models or environments simply do not exist emerging software tools, including robotic process and, in such cases, companies might need to look at types of automation (RPA) in testing, have great difficulty reaching data that have never been dealt with before. scale as many of them are still relying on clunky, high- investment models, such as sub-setting, or solving their data There is no unified view today or standards for how to delivery plumbing problems (for things like synchronicity, consume such data, how to set up such environments, or distributed referential integrity) by writing more logic.
38 World Quality Report 2018 –19 38 Each of these issues is reflected in our survey results. For how to deal with the security issues that are a big concern instance, when asked about the challenges faced in managing for connected devices. Moving forward, we are likely to see test data, respondents gave the highest weighting to an exponential increase in the number of connected devices “maintaining test data consistency across different systems and the industry will need to come up with creative solutions under test.” Sixty-one percent of respondents said that this to manage both test data and test environments for testing was a problem, followed by 56% who pointed to “managing such devices and networks. the size of test data sets,” and 55% who said that “creating and This situation is further compounded by regulatory maintaining test data which are not copies from production requirements such as the GDPR and IFRS 9. Since many of data” was an issue. Each of these challenges is related to the these laws have a data protection element, they lead to problems of volume, complexity and lack of standardization considerable uncertainty about how to handle test data, what mentioned earlier. masking techniques to use to protect customer information, or whether to use production data at all. Challenges faced with managing test data Figure 25 2013 2014 2016 2017 2018 2015 65% **New statement** 61 % 56% 56% 56% 55% 53% 52% 51% 50% 50% 50% 48% 48% 47% 47% 47% 47% 47% 46% 46% 46% 44% 44% 44% 43% 43% 41% 41% 40% 38% 37% 37% 35% 32% 32% 29% 29% Having to Lack of test Creating Creating and Managing Finding the Maintaining Complying maintain the data for and retaining maintaining test the size relevant test data with data right test data complex useful data that of test data test data in consistency security and set versions with integration copies from are not copies sets large across data privacy di ff erent testing across production production from test data di ff erent regulations test ve systems data data sets systems for test data rsions and organizations under test
39 Current Trends in Quality Assurance & Testing 39 of test environment” (54%), followed by “defects due to Interestingly, most of these challenges seem to have grown inaccurate configuration of test environments” (54%), and over the years, perhaps due to the increasing complexities “lack of facilities to book and manage your own environments” associated with digital transformation and the adoption of (38%). Interestingly, the percentage of respondents quoting agile and DevOps. “lack of facilities to book and manage your own environments,” “having to maintain multiple versions of test environments,” Unlike test data, there seems to have been some improvement “lack of visibility to test environment availability,” and “lack of over the years when it comes to test environments. For availability of the right test environment at the right time” as instance, when asked about their test environment-related challenges have fallen over time. challenges, respondents gave the highest weighting to “cost Challenges encountered with test environments Figure 26 2016 2018 2017 54% 54% 48% 46% 47% 46% 46% 41% 39% 41% 40% 38% 38% 37% 37% 34% 31% **New statement** Lack of Lack of Lack of Lack of the Having to Defects due to Cost of maintain visibility into visibility facilities test inaccurate availability of right environment test environment of what can to book and multiple versions configuration test environment manage or cannot of test environ- of test environ- availability (servers, storage, networks, working (utilization and ments be tested in ments your own stations, databases, incomplete environments demands) test environments etc.) at the right time Summary challenges facing the industry. The rise of the API economy All the changes that have taken place in the last few years also holds promise because it will allow testing to be carried seem to have had a compounded impact on test data and test out at an API level rather than requiring UI testing all the time. environments. Moving forward, we see trends such as IoT, The increase in adoption of service virtualization, along with machine learning and blockchain creating fresh challenges hardware virtualization, will bring down test environment for test data and test environment management. However, costs especially during system and incremental system not everything is doom and gloom, as we can also see integration testing. notable examples of the industry adjusting and responding to these challenges. Trends such as the growing utilization One thing is clear – no matter what challenges or solutions of containerized test environments, the use of Bots for zero- come up in the future, test data and test environments are touch automated testing, the creation of tools for solutions the two areas which the industry absolutely needs to get for better test data sampling, as well as initiatives, such as right if it wants to move forward on every other front. open data projects launched by governments across the world, are all positive developments that will help tackle the
40 2018 –19 World Quality Report 40 Efficiency and cost containment in quality assurance QA and testing costs showing signs of having stabilized, though a wave of investments in new technologies is expected Sathish Natarajan Maheshwar Kanitkar Vice President - Digital Assurance & Testing, Sogeti, Senior Director, Sogeti Testing Capgemini Group Kumar Balasubramaniam Senior Director, Financial Services, Capgemini ecosystems, the greater leveraging of cloud, the inclusion Information technology (IT) has gone through such a rapid period of change that it is almost unrecognizable of test data management (TDM) and test environment when compared to what it was 10 years ago. This is due to management (TEM), and the use of automation and analytics both the dizzying pace of technological evolution and the in IT started gaining real traction. These changes led to a wave emergence of new process frameworks, operating models, of investments in new infrastructure and tools as well as in and organizational structures over the last few years. When re-organization, restructuring, and reskilling initiatives. All of we look at the budgets for quality assurance (QA) and testing, this led to the spike in QA and testing budgets that we saw in we can see a reflection of all these changes. 2015 (35%) and 2016 (31%). Our yearly WQR surveys capture the fluctuations in QA and Since then, as organizations have gained experience and testing spends over time. For instance, according to this maturity in handling these new frameworks and technologies, year’s survey, the proportion of the IT budget spent on QA they have started reaping the benefits of these changes. A and testing is pegged at 26%. This is the same as last year, number of testing activities have gained efficiency and this though considerably below the highs of 31% in 2016 and 35% has driven down costs. This is reflected in the fall in the in 2015. Before that, QA and testing budgets accounted for proportion of IT budgets devoted to testing that we have 26% in 2014 and 23% in 2013. seen over the last two years. This pattern reflects the changes of the last three to five years, This, however, is just the picture at the aggregate level; it when trends such as digital transformation, the move from hides a lot of complexity underneath. waterfall to agile and DevOps and from application to product Proportion of total IT budget allocated to QA and testing (including testing processes, tools, and resource costs) Figure 27 35% 31% Mean percentage 26% 26% 26% 23% 2018 2013 2014 2015 2017 2016
41 Current Trends in Quality Assurance & Testing 41 QA and testing budgets: what’s beneath the numbers? the overall budgetary trends, which point to a fall in the According to our 2018 survey, when respondents were asked percentage of IT budget being spent on QA and testing (from whether they had seen an increase in the proportional effort 35% in 2015 to 26% today). Each of these numbers come and cost spending on QA and testing over the last four to five from our respondents, so why is there so much confusion? years, a whopping 72% said “yes.” This directly contradicts Experience of increase in proportional effort and cost spent on QA and testing over the last 4~5 years Figure 28 Summary of Yes answers 72% 50% 43% 2016 2018 2017 user-interfacing and front-office systems). We are seeing a lot Three major factors are clouding the picture and creating all of efficiencies and reduction in QA and testing costs when it this confusion when it comes to QA and testing budgets. comes to the legacy side today because there is a lot of focus The first major factor is that, in absolute terms, both effort on cutting costs, optimizing processes, etc. However, for the and spending on QA and testing has indeed been rising. Today, front-office systems, which are usually digital applications, there is a huge focus on the virtualization of test environments, the main driver is speed. As we have seen, huge investments test data management, test automation, and the use of are being made in this area and these are driving up costs. analytics across the testing lifecycle. Many organizations are These two opposing, juxtaposed trends, could also be leading also exploring the use of robotics process automation (RPA) to this confusion about QA and testing budgetary trends. and artificial intelligence (AI) in testing. This has led to a period of intense investments, similar to that seen two to three The third and final factor is probably the biggest of them all. years earlier. Additionally, the amount of effort that goes into This is the difficulty in accurately capturing testing spends testing today has drastically increased due to the addition of due to the coming of age of agile and DevOps. Before agile new capabilities, new systems, and a vastly expanded scope and DevOps, testing often operated as a separate profit or for IT that was not there earlier. This increase in effort put cost center, with operations centralized in a Test Center of into testing is further heightened by the increased number of Excellence (TCoE). This made it easier to measure spends and release cycles mandated by businesses today. track how much was being spent on what. However, agile and DevOps have made this kind of tracking difficult since testing Thus, both the spending and the effort that goes into is now integrated into the project or the Scrum teams. This testing have increased over the last few years. However, it is makes it extremely difficult to track exactly how much time is important to note that IT budgets have also been increasing spent on testing activities, especially with the now-prevalent with the adoption of these technologies. Therefore, it is Software-Developer-Engineer-in-Testing (SDET) profile (who entirely possible that, while QA and test budgets and efforts engages in development, analysis, and testing activities). It is have been rising in absolute terms, in relative terms they have entirely possible, for instance, that the efforts of these SDETs, stayed the same or even fallen. This is the first factor that or of entire Scrum teams, is being tagged to the development could be causing confusion. or the testing budget or allocated on the basis of a thumb-rule The second factor is that a closer examination of costs, percentage between these two budgets. budgets, and efficiencies reveals two different and While all three factors are important, it is this last point contradictory trends at play. This is because today, there are that is probably the most important when it comes to two parts to every organization’s IT infrastructure. There are explaining some of the apparent contradictions in our the systems of records (typically, the legacy and back-end respondents’ answers. systems) and there are the systems of engagement (mostly
42 2018 –19 World Quality Report 42 What’s driving testing spend? over time from 33% of the budget in 2015, to 31% in 2016, When asked about the individual components making up their 21% in 2017, and 26% today. This reflects a reduction in the QA and testing budget, respondents said that they allocated amount of manual testing since 2016. At the same time, the an average of 44% to hardware and infrastructure, 31% to rise over the last year is probably due to the increase in the tools, and 26% to human resources. number of test cycles and the overall increase in the amount of testing being done today. We can see that the human resources component has fallen Factors having an impact on increase in QA and testing budget Figure 29 52% 41% 33% 36% 30% 29% 29% 31% 31% 31% 27% 29% 27% 24% 25% 24% 23% 22% 21% 20% 19% 18% 16% Increased We detect more Increased Shift to Business Increased Increased Increased defects, which complexity need for inefficiency agile/DevOps amount of challenges demands with test co-location causing leads to higher IT of IT of test developments and releases activities environments quality applications more/longer test more test iteration cycles cycles 2016 2017 2018 and test budgets, respondents gave the highest weighting On the other hand, the percentage of the QA budget spent on tools or software licenses seems to have remained more or to “business demands higher IT quality” (27%), followed by less constant over the last three years. Thus, the proportion “increased inefficiency of test activities” (25%), and “increased of the QA budget spent on tools was 30% in 2015, 29% in complexity of IT applications” (24%). It is interesting to note 2016, 33% in 2017, and 31% today. that the importance of the top-two factors quoted above has actually been falling over the last two years. For instance, the According to our respondents, it is the expenditure on “business demands higher IT quality” came down from 33% in hardware and infrastructure that has gone up over time. 2016, to 29% in 2017, to 27% in this year’s survey. The percentage of the QA and testing budget allocated to hardware and infrastructure was reported to be 37% in 2015, Similarly, the “increased inefficiency of test activities” has also 40% in 2016, 46% in 2017, and 44% in 2018. This reported rise in the percentage spent on hardware and infrastructure is come down from 29% in 2016, and 31% in 2017, to 25% today. surprising, given the fact that cloud adoption has been rising This perception of increasing efficiency is perhaps a result of recently. Given this cloud adoption, we expect the share of greater maturity on the part of organizations in handling the infrastructure cost in the total QA and testing budget to new frameworks and technologies and realizing the benefits come down significantly in the future. of automation. In other words, it seems that organizations have become better at deriving the benefits promised by According to our 2018 survey, when asked about the these technologies and frameworks. individual factors that had an impact on the increase in QA
43 Current Trends in Quality Assurance & Testing 43 Proportion of QA and testing budget allocated to hardware, infrastructure, tools, and human resources Figure 30 46% 44% 40% 33% 40% 32% 35% 37% 31% 33% 33% 31% 30% 29% 23% 26% 28% 21% Tools Human Hardware and infrastructure (software licences) Resources 2016 2015 2013 2014 2017 2018 In addition, expert opinion holds that the increased number of test cycles brought about by the shift to agile and DevOps is perhaps one of the biggest reasons for a rise in testing effort and expenditures. Summary QA and testing budgets have been through a period of intense three key areas over the next couple of years. First, work on investments, followed by a period of benefits’ realization, creating successful use cases (in testing) for new technologies resulting in increased testing efficiency. Organizations are such as AI, machine learning (ML), or robotics process seeing a stabilization of spends related to agile and DevOps automation. Second, create detailed and elaborate tracking adoption, which is becoming increasingly mainstream today. mechanisms to understand exactly how much cost and effort However, over the next two to three years, we are likely to is going into testing in Agile or DevOps teams. It would be see increased investments in testing lifecycle automation, AI, impossible to reduce costs without understanding clearly RPA, and analytics in testing. As with every other technology, how much is being spent and where. Finally, there is one step this is likely to increase QA and testing budgets in the short- that organizations can immediately take to improve testing term before the expected efficiencies start to kick in and drive efficiencies, that is the use of end-to-end automation in costs down. According to our respondents, the percentage testing. While investments are being made, they are nowhere of the IT budget dedicated to QA and testing, will rise to an near the optimal levels. All three of these steps will go a long average of 33% within three years. way in improving testing efficiency and the quality of their IT To gain the maximum benefit from their QA and testing systems in the long term. spends, we would recommend that organizations focus on
44 Sector Analysis
45 Automotive 46 49 Consumer Products, Retail, and Distribution 51 Energy and Utilities Financial Services 53 Healthcare and Life 56 Sciences 58 High-Tech Government and 60 Public Sector 63 Telecom, Media and Entertainment
46 2018 –19 World Quality Report 46 Automotive Software assurance vital as Automotive sector shifts to mobility services Michael T. Hessler Vice President, North America Automotive Lead, Capgemini the acquisition of tools required for this. This complexity The Automotive sector is changing, and it’s not an is a huge challenge, especially as the global automobile evolutionary change but a revolutionary one. Over the companies still have a significant legacy component in their years, the amount of software in a typical car has been IT systems. While the newer, digital applications are usually rising steadily and today, original equipment manufacturers on the cloud, their core manufacturing, core engineering, (OEMs) are investing heavily in next-generation vehicles and and core finance applications are still on legacy systems. The drivetrains. They are gearing up for market disruptions challenge posed by this is further complicated by the need for caused by autonomous vehicles, ubiquitous connectivity, new skill sets to handle platform migrations, agile processes, vehicle electrification, and the declining stocks of fossil fuels. and automation efforts. These trends mean that OEMs will need to reorient their organizations (and supply chains) from a focus on mechanical, electrical, or product engineering to a focus on software and, QA and testing budgets and what’s driving them in terms of revenue streams, from a focus on the driver to a According to this year’s survey, the percentage of IT budgets focus on the passenger. spent on QA and testing in the Automotive sector is 29% as This requires a change in the processes, structure and against a corresponding figure of 26% across all sectors. In most importantly, the mindset of organizations. Given particular, 21% of the respondents in the Automotive sector the emergence of competitors such as Tesla and Uber, (as against 10% across all sectors) said that their QA and automobile companies and their suppliers will increasingly testing spends were between 40–50% of their IT budget. need to think and behave like software companies. This is This wide variation can be explained by the huge investments likely to be an unwieldy and complicated transformation, as being made in software for connected car projects and the almost every piece of hardware going into an automobile need to test them. This has also led to a rise in the proportion today has some software on it, thus giving rise to the need of the IT budget spent on QA and testing. In our survey, 72% to maintain updated versions across all platforms, for all of respondents said that they had observed an increase in the vehicles. In addition, the inevitability of driverless vehicles proportional effort and cost spending in QA and test activities means that new revenue streams and models will have to over the last four to five years. come up which put the passenger rather than the driver at Spending on QA and test activities can also be linked directly to the center of everything. The car will change from being an IT and testing objectives. For instance, when asked about the asset that someone buys or leases to becoming a platform objectives of their IT strategy, 50% of respondents from the for mobility that can be owned, or leased, or rented by the Automotive sector wanted to “enhance security,” followed minute. OEMs will need to figure out what services they can by 44% who wanted to “enhance customer experience,” provide to drive this change in consumer behavior and create and 39% who wanted “higher responsiveness to business revenue streams. They have already been collecting vehicle demands.” Similarly, when asked about the objectives of data for a number of years, which has led to huge data lakes their QA and testing strategy, 43% of respondents from that are being leveraged to generate the required insights to the Automotive sector wanted to “detect software defects help this transformation. before go-live,” followed by 42% who wanted to “increase A related trend is that a number of OEMs are beginning the quality of software/product,” followed by 41% who to reach the customer directly, bypassing their traditional talked about “ensuring end-user satisfaction.” marketing channels. This means a lot of focus on customer These objectives give us a fair idea of the direction in which intimacy and digital applications. the Automotive sector is headed. On the one hand, there is These changes increase the complexity of the ecosystem an increased focus on shifting business demands, customer exponentially. This increased complexity and the need to experience, and end-user satisfaction, all of which require a complete testing within shorter timeframes has led to an greater alignment between business and IT. At the same time, increased focus on test automation, continuous testing, and the results reflect the exponential increase in the importance
47 Sector Analysis 47 ccording to this year’s survey, the A percentage of the IT budget being spent on QA and testing in the automotive sector is 29% as against the corresponding figure of 26% across all sectors. According to our survey, an average 33% of testing in the of digitalization and IoT in the context of connected cars. They Automotive sector occurs in a traditional permanent test also reveal the motivation behind OEMs’ efforts to reach the environment. Approximately 19% occurs in a cloud-based customer directly and increase customer intimacy. temporary test environment, 17% occurs in a temporary non-cloud test environment, and 15% in a virtualized test Digitalization and the challenges of test environment. These percentages show the persistence of environments and test data traditional approaches to testing. Encouragingly, however, More and more automobile companies are releasing the Automotive sector is also using containerized test vehicle-related applications today. Even though the use of environments, with an average 15% of all test environments such applications remains low, a key part of their strategy being of this type. is to achieve greater customer intimacy and is integral to When asked about the challenges in provisioning test becoming a more software-oriented company. This has environments, 58% of Automotive sector respondents (as created additional challenges for the QA and testing function. opposed to 54% across all sectors) quoted “defects due to For instance, when asked about their challenges in testing inaccurate configuration of test environments,” followed by mobile, web, and other types of front-office applications, 53% (54% across all sectors) who blamed the “cost of test 53% of Automotive sector respondents said, “not enough environment,” and 41% (37% across all sectors) who identified time to test,” 36% said that they “don’t have the right tools a “lack of visibility to test environment availability.” Similarly, to test,” and 33% said that they “don’t have the right testing when asked about the challenges in managing test data, 63% process or method.” of Automotive sector respondents (61% across all sectors) said “maintaining test data consistency across different In the first place, the current QA challenges in the sector systems under test,“ 60% (46% across all sectors) said “finding are driven by the focus on faster time-to-market and the the relevant test data in large test data sets,” and 59% consequent adoption of Agile and DevOps, which has led to (51% across all sectors) said a “lack of test data for complex more frequent release cycles and increased the pressure on integration testing across systems and organizations.” QA and testing. In addition, the QA challenges are also related to a lack of the right test tools and test processes within the These results show that this sector experiences more test sector. Finally, a lack of the required skill sets, in particular of data and test environment challenges than others. Things like profiles such as software development engineers in testing the lack of visibility, to test environment stability, and issues (SDETs), test automation experts, and migration architects with test data consistency all point to issues with managing is a key factor holding back the industry from more modern the complexity of test data and environments and issues with approaches to QA and testing. integration of data coming from different sources. Another key challenge is the huge legacy component that is a part of Automotive IT systems. As mentioned earlier, latest Two major trends – IoT and automation applications in the Automotive sector are being built using The rise of the connected car has put a lot of focus on the newer approaches, such as Agile and DevOps. This leads to Internet of Things (IoT) in the Automotive sector. This came issues when these applications need to be integrated back through in our survey results, when respondents were asked with the legacy applications. One of the biggest challenges about their test strategy for testing IoT products. Forty-eight is that of recreating test environments on a mass scale due percent of respondents from the Automotive sector claimed to the complexity and number of permutations/combinations to “have a fairly mature IoT test strategy,” as compared to of the environment and data that needs to be replicated. It is 41% across all sectors. This figure was the second-highest also not always possible to just take copies of production data, across all sectors surveyed. The results are not surprising due to concerns about the privacy and security of customers’ when you consider the number of OEMs that have had personal data.
48 2018 –19 World Quality Report 48 connected vehicles for some time. These OEMs also have a lot At present, there are many challenges holding back the adoption of automation. For instance, 66% of Automotive of data collected from these vehicles and moving forward, it respondents (61% across all sectors) said they found it will be interesting to see how they use these insights to come difficult to automate as their “applications changed with up with new and innovative services. every release,” 62% (42% across all sectors) said that they Another big trend in this sector is the need for test automation. “didn’t have the right automation tools,” and 55% (48% across Even though it is a growing trend, the levels of automation all sectors) said that they had “challenges with test data and environment availability and stability.” for QA and testing remain low. According to our survey, the most popular activities to automate are the generation of Despite these challenges, there is an emerging trend around functional test cases (20% generated using test generation the use of analytics and artificial intelligence (AI) for optimizing tools), execution of security tests (18% executed with test test automation. According to our survey, 44% of Automotive automation tools), and the execution of performance test sector respondents said that they were leveraging analytics cases (18% executed with test automation tools). Thus, the and AI for intelligent automation. An additional 41% said that levels of automation are still quite basic, but we expect these they were leveraging these technologies for “self-learning percentages to increase in the future. cognitive platforms” to help optimize QA activities. Summary The Automotive sector is going through a period of transition that will place great demands on its QA and testing function. The QA and testing in this sector will have to scale up rapidly with greater adoption of automation, agile and DevOps. To make this move successfully, OEMs will have to move up the maturity curve in terms of testing digital capabilities and digital offerings. They also need to aggressively deal with their legacy applications on a priority basis as the challenges around this will become more and more complicated as QA and testing evolves.
49 Sector Analysis 49 Consumer Products, Retail & Distribution Customer-centricity and speed-to-market call for improvement in QA approach Hitesh Naidu Anand Banka Adam L Agnew Cyndi Fulk Lago Associate Vice President, Vice President, Senior Manager, Executive Vice President, CPRD Practice, Capgemini CPRD Practice, Capgemini Capgemini Application Services, Capgemini followed by 47% who said, “enhance customer experience” The Consumer Products, Retail and Distribution (CPRD) (once again, the highest rating across all sectors), and another sector is the most customer-centric sector included in our 47% who said, “higher quality of software solutions.” The survey. Due to this, some of the most important trends in implications are clear – factors such as quality, brand name, CPRD today relate to driving end-user satisfaction, data, and customer experience are extremely important here. and analytics. Additionally, the multi-channel engagement with consumers The CPRD sector was one of the first to be a part of the also increases the importance of security. There have Digital Revolution. Starting with the first, tentative steps been a number of high-profile data leaks over the past few of e-commerce sites and multi-channel engagement, today years, all of which have received widespread condemnation digitalization is impacting almost every part of the CPRD and adversely impacted the brand images and customer sector from the shopping experience, to the supply chain and relationships of the organizations concerned. the distribution network. Brick-and-mortar operations are The importance of security has also been increased by CPRD’s becoming obsolete and a lot of the smaller organizations that move to the cloud. This has created a bit of an issue for many have not embarked on the Digital Transformation journey organizations, as they suddenly have to deal with a number are simply not able to compete and are being pushed out of of very stable but non-business critical applications, which the market. they had not bothered to get upgraded earlier. Moving such Another big trend impacting the CPRD sector is a wave applications to the cloud, calls not just for an upgrade but also of investments in ERP upgradation. This is a challenge, as for a lot of additional QA and testing work. most organizations are trying to make this move in an agile These trends are also having an impact on QA and testing manner and within a limited timeframe. Moving to the budgets. According to our CPRD respondents, the percentage latest ERP business suites successfully also involves a lot of the IT budget allocated to the QA and testing function of quality assurance (QA) and testing work and to speed was 26% in 2018, down from 28% last year. However, it is it up many organizations are looking at ways of doing important to note that while the proportional share has progressive automation. decreased slightly, both the overall IT budget as well as the absolute spends on QA and testing have been rising for most Consumer centricity, quality, and testing organizations. Importantly, it is the composition of this spend In CPRD, everything revolves around the consumer. According which has changed. to this year’s WQR survey, when asked about the objectives As we have already noted, some of the QA and testing spend of their QA and testing strategy, CPRD respondents gave the is being driven by the move to the latest ERP business suites highest weighting to “ensuring end-user satisfaction,” with as well as security testing linked to the move to the cloud. as many as 50% of CPRD respondents (the highest across all QA and testing spends today are no longer oriented only sectors), pointing to this as an important objective. This was towards items such as black-box testing, package testing, followed by 44% who indicated, “protect the corporate image or automation testing. Instead, a growing proportion of the and branding,” and another 44% who said, “detect software QA and testing budget can be called architectural spend, defects before go-live.” since it arises from the integration of testing ecosystems Similarly, when asked about the objectives of their IT strategy, with external and internal partners or from the move to the 52% of CPRD respondents mentioned “enhancing security,” cloud and the integration efforts on various applications.
50 2018 –19 World Quality Report 50 were organized, 54% of CPRD respondents (as against an In general, the CPRD sector is moving toward an open average of 45% across all sectors) said that “test activities architecture model, as everyone is moving away from are performed by all team members, supported by a test monolithic ERP applications and trying to create nimble professional,” while another 46% (43% across all sectors) systems using SaaS-based applications and open-architecture said that “test activities are performed by all team members, tools. This has increased the spends on integration testing. In without a specific test professional.” Both these statements addition, there are significant investments being made today reflect agile organizations and the data demonstrates the in tools, organization, and processes related to automation, greater adoption of agile in the CPRD sector, as compared test-driven development (TDD), and DevOps. to other sectors. With agile and DevOps adoption we also These investments are expected to lead to multiple see increased challenges around efficiency of testing the efficiencies and bring down the QA and testing costs in the incremental elements. As a consequence, CPRD organizations long run. However, in the short term, this spate of investments have increased their focus on test automation to solve these has led to a bump in QA and testing spends. According to our agile quality challenges. survey, as many as 72% of CPRD respondents said that they had seen an increase in the proportional effort and cost of QA When asked about the challenges in applying testing to agile and test activities over the last four to five years. Of course, development, as many as 59% of CPRD respondents pointed when going through these results, one should also keep in to a “lack of appropriate test environment and data.” This mind that the adoption of agile and DevOps have made it was the second-highest figure across all sectors. According more difficult to track exactly how much is being spent on QA to the survey, the biggest challenges in establishing and testing versus other activities. test environments for test teams were “cost of test environments” (55% of CPRD respondents), “defects due to A head start in agile, DevOps, and inaccurate configuration of test environments” (53% of CPRD respondents), and “having to maintain multiple versions of omni-channel testing test environments” (40% of CPRD respondents). Similarly, Another big trend in the CPRD sector is omni-channel the biggest challenges with regard to test data management engagement. As already mentioned, CPRD has always been were “creating and maintaining test data which are not copies ahead of the curve when it comes to this. This year’s survey from production data” (59% of CPRD respondents), and reveals an increased demand for reducing the test cycle “maintaining test data consistency across different systems times for these types of applications. When asked about their under test” (58% of CPRD respondents). greatest challenges in testing mobile, web, and other types of front-office applications, 51% of CPRD respondents said, Finally, another big challenge being faced by the CPRD sector “not enough time to test,” followed by 42% who said, “don’t is a lack of the new skill sets required by some of the new have the right tools to test,” and 30% who said, “don’t have technologies or frameworks such as Agile and DevOps. For the right testing process/method.” instance, when asked about the technical challenges they faced in developing applications, 47% of CPRD respondents The adoption of Agile and DevOps is driving the focus on replied with “lack of proper skills for QA and testing.” Some reducing test cycle times in this sector. According to this of the skills which are in short supply in the CPRD industry year’s survey, 100% of our CPRD respondents said they were relate to automation testing and architecting the move to using DevOps principles for at least some processes in their the cloud. organization. Similarly, when asked about how test activities Summary The CPRD sector today, displays an interesting mix of trends. While it has always taken the lead when it comes to Digital Transformation and omni-channel engagement, today the aspiration is to use these channels to customize the end-user experience for each individual customer. Organizations are also experimenting with the application of some emerging technologies such as artificial intelligence (AI), and machine learning (ML), to QA and testing. In the CPRD industry this is mostly seen in the form of chatbots to interact with consumers. CPRD organizations are also massively moving to the cloud and this trend is expected to pick up over the coming years. Despite all these encouraging signs, serious QA challenges still exist today. The biggest among them is related to a lack of the kind of skill sets required by trends such as agile, DevOps, cloud migrations, and automation. To tackle this challenge, the industry will have to find ways and means to attract younger people towards the testing function and put the right training programs, internships, and retraining programs in place. To some extent, this lack of skills can be alleviated by better automation, powered by the coming together of technologies, such as AI and ML. This combination of automation, AI, and ML is likely to be the focus over the next two to three years.
51 Sector Analysis 51 Energy and Utilities A period of disruption and competitive pressures driving QA and testing trends Randall Cozzens Executive Vice President, Head of North America Energy, Utilities, Chemicals & Services Market Unit, Capgemini to this year’s World Quality Report (WQR) survey, an average When we look at the Quality Assurance (QA) and testing trends in the Energy and Utilities sector something rather 72% of applications in this sector are based in the cloud. interesting emerges. While on one hand, the assets-heavy, There is a clear preference for private cloud in this sector, capital-intensive sector seems to be experiencing disruption with an average 25% of all applications based in the private in a big way with the onset of the “smart” era, the dependency cloud and just 14% in the public cloud. This is most likely a on legacy systems and a lack of the required skills is slowing result of security concerns, though we expect the percentage down the move toward this digital era. of applications in both private and public cloud to rise in the future. Though at a broad level, the sector is going through a period of disruption, there is a lot of variation across different Another major issue is the heavy dependency on legacy regions and organizations. This is due to the fact that, while systems which have been a major hindrance in the Energy broadly means Oil and Gas, the word Utilities is digitalization of the sector. However, digitalization is clearly a different in the US, Canada, and Europe. In Europe, the bulk focus area, given the competitive pressures and the need to of utilities remain regulated as opposed to some of the non- attract and retain customers. Of course, there are significant regulated trends we see in North America. While regulated challenges here as well. When asked about the biggest utilities are still trying to work their way towards agile and challenges in testing mobile, web, and other types of front- DevOps environments from legacy systems and waterfall- office applications, 56% (versus 52% across all sectors) said, type approaches, non-regulated utilities players have made “not enough time to test,” followed by 48% (versus 43% more impactful digital strides, particularly in their interaction across all sectors) who said, “we don’t have the right tools to with customers. test” and 37% (versus 34% across all sectors) who said, “we don’t have the right testing process or method.” Need for speed Our survey also revealed that an astounding 99% companies The utilities space is witnessing massive disruption at the are working with Internet of Things (IoT) products, a moment. Ten years ago, we had an electric grid that was statistic that is the highest across all sectors. That said, most reliant on coal-fired power plants and nuclear power plants. respondents (42%) shared that they do not have any specific Now, you see several dynamics at play: (i) a slow down or lack test strategy for IoT products, but plan to include one in the of expansion in nuclear power generation (more acute in the near future. This is despite the fact, that the industry is already US), (ii) a move away from the “dirtier” coal-fired generation, very asset-intensive and has sensors everywhere helping to and (iii) an uptick in “cleaner” burning natural gas as a collect data and monitor assets. This trend is likely to become generation source. Utilities are investing in meter services, for even more prevalent and important in the future. example putting in smart devices in homes so that a customer can manage their energy usage, know when usage peaks, Similarly, another emerging trend is related to blockchain, and get an insight into the energy consumption patterns with 80% of Energy and Utilities sector respondents saying of various appliances, etc. Companies are also supporting they are using blockchain in their portfolio today or plan to use other tangential services, such as landscaping, to keep the it in the coming year. These trends create a huge opportunity transmission lines and power lines clear of vegetation. as the journey to the cloud will help unlock the ability to be more agile, as well as to leverage artificial intelligence (AI) and Despite these major disruptions, Energy and Utilities IoT products. organizations have been slow to move to the cloud. According
52 2018 –19 World Quality Report 52 because their applications changed too much with every Lagging behind in agile and DevOps adoption release. In part, this could be due to an increased number of Historically, the sector has been slow to adopt frameworks, releases being mandated by business as well as an inability such as Agile and DevOps, with teams typically working to define requirements as robustly as required. Some of the in large silos. However, most organizations seem to have other responses include, “we have too many automation caught up with this trend over the last two years. According tools” (42% in the Energy and Utilities sector versus 29% to our survey, 97% of Energy and Utilities respondents (as across all sectors), “poorly defined project requirements opposed to an average 99% across all sectors) are using impeding decisions on right test scenarios” (38% in Energy DevOps principles in at least a subset of their projects. Of the and Utilities versus 25% across all sectors), and “a lack of companies using Agile, 38% said, “test activities are mostly skilled and experienced test automation resources” (38% in performed by test professionals,” and another 38% said, “test this sector versus 46% across all sectors). Overall, the Energy activities are performed in a distributed team.” As opposed to and Utilities sector seems to suffer from a lack of enterprise this, 25% of respondents said, “test activities are performed standards related to automation. by all team members, supported by a test professional,” and 42% said, “test activities are performed by all team Test budgets, cloud, and bimodality members, without a specific test professional.” These results According to our survey, 81% companies spend more than 10% not only show that there is an almost bimodal split between of their IT budget on testing and QA, with most companies agile and non-agile methods of organization of QA and allocating 21%–30% to the function. This covers testing testing activities, but that the split is still weighted towards processes, tools, and resource costs. Additionally, 77% of waterfall methods. Energy and Utilities sector respondents have seen an increase The slower move to Agile and DevOps is linked to the general in the proportional effort and cost spending on QA and test lack of maturity of QA and testing in this sector. When asked activities over the last few years, as opposed to an average about the challenges faced in applying testing to agile 72% across sectors. This possibly reflects the investments development, 52% of Energy and Utilities respondents (as taking place in terms of agile, DevOps, automation, and IoT. opposed to 50% across all sectors) said, “inability to apply As already mentioned, this sector has a significant legacy test automation at appropriate levels,” followed by 48% component, which creates a kind of bimodality in their IT respondents (48% across sectors) who cited, “difficulty systems as most as most front-office apps are often hosted on in slicing test activities for more than one location for the cloud, while their back-office systems are based on legacy distributed Agile,” and 46% respondents (39% across all systems. These monolithic IT systems are often not conducive sectors) who said, “difficulty to re-use and repeat tests across to agile and DevOps approaches, making it very difficult for sprints/iterations.” enterprises to integrate the newer, front-end solutions with Expert opinion holds that challenges related to test data and this legacy back-end. This is a knotty issue, which defies any environments are also very important in holding back this quick-and-easy solutions. sectors transition to agile and DevOps. The steady albeit slow Many of the trends seen in this survey are a direct reflection adoption of agile and DevOps is also driving the adoption of this bimodality which also leads to several of the challenges of automation in this sector. At present, automation is most in adopting more modern methods of QA and testing. Lack of popular for execution of test cases (17% automated versus the required skills is another issue which needs to be solved 16% across all sectors), followed by test data generation, on a priority basis. For instance, 98% of respondents reported (17% versus 15% across all sectors) and execution of end-to- difficulties in testing with agile and 52% of respondents cited end business scenarios (16% versus 15% across all sectors). the “inability to apply test automation at appropriate levels.” The Energy and Utilities sector also faces significant This is a clear indication of the lack of maturity and proper challenges when it comes to the adoption of automation. skills persisting in the public sector. Experts believe that lack An overwhelming 71% of respondents (as opposed to 61% of professional test expertise is one of the biggest challenges across sectors) said that they had difficulties in automation faced in the sector. Summary The Energy and Utilities sector needs to accelerate in digitizing its operations and move to an agile and DevOps environment. This sector must overcome the challenge of poorly defined project requirement to get the right test scenarios and deploy proper tools. The biggest trends to watch out for in the Energy and Utilities sector are grid modernization and customer engagement. More and more consumers are expecting to engage with their utilities companies as they engage with Amazon or any other leading retail company. Digital operations/manufacturing will continue to see adoption of AI and IoT, and business cases put together around the new technologies. Experts believe we will see an increase in the use of smart analytics, insights and data-related trends in QA and test activities in the future. In the coming years, the sector will also increase its focus on assuring quality, and testing and deploying rapidly. We believe that it will be the organizations which can scale agile and DevOps to the enterprise level quickly, which will emerge as winners in the future.
53 Sector Analysis 53 Financial Services An intensification of earlier trends in agile, DevOps, automation and intelligent QA Clemens Dietzsch Nilesh Vaidya Dhiraj Sinha Executive Vice President, Head Banking and Head of Finance Sector, Vice President, Sogeti Deutschland GmbH Financial Services Testing, Capgemini Capital Markets Consulting & Solutions, Capgemini Financial Services (FS) organizations are going through a Agile teams taking control fundamental change in outlook and modes of operation. Over FinTechs and larger financial organizations have been working the last couple of years, this sector has seen a lot of disruption with agile and DevOps project teams for a couple of years now. caused by digitalization and competition from technology This is now spreading to the small and middle-tier companies companies. While FinTech companies still lead in terms of as well. According to our survey, 98% of our Financial Services innovation, it is interesting to note that today even the large respondents said they were using DevOps in at least a subset financial institutions consider themselves to be FinTech of their projects. In addition, 35% of Financial Services sector companies as far as speed and innovation is concerned. respondents reported that implementing agile or DevOps This year, we see an acceleration of all the trends pointed was an important aspect of their IT strategy, as opposed to out in last year’s World Quality Report (WQR). Take Digital an average of 30% across all sectors. Today, FinTechs and the Transformation, for instance. While Financial Services larger organizations in the Financial Services industry have organizations have always been slightly ahead of the curve successfully adopted agile as a mindset and not just a fancy when it comes to digitalization, this year has seen even set of tools. greater strides being made in this direction, with a number When asked about the organization of their QA and testing of smaller and mid-tier organizations embarking upon their activities, 47% of our Financial Services respondents (versus Digital Transformation journeys. In addition, while initial Digital 45% across sectors) said that “test activities are performed Transformation efforts were focused on customer experience, by all team members, supported by a test professional,” today we are witnessing a more broad-based movement 43% (versus 43% across sectors) said that “test activities towards the digitalization of back-end processes. are performed by all team members, without a specific test Another trend that has picked up is the automation of quality professional,” 40% (versus 39% across sectors) said that “test assurance (QA) and testing. Certain parts of testing such activities are performed in a distributed team,” and 23% as performance and regression testing have always been (versus 33% across sectors) said that “test activities are mostly automated but today, there is a growing interest in automating performed by test professionals.” This shows that while the entire testing lifecycle. There is also a great interest in the testing is organized according to both agile and waterfall combination of artificial intelligence (AI) and automation to frameworks, there is today, a greater usage of Agile rather optimize and cut down on cycle time in testing. Organizations than waterfall for QA and testing activities in the Financial are experimenting with analytics, AI, and machine learning (ML) Services sector. to optimize the automation of QA and test activities. One such For financial services companies, distributed agile is also an example is the use of AI dashboards, which have helped agile important factor, given their global presence. Teams work teams work better, by providing easy project visibility to all across different regions and different time zones. The ability the stakeholders. to manage these teams well will spell increased success The adoption of agile and DevOps also continues apace this for these institutions. But, 47% of respondents state that year. This, along with all the above trends shows up as an they have difficulty in slicing test activities for more than increased demand for software development engineers in one location. testing (SDETs) and, in general, a higher demand for testers with Agile teams also create a demand for more automation tools. advanced technical skills. This factor is likely to become even The speed at which an agile team operates simply cannot be more important as the complexity of applications increases. matched by manual testing. The Financial Services sector has Moreover, advances in technologies such as robotic process employed automation tools in the past, but now there is a automation (RPA), ML, and AI, and their use in testing, are likely need to widen the scope of automation. to drive the demand for newer types of skills in project teams.
54 2018 –19 World Quality Report 54 that their products have IoT functionality but lack a specific A wider scope for automation test strategy. This is something that needs to change in the One trend that has gained traction since last year’s WQR survey future to accommodate the growing usage of IoT in the is test automation. Today, the automation of the regression Financial Services sector. pack is almost a given in most Financial Services organizations and now there is a clear movement toward progression/ in-sprint automation and end-to-end automation. In other Intelligent QA – the science behind testing words, automation is no longer confined to test design and The financial services sector also has a growing appetite for test data execution – it is increasingly entering areas such as intelligent or predictive QA tools. Twenty-four percent of test data and environments, reporting, and dashboards, etc. respondents say that cognitive capabilities are an important We expect that these trends will continue to gain strength AI research area for their business. Companies are now with the increasing adoption of agile and DevOps in the investing in cognitive QA make testing more scientific. With future. In addition, there is also a move toward reducing cognitive QA there are lesser chances of human error and operations costs by reducing paperwork, digitizing most more control on important parameters like test environment processes, and increasing the levels of automation. However, and test data management. the overall levels of test automation remain low, and several Robotic process automation (RPA) is also an important challenges stand in the way of greater adoption. For instance, feature of intelligent QA and helps companies to reduce the when asked about the challenges in achieving their desired time wasted on repetitive tasks. In fact, 34% of respondents level of test automation, 64% of Financial Services sector say that they are considering using bots for data generation. respondents (vs. 61% across all sectors) said they had difficulties in automating as their applications changed too Further, such technologies can also provide companies with much with every release. A further 62% of respondents (vs. interactive dashboards that can be accessed by different 48% across all sectors) said they faced challenges with test stakeholders to get a comprehensive view of testing data and environment availability and stability, while 40% of activities. Cognitive QA can also make automation smoother, respondents (vs. 42% across all sectors) said they didn’t have thus helping agile teams deliver much more efficiently. the right automation tools. Another interesting aspect within AI is the need for testing Encouragingly, 71% of Financial Services respondents said business processes created on AI platforms. For instance, that automation had led to better test coverage, 71% said it companies are also using AI tools for creating more intuitive had led to a reduction in test cycle time and a further 69% and engaging tools for customers such as chatbots. Testing said it had led to better reuse of test cases. It is important to the functionality and security of chatbots will be a new note that today, one of the biggest drivers for the adoption of experience for testing teams. automation is better time to market rather than cost, which Companies are also investing more money to test their AI has become a secondary concern. In this regard, automation applications with real devices. Twenty-three percent of users seems to be delivering on its promises. say that they test with real users and real devices before launching an application. For companies, using real devices is an expensive option but guarantees better testing results. Making the most of data Further, 18% of respondents say that they use virtualization To further improve their speed-to-market, companies are technologies to test before launch. This clearly points to a turning to analytics. With predictive analytics, companies can need for creating a robust virtualization infrastructure by identify patterns in testing and predict pitfalls in advance. combining real devices, simulators, and emulators to test Fifty-nine percent of respondents say that predictive analytics their new AI applications. is an upcoming trend in the coming year. Another interesting development for Financial Services Technical developments impacting QA in the organizations is the Internet of Things (IoT). Although, the impact of IoT is not as huge as in manufacturing, many Financial Services sector financial companies now use data captured from connected As mentioned earlier, there is today an increasing focus on devices to predict patterns in their customer behavior. For digitalization of back-end applications. This means a greater instance, car insurance companies analyze the driving data of stress on integration of the customer-facing applications their customers to offer them the right premium. with back-end systems and processes. This has increased This means an added layer of testing for companies to ensure the importance of integration testing which helps ensures that data is accurate. However, according to our survey, that the different components in a system work smoothly twenty seven percent of Financial Sector respondents said together. However, there are a few challenges in this regard.
55 Sector Analysis 55 For testing organizations, APIs present a unique challenge For instance, 21% of respondents said that an inability to owing to many participants and here too, integration testing test integration at an early stage is a challenge in developing can help companies test the performance and security applications today. In addition, 50% of Financial Services sector of APIs. respondents also said they lack proper test data for complex integration testing. Multi-channel testing will also be an important focus area for Financial Services companies. Presently 23% of respondents Today, Financial Services sectors companies are increasingly state that automation of customer experience testing is a harnessing the power of the application programming interface challenge in multi-channel testing. Another 21% state that (API) economy. With a rise in platform-based business models, establishing the test data for customer-experience testing is APIs help in bringing stakeholders on a common platform to also a challenge in this area. create innovative customer-facing applications. Summary This year’s survey reveals not just a continuation but rather an intensification of the trends seen over the last few years in the Financial Services sectors. There is a continuing increase in digitalization, agile, DevOps, and automation along with the emergence of new technologies such as blockchain, IoT and AI which are reshaping the way QA and testing is done. Over the next two to three years, it is the combination of automation, AI, and analytics that will emerge as a key competitive differentiator and shape the evolution of QA and testing.
56 2018 –19 World Quality Report 56 Healthcare and Life Sciences The rise of digital health Mallick Azfar Vice President, Life Sciences, Sogeti, Capgemini Group respondents said that blockchain was already a part of their The Healthcare and Life Sciences sector consists of portfolio or would be within the next year. The Healthcare biotechnology, pharmaceutical, animal health, crop science, and Life Sciences sector is also seeing an increasing number and medical device companies on the Life Sciences side and of IoT-enabled devices coming out every year, to the extent healthcare service providers and payers on the Healthcare that a new term – Software-as-a-Medical Device (SaMD) has side. The entire sector is heavily regulated and has historically been coined for this class of devices, which require a different had a high degree of quality consciousness. Over the years, it kind of lifecycle testing. has seen a growing trend of patient centricity, a fact reflected in this year’s WQR survey. For instance, when asked about According to our survey, 46% of Healthcare and Life the objectives of their QA and testing strategy, Healthcare Sciences respondents said they had a mature IoT test respondents gave the highest weighting to “ensuring end- strategy, as opposed to 42% across sectors. Expert opinion, user satisfaction,” with 43% of respondents indicating this as however, holds that most organizations in the sector do not an important objective. differentiate between IoT and standard enterprise application testing. Therefore, one could question the true maturity of This focus on patient centricity drives some of the important IoT testing strategies in this sector. For example aspects like trends that are shaking up this sector today. Firstly, the need security, connectivity, reliability, and interoperability of IoT to provide end-to-end servicing of patient’s needs is bringing devices would typically not be covered to the required level. together various entities across the Healthcare and Life Regardless, the fact is that more and more IoT devices are Sciences spectrum. Secondly, in the last two years, social, being released in the market today and the testing strategies mobile, analytics, and cloud (SMAC) solutions have majorly themselves must mature within the next three years. impacted this sector, changing how organizations run their business and interact with their patients, hospitals, and regulatory authorities. Cloud and digital transformation These solutions have also given rise to the concept of digital Just like the Public sector, the Healthcare and Life Sciences health. Where once, pharmaceutical companies would merely industry has been slow in moving to the cloud and the reasons create the pills for managing your blood pressure, today, they are similar. Both sectors provide services that are essential to also create lifestyle apps to help you monitor and manage that the well-being of people and have, over a long period of time, blood pressure. Such apps might allow you to get advice from depended upon monolithic IT systems that delivered well. doctors, connect with other patients, track your progress, and That factor, coupled with the critical nature of the services send alerts to hospitals. provided, help explain why organizations in this space are Just like the wider sector, these apps also have a regulatory reluctant to move to the cloud. aspect connected to them and thus require testing and Security concerns are equally important since Healthcare due diligence. Such regulatory challenges have further and Life Sciences organizations often hold sensitive personal complicated the traditional application development data. In general security is very important to all organizations challenges, which continue to persist. When quizzed about in this sector. When asked about the objectives of their IT the technical challenges in developing applications, 48% strategy, 47% of Healthcare and Life Sciences respondents of Healthcare and Life Sciences respondents indicated the said they wanted to “enhance security,” making it the most “lack of end-to-end automation from build to deployment,” important objective. This importance placed on security and followed by 45% who said, “too slow testing process,” and the sensitive nature of the personal data in their possession, 42% who said, “lack of proper skills for QA and testing.” has held back many organizations in this sector from moving Another major trend in the Healthcare and Life Sciences to the cloud. sector is the adoption of a slew of new technologies such We can see the impact of these factors in our survey results. as digital manufacturing, 3D printing, and blockchain. According to our respondents, approximately 70% of For instance, 51% of Healthcare and Life Sciences survey
57 Sector Analysis 57 applications in Healthcare and Life Sciences organizations Automation and AI are based in the cloud, versus roughly 73% across sectors. When asked about the technical challenges in developing On average, 22% of Healthcare and Life Sciences applications applications today, “the lack of end-to-end automation” run in a private cloud, 19% run in a public cloud, 15% run in a from build to deployment came out as the biggest challenge hybrid cloud and 16% on an on-premises cloud. across sectors, with 48% of Healthcare and Life Sciences respondents indicating this as an issue. Apart from the cloud, another major area of transformation in the Healthcare sector is the concept of digital health. As According to our survey, test automation in this sector is mentioned earlier, digital health involves the mobile and mostly used for the generation of functional test cases (18% web-based lifestyle apps being launched by organizations of such cases are automated), followed by the execution of across the sector. This new trend has given rise to a number end-to-end business scenarios and the execution of functional test cases (approximately 16% each). The low adoption rates of specific challenges when it comes to QA and testing. are directly related to the underlying lack of a well-developed According to Healthcare and Life Sciences respondents, their and thought-through test strategies. biggest challenges in testing mobile, web, and other types of front-office applications were, “not enough time to test” (54% One of the keys to making test automation more effective of respondents), “don’t have the right tools to test” (46% of is the application of AI in testing and automation. Given the respondents), “don’t have in-house testing environment,” and testing workloads today, the reality is that organizations need “don’t have devices readily available.” to figure out what to automate, what not to automate, and what to automate first. This level of cognitive planning or Underlying each of these challenges is the fact that the smart test automation will become essential over the next adoption of agile and DevOps has increased the frequency two to three years as it is not humanly possible to test, for of releases and the amount of testing required. The pressure example, 3,000 scenarios every four weeks. This is where this puts on QA and testing is further compounded by smartness or the use of predictive analytics comes in. the sheer number of scenarios that need to be tested. To There are signs that companies are looking at AI seriously. solve these challenges, organizations need to stop treating 56% of Healthcare and Life Sciences respondents said they testing as an after-the-fact activity and embrace test-driven had AI projects for QA in place or planned for the next 12 development (TDD). This is a shift that has not yet happened, months. Similarly, 45% of Healthcare and Life Sciences sector as most organizations are still working along the lines of respondents said they were leveraging analytics and AI for requirements-driven development. intelligent automation. The focus on intelligent automation is increasing for two main reasons. The first is the belief that The other step that will help to deal with the increased testing intelligent automation will increase speed-to-market and the workload is the adoption of automation. second is the belief that it will reduce or eliminate human error. Summary manual testing, and this is what increases the importance of The Healthcare and Life Sciences sector has lagged behind automation and more specifically, of smart automation in this in the adoption of the latest technologies and frameworks sector. for smarter and faster QA and testing. This sector has also seen the persistence of a significant legacy component It is clear that digitalization and smart automation are going and an over-reliance on permanent test environments to be a core theme for this sector over the next two to three (34% of testing done using permanent test environments years. However, to adopt these technologies successfully, in Healthcare and Life Sciences vs. 31% across all sectors). organizations need to overcome the following challenges: Moreover, the adoption of agile has been held back by this Organizations need to have a well- - Clarity of roadmap: legacy component. While agile has been a buzzword with thought out testing strategy and roadmap in place. the big health companies for at least two years, it is mostly Currently, we see many organizations adopting different applied to front-end and customer-facing applications, with automation frameworks or pieces on an ad-hoc basis the back-office processes still using waterfall methodologies. for different business processes without thinking of a cohesive, end-to-end automation strategy. When asked about the challenges in applying testing to agile development, Healthcare and Life Sciences respondents gave Testing is still extremely The digital labor workforce: - the highest weighting to,“lack of appropriate test data and manual and is struggling to handle the increased environments,” with 60% of respondents (the second highest workloads brought on by digitalization, agile, and DevOps. across all sectors surveyed citing this as a challenge). Test data Technologies such as IoT, automation, and AI all require and test environments are a big challenge, simply because of specialized skills for which the existing labor force needs to upgrade its skills. Organizations need to put in place the sheer number of possible scenarios that must be tested. skill development plans to make this happen. It is virtually impossible to deal with this relying solely on
58 2018 –19 World Quality Report 58 H ig h -Te c h Ahead of the curve in technological change and the adoption of QA and testing to validate new technologies Vivek Jaykrishnan Malavika Athavale Sidharth Kapila Senior Director - Technology, Engineering Vice President, Vice President, Head-Product V&V Practice, Engineering Services, Capgemini High-Tech Portfolio Leader, Capgemini Services Global Business Line, Capgemini satisfaction. This has led to a much greater integration of IT Rapid and ever-increasing change has been a theme we and business, with business goals becoming all-important. have already seen in many of the other WQR chapters this Thus, when asked about the goals of their QA and testing year. The IT industry has changed so much that it is virtually strategy, 48% of High-Tech industry respondents pointed unrecognizable from what it was ten years ago. While to,“ensure end-user satisfaction” as opposed to 42% across every sector has been impacted by the relentless pace of all sectors. Similarly, when asked about objectives of their IT advances in technology, business models, and organizational strategy, 37% of respondents from the High-Tech sector said frameworks, it stands to reason that the High-Tech sector “faster time to market’ was important as opposed to 30% (covering manufacturers of hardware products, electronics, across all sectors. This emphasis on time to market as well aerospace and advanced defense technologies) which itself as other business goals has led to the increased adoption underpins many of these technological evolutions is feeling of cloud and reinforced the adoption of agile, DevOps, and the effects of this rapid evolution more than other sectors. automation mentioned earlier. The velocity of change is even greater here and this basic fact imbues this sector with characteristics that set it apart. There are two other factors, both deriving from its history, Being the sector driving many of these changes, High Tech which make the High-Tech sector unique. The first is that large has also been among the first to adopt them. This has led to parts of the High-Tech industry (the aerospace and defense it being ahead of the curve when it comes to the maturity product manufacturers) have traditionally been heavily of its information technology (IT), quality assurance (QA) regulated with a significant proportion of their business coming from government contracts. This historic focus on meeting regulatory and compliance requirements has led to According to our survey, an average 77% of a high level of quality consciousness, with a lot of emphasis all applications in the High-Tech sector are placed on QA and testing in this sector. cloud-based as opposed to an average of The second factor that sets High Tech apart, is its considerable 73% across all sectors. experience with handling of connected products. This gives the sector a lead in terms of IoT testing. According to our survey, 48% of High-Tech sector respondents said they have a fairly mature IoT strategy, compared to an average of 41% and testing practices. Over the last three to four years, across all sectors. Here, it’s important to add the cautionary there have been a spate of investments in agile, DevOps, note that a number of these respondents might have been and automation projects. This has had an impact on QA referring to the testing of their embedded products rather and testing as well; something that came through clearly than the complete end-to-end testing required for IoT in this year’s WQR survey results. For instance, when asked products. whether they had seen an increase in proportional effort and spending on QA and test activities, as many as 79% of High- One can see the impact of all these factors running through Tech respondents said yes as against an average 72% across many of the survey results. This is seen even at the broadest all sectors. Today, these projects have moved beyond the PoC levels. For instance, when asked about the objectives of their stage to the implementation stage and High Tech has a lead IT strategy, 53% of High-Tech respondents said, “enhance over most other sectors when it comes to experience with security” (as opposed to 47% across sectors), followed by agile, DevOps, cloud, and automation. 45% who said, “enhance customer experience” (as opposed to 42% across sectors), and 46% who said, “higher quality of Technological changes have also brought about business software solutions.” The focus on security is understandable, disruption particularly for the High-Tech manufacturing given the levels of cloud adoption as well as the work with industry, which has moved from selling products to selling connected devices. Similarly, the focus on quality results from subscriptions. This is known as the subscription economy and the historical factors pointed out earlier, while the focus on success in this new model depends upon speed and customer
59 Sector Analysis 59 from its two distinguishing features; namely, its early adoption customer experience can be explained by the rise of the subscription model and the process of Digital Transformation of emerging technologies and the rapid pace of change in the which has impacted every sector. industry. These challenges include issues with test data and test environments, as well as the lack of the kind of skill sets High-Tech: Being the change required by the new testing approaches. As already mentioned, the High-Tech sector is ahead of the For instance, when asked about the challenges in applying other sectors in terms of the maturity of its IT and QA and testing to agile development, 47% of the High-Tech testing practices. This is clearly reflected in our survey results. respondents said that lack of appropriate test environment The first place we see this is in the adoption of agile and and data was an issue. These issues stem from the increased DevOps. For instance, when asked how test activities were complexity as well as greater number of releases today. In a performed, 62% of High-Tech respondents (as opposed to situation where some companies, are coming out with new 45% across all sectors) said “test activities were performed releases every week, if not every day, how do you understand by all team members, supported by a test professional,” the incremental functionality of the new release and how and 50% said “test activities were performed by all team do you design test environments to provide adequate risk members, without a specific test professional” (as opposed to coverage? To solve such problems, organizations are now 43% across all sectors). These two statements reflect that QA looking at the next level of predictive analytics solutions to is much more integrated in each process step and everyone’s automate the identification of risk areas. This also came up business, as compared to other industries. in our survey results. For instance, when asked about the At the same time, the stress on time-to-market has resulted opportunities for AI in testing, 58% of the respondents from in a greater adoption of cloud. Per our survey, an average the High-Tech sector pointed to the use of AI to automatically 77% of all applications in the High-Tech sector are cloud- create test cases which ensured the right risk coverage. based as opposed to an average of 73% across all sectors. Similar challenges exist with regard to test data as well. In Out of the different cloud models, private cloud is the most general, test data management today has to deal with several popular (23% of all applications), followed by public cloud issues related to the complexity of data, newer forms of data (18%), on-premises cloud (18%), and hybrid cloud (15% of as well as integrating different formats of data derived from all applications). different sources. These problems are made worse in the High- The impact of this high level of cloud adoption can also be Tech manufacturing sector, where most companies outsource seen when it comes to test environments. For instance, manufacturing to original device manufacturers (ODMs), according to High-Tech respondents, on average, only 26% who, in turn source parts from multiple smaller organizations. of their testing occurred in a permanent test environment, Setting up accurate data sets that mirror an end-to-end compared to the 31% average across all other sectors. business transaction becomes very difficult in such situations Additionally, an average 22% of their testing occurred in a as a lot of the data sits outside the organization. Sharing such cloud-based temporary test environment (as opposed to data with partners also leads to questions around privacy 19% across all other sectors) and 18% of testing took place and security. This is an ongoing challenge with no universally in containerized test environments (compared to 16% across accepted answers having been developed as yet. all other sectors). There is also a lot of interest in the use of both AI as well Finally, there is also a huge challenge around skill sets. The as predictive analytics in testing. For instance, 65% of combination of changes related to agile, DevOps, continuous respondents from the High-Tech sector, as opposed to 57% engineering, and automation have all led to a change in across all sectors, said they had AI projects in place or planned the job profile of a tester. The specialized manual tester of for the next year. Organizations are also looking at ways to use yesteryear is no longer relevant. Instead there is an increasing descriptive analytics in test lifecycle management data which requirement for software development engineers (SDEs) and we can see from the 60% of WQR High-Tech respondents software development engineer in test (SDET). At present, who want to use AI in smart dashboards. there is a shortage of skills in the market but given the amount of re-skilling and training programs that companies The perils of being an early adopter are going in for, we expect this problem to go away over the next few quarters. Most of the challenges faced by the High-Tech sector stem Summary The High-Tech sector has been one of the early adopters for many new frameworks and technologies. This basic fact is the source of both the maturity of its QA and testing practice as well as the challenges faced by its QA and testing teams. The rapid pace of change and the newer technologies all place great demands on its QA teams and moving forward, the sector is increasingly going to look towards newer technologies such as AI, machine learning, and predictive analytics to solve some of these issues.
60 World Quality Report 2018 –19 60 Public sector QA approaches need to mature to be able to service citizens better, faster, and more efficiently Michael Strecker Roland Schäfer Nikki Green Matt Howell Head of Sector Public, Executive Vice President, Delivery Director, Senior Solution Architect Sogeti Deutschland UK Test Centre, Head of Capgemini Public – Public Sector, Sogeti GmbH Capgemini Deutschland GmbH Market Unit, U.K. Our 2018 survey results reveal some very mixed trends for The rocky road to digitalization quality assurance (QA) and testing in the Public sector. On Digital Transformation is driven by the growing awareness the one hand, we see a maturation of some of the broad that the Public sector needs to up its game to provide the themes pointed out in last year’s WQR. On the other, we see same levels of comfort, convenience, and service as the considerable variation across regions mainly due to Public private sector. This came through loud and clear in this year’s sector budgets as well as policy imperatives being driven by survey. When asked about the objectives of their QA and political considerations, which themselves vary from country testing activities, 41% of respondents in the Public sector, to country. Despite this variation, three basic facts hold across pointed to “ensure end-user satisfaction” as an important different regions and they explain many of the survey results objective. It is important to note that this figure has risen for the Public sector. sharply over the last two years: in 2017, only 34% of Public sector respondents mentioned this as a priority, while in The first is the evolution of a theme we talked about last year – 2016, this figure stood at 31%. the disaggregation of contracts. More and more governments are switching to smaller deals with multiple suppliers, on the To meet these objectives, governments everywhere are premise that the huge, single prime IT contracts that used to simplifying access to information and services by making be in vogue earlier have not delivered the agility, flexibility, them available online or through mobile apps. This gives rise or value for money that was expected of them. Ironically, this to new challenges for QA and testing. When asked about has added a lot of complexity to the IT ecosystem and given their challenges in testing mobile, web, and other front-office rise to a new set of challenges. Mostly challenges around applications, Public sector respondents gave the highest integration and end-to-end testing, their influence can be weighting to “not enough time to test” (44% of respondents), followed by “don’t have the right tools to test” (38%), “don’t seen when we look at the survey results for questions relating have the right testing process/method” (28%), and “don’t to agile and DevOps, cloud, and automation. have an in-house testing environment”(28%). Many of these The second fact is the continuing reliance on legacy systems. issues arise due to the lack of a consistent end-to-end testing This leads to a kind of bimodality in the Public sector IT and tooling strategy, a task made harder by the integration systems, with newer, more front-end apps often being hosted challenges resulting from disaggregation and the pre- in the cloud and developed using agile and DevOps, while ponderance of legacy systems. older apps continue to be hosted on these legacy systems. These are large, stable, monolithic IT systems which often handle masses of data and deliver services that can be critical Agile and DevOps adoption hampered by legacy to the security or well-being of a country. Such systems are The desire to serve citizens better, faster, and more efficiently often not very conducive to the newer QA approaches and is also driving the adoption of agile and DevOps. But the technologies. In fact, integration of the newer, front-end Public sector is slower in adopting the distributed and self- solutions with this legacy back-end is one of the biggest empowered team approaches that come with this transition. challenges facing this sector. The criticality of these systems According to this year’s survey, 39% in the Public sector (as (and the services they deliver) makes this a very delicate opposed to 43% across all sectors) said that “test activities problem that defies quick solutions. are performed by all team members, without a specific test professional,” while 34% (as opposed to 33% across all The third fact is QA and testing for Digital Transformation sectors) said that “test activities were mostly performed by projects. While there is considerable variation across different test professionals.” This shows that the Public sector still lags regions, there is no doubt that today, Digital Transformation other sectors in the maturity of its agile practices. In addition, (and the associated QA and testing) is a top priority for the the Public sector is slower in the change in processes, Public sector all over the world.
61 Sector Analysis 61 hen asked about the objectives of W their QA and testing activities, 41% of respondents in the Public Sector, pointed to “ensure end-user satisfaction” as an important objective. as well as the lack of automation testing skills. For instance, workflows and culture that come with the transition to when asked about the challenges in achieving their desired agile. As a result, we often see the existence of both agile level of test automation, 55% of respondents from the Public and waterfall within the same ecosystem in many Public sector (as opposed to 46% overall) pointed to the lack of sector organizations. skilled and experienced test automation resources. This was The survey results reflect this bi-modality, a feature which the second highest across all sectors surveyed. This lack of the gives rise to several specific challenges. For instance, when required skill-sets impacts most of the QA trends seen in this asked about their challenges in applying testing to agile year’s survey. development, 57% of respondents reported “a lack of appropriate test environment and data,” 46% reported “an inability to apply test automation at appropriate levels,” and The skills gap: addressing the elephant 44% reported “difficulty in identifying the right areas on in the room which test should focus.” All three of these statements can The trend towards disaggregation means that Public sector be explained by the Public sector’s lack of maturity in agile organizations now need to have a certain level of in-house and DevOps as well as a serious challenge regarding skills and IT skills to manage multi-vendor contracts as well as agile capabilities. The challenges that we see here also apply to the construct. This is in sharp contrast to the earlier scenario, adoption of cloud and automation. in which many organizations would leave most IT decisions in the hands of their suppliers. In addition, the existence of Slowly taking root: cloud and automation both waterfall as well as agile methodologies within the same ecosystem means that both the old, classic QA testers as well Even though there is a noticeable move towards both as the newer, T-shaped testers (QA professionals with a deep cloud and automation, the Public sector still lags behind expertise in testing, development and business) have a role to when it comes to the adoption of these two technologies. play in the Public sector. Survey results indicate that an average 67% of applications in the Public sector (vs. 73% across all sectors) are hosted The difficulties in finding some of the newer skills required in the cloud. It is important to note, that this 67% is an have been clearly brought out by our survey. For instance, average figure with results varying widely across different when asked about their greatest challenge in testing mobile, regions depending upon the country’s laws and regulations web, and other front-office applications, 31% of Public sector regarding customer’s personal data. 67% is also the lowest respondents said they did not have the right experts. This was across all sectors surveyed. This is mostly due to the wide the second-highest weighting across all sectors. Skills were dependency on legacy systems in handling huge volumes of also an issue when it came to the adoption of agile. This year, data and supporting critical applications and services. The 41% of Public sector respondents, as opposed to 39% last satisfactory performance of the legacy systems along with year, pointed to the lack of professional test expertise in agile the risks involved in migrating critical applications have both teams as a challenge. To address this issue, organizations weakened the business case for a move to the cloud. need to focus on creating capability as well as a pipeline of talent by setting up training programs, apprenticeships, and Similar trends hold for test automation. When asked about on-boarding capabilities. the percentage of test automation for different activities, Public sector respondents gave the third lowest responses Apart from the above trends, there is also a lot of talk about (across all sectors) for “percentage of performance test cases AI and cognitive technologies and a desire to move towards that are executed with test automation tools,” at 15.3% more predictive testing, though this is still at a very nascent and “percentage of end-to-end business scenarios that are stage. The organization of testing activities too is seeing executed with test automation tools,” at 14.2%. This lower some interesting shifts. While we saw a move away from adoption can be explained by the existence of a legacy estate managed testing services over the last two to three years,
62 2018 –19 World Quality Report 62 management were “maintaining test data consistency across it appears TCoEs are again coming back, though in a newer iteration. This new, more evolved form of the TCoE is almost different systems under test,” “managing the size of test data a hybrid, with some components centralized and others sets,” and “creating and maintaining test data which were not decentralized. The roles suited to centralization tend to be copies of production data.” To some extent, these issues can around performance, security, and integration. This was also be addressed by a greater use of automation, which can help reflected in our survey results, in which the most centralized in creating re-usable data and environments as well as reduce roles in the Public sector were those of business domain- defects and errors. based testers (22% for the Public sector vs. 19% across all sectors) and software development engineer testers (SDETS) There are many challenges hindering the greater usage of (20% for both the Public sector as well as across all sectors). automation in testing. In some sense, it is a vicious circle in which Public sector is stuck, i.e. the existence of legacy Considerable challenges also exist with test environment systems holds back the adoption of cloud, Agile, and Digital and test data management. According to our survey, the Transformation - all of which would have naturally lead most important issues with regard to test environments to higher levels of test automation techniques, which in in the Public sector were “the cost of test environments” turn could have helped solve challenges related to data and “defects due to inaccurate configuration of test environments.” Similarly, the biggest issues for test data and environments. Summary At some point, the Public sector will have to bite the bullet with regard to its dependence on legacy systems. However, this cannot happen overnight. Over the next two to three years, we expect to see a lot of focus on integration, on developing skills and capabilities within organizations and a focus on delivering end-to-end test assurance. We also expect a convergence in testing processes as the evolution of automation technologies allows a more integrated and automated approach to testing.
63 Sector Analysis 63 Telecom, Media and Entertainment Convergence and changing business models driving a focus on business adaptiveness and flexibility Darren Coupland Madan Sundararaju Vice President and Executive Vice President and Chief Portfolio Leader – Media, Capgemini Operating Officer, Sogeti U.K. The Telecom, Media and Entertainment (TME) industry is The increasing adoption of cloud, agile, the midst of a shakeup. While this market has always been a DevOps, and automation dynamic and fast-moving one, it has today entered a period This year’s survey clearly reveals the value placed on customer of particularly rapid change, driven by cost pressures, a highly satisfaction and the increasing alignment between IT and competitive market, and technological development. On business. When asked about the objectives of their IT strategy, the one hand, emerging technologies such as 5G are driving 50% of TME respondents (versus 47% across sectors) said, organizations to update their estate and their services. Thus, “to enhance security”, followed by 43% (42% across sectors) we see big investments being made in network innovation, who said, “to enhance customer experience,” and 36% (36% with a move toward software-defined networks (SDN) and across sectors) who said, “higher responsiveness to business network function virtualization (NFV). On the other hand, demands.” Similarly, when asked about the objectives of viewership habits have also changed in the Media and their QA and testing strategy, 47% of TME respondents Entertainment industry, with more and more content being (39% across sectors) said, “to protect the corporate image consumed via smartphones. This in turn has sparked a wave and branding,”46% (42% across sectors) said “to detect of mergers and acquisitions as big Telecom companies try to software defects before go-live,” 44% (42% across sectors) control both the content and the channels through which it is said “to ensure end-user satisfaction,” and another 44% (41% consumed. As big Telecom companies push ahead with such across sectors) said, “to contribute to business growth and quad-play strategies, there is a convergence between the business outcomes.” Telecom and Media and Entertainment worlds. In addition, internet-based media companies, such as like Netflix or Almost all the answers (other than “enhance security”) are Amazon, have also caused massive disruptions by increasing directly related to business outcomes. The high premium competitive pressures. put on security in this sector can be explained by the large subscriber base of a typical Telecom company. These All of this has implications for quality assurance (QA) and organizations have access to their customers’ personal data testing. Increased competition has led to a fight to attract and and breaches can have an adverse impact on brand value retain customers, thus increasing the importance of customer and corporate image. In addition, regulations, such as the satisfaction. With disruptors, such as Netflix and Amazon, General Data Protection Regulation (GDPR), have introduced working on rapid cycles and producing a lot of content very penalties that increase the costs of such breaches. quickly, time to market has emerged as a key competitive It is this focus on adaptive business objectives that drives the differentiator. This is driving the movement to the cloud as increasing adoption of cloud, agile and DevOps. According well as the adoption of agile and DevOps. Overall, there is to our survey, an average 74% of applications (versus 73% movement toward service-type operating models, with QA across sectors) in the TME sector are based in the cloud. Given and testing services also being bundled and offered through the concerns regarding security pointed out earlier, there service catalogs. The need for speed and flexibility has also is a preference for private cloud, with an average 24% of led to a desire for greater automation. applications in the TME sector (versus 22% across all sectors) being hosted there. The public cloud is adopted mostly for These changes have also led to QA and testing challenges. non-critical applications, with an average 17% of applications For instance, there is a tendency in certain quarters to push in the TME sector being hosted in a public cloud. In addition, for speed, even at the cost of quality. This is an unfortunate an increasing movement toward hybrid cloud started this development, and, in some sense, it implies a faulty year, with organizations trying to get the best of both worlds application of agile and DevOps. In addition, there are (the cheapness of the public cloud, together with the security challenges around test data and environments, as well as with promised by private cloud solutions). We expect that this the skill sets required by the new operating frameworks. percentage will rise in the future.
64 2018 –19 World Quality Report 64 Today, there are encouraging signs of at least a few big TME A move toward intelligent automation companies resolving to move their entire estate to the cloud. The “speed-versus-quality” dynamic brought about by the For many years, the TME sector simply avoided dealing with adoption of agile and DevOps shows up in a number of places these issues, due to the age of the application estate and in our survey results. For instance, when asked about their the complexity of applications such as customer relationship greatest challenge in testing mobile, web, and other types of management (CRM) applications and billing systems. The front-office applications, TME respondents gave the highest focus was on achieving efficiency and improving time-to- weighting to “not enough time to test” (60% of respondents), market, without making hard, strategic choices about the followed by “we don’t have the right tools to test” (49% of legacy estate. This is what led to the bimodality in a number respondents), and “we don’t have the right testing process/ of the WQR survey results we saw in last year’s TME chapter. method” (34% of respondents). Today, however, the desire for a greater adoption of agile, The perceived lack of time is not surprising, given that TME DevOps, and automation (driven by promised benefits, such organizations are usually marketing-led organizations in as greater efficiency, speed, and better risk coverage) are which business puts a lot of pressure to get new releases forcing organizations to take these hard decisions. However, out before the competition. Agile has led to more frequent given the complexity of the legacy estate, this will be release cycles and this also adds to the problem. The lack of neither quick, nor easy. It will also involve a large amount of tools to test is a result of the complexity of the applications investments as a new stack will have to be prepared, tested, estate as the sheer number of applications, all using different and configured – following which data migration will start. technologies requires multiple tools to test, thus making end Throughout this process, two operations will have to be to end testing a huge challenge. Finally, the issues around not up and running, requiring huge operating expenses. These having the right testing process or method, possibly reflects expenses present a major challenge. However, the good news the aforementioned “speed-over-quality” dynamic as well as a is that a few organizations have at least started trying to solve very real lack of skill sets required by new technologies, such these challenges posed by the legacy estate. as automation and artificial intelligence (AI), as well as the new frameworks, such as Agile that call for profiles such as The same imperatives that are driving cloud adoption are also software development engineers in test (SDETs). driving the adoption of agile and DevOps. According to our survey, 100% of TME organizations have adopted DevOps The TME sector’s desire to automate is propelled by the same for at least some of their projects. When asked how test objectives driving the adoption of cloud, agile and DevOps. activities were performed, 55% of TME respondents (versus According to our survey, 17% of all functional test cases in 33% across all sectors), said that “test activities were mostly the TME sector are generated with test generation tools, 17% performed by test professionals.” The fact that just a little of performance test cases are executed with test automation over 50% of Scrum teams have specialized testers is worrying, tools and another 17% of security tests are executed using as it indicates that QA and testing has perhaps still not found test automation tools. These percentages will certainly its place within agile teams. This is typical, as both agile and increase in the future, particularly as AI and analytics is applied DevOps are developer-driven activities and it can be difficult to QA and testing. for testing to get equal importance. In such cases, if an According to our survey, 51% of TME respondents (45% organization does not have a well-defined QA methodology across all industries) said that they were leveraging analytics that enables them to understand the demands of testing and AI for intelligent automation. Given the time pressure on in an agile framework, it can put quality at risk and lead to QA and test activities, this is a very welcome trend, for the use a speed versus quality dynamic, which can prove disastrous. of AI can help define the scope of testing, make sure testing This is one of the biggest challenges for QA and testing when is focused on the right areas, and reduce false positives, thus it comes to the adoption of agile. reducing the time, effort, and cost of QA and testing. When asked about the challenges in applying testing to agile development, TME respondents gave the highest weighting Test data and environment challenges to “a lack of appropriate test environment and data” (58% One of challenges that kept coming up throughout the of respondents), followed by “the difficulty in slicing test survey results was related to test data and environments. For activities for more than one location for distributed Agile” instance, when asked about the challenges in achieving their (56%), “early involvement of the test team in the inception desired level of test automation, TME respondents gave the phase or sprint planning” (53%), and “a lack of professional highest weighting to “our applications change too much with test expertise in agile teams” (53%).
65 Sector Analysis 65 every release” (69% of respondents), followed by, “challenges environments”, followed by 59% (54% across all sectors) who pointed to “the cost of test environments,” and 45% (38% with the test data and environment availability and stability” across all sectors) who indicated “the lack of facilities to book (53% of respondents). While the first can be explained by the and manage your own environments.” Similarly, when asked dynamic delivery environment in the TME sector, the second about the challenges in managing test data, 68% of TME has to do with the number and complexity of applications and respondents (versus 61% across sectors) said, “maintaining associated environments. test data consistency across different systems under test,” The degree of these challenges came through clearly in followed by 61% (56% across sectors) who said, “managing our survey results. For instance, 63% of TME respondents the size of test data sets,” and 60% (50% across sectors) who (versus 54% across sectors) said they faced challenges said, “having to maintain the right test data set versions with arising from “defects due to inaccurate configuration of test different test versions.” Summary The TME sector is going through a period of rapid change and we can see the impact of this on QA and testing. The need to be more responsive, flexible, and customer-centric is driving an increasing number of organizations to adopt cloud, agile and DevOps. In addition, there are a number of challenges around test data and environments, skill sets, and the pressure brought to bear on QA and testing in this competitive environment. Over the next two to three years, the challenge before TME organizations is going to be how to remain competitive while investing hugely in upgrading and re-platforming. For QA and testing to perform, the most important factor will be managing the cultural change that is required by frameworks such as agile and DevOps. Challenges related to skill sets, test data, and environments as well as the “speed-over-quality” mindset will need to be tackled and testing will need to find its natural place in agile and DevOps teams. Technologies such as automation, analytics, and AI hold a lot of promise and are likely to revolutionize QA and testing in the future.
66 World Quality Report 2018 –19 66 About the study
67 About the study 67 67 subset of these questions depending on the interviewee’s is based on research The World Quality Report 2018-19 role in the organization. The quantitative research study was findings from 1,700 interviews carried out during April and complemented by additional in-depth interviews to provide May 2018 using CATI (Computer Aided Telephone Interviews). greater insight into certain subject areas and to inform the The average length of each interview was 30 minutes and analysis and commentary. The main themes for all survey the interviewees were all senior executives in corporate IT questions remained the same, though a few objective management functions, working for companies and Public first time this year. responses were also added for the Sector organizations across 32 countries. Quality measures were put in place to ensure the This year, the interviews were based on a questionnaire of questionnaire was understood, answered accurately and 41 questions, with the actual interview consisting of a completed in a timely manner by the interviewee. Survey Sample The required sample size varies depending on the population For this year’s research, we selected only organizations with it represents – usually expressed as a ratio or incidence rate. more than 1,000 employees (in the respondent’s national In a business-to-business (B2B) market research study, the market) – an approach used for the last three years to average recommended sample size is 100 companies. This is provide us with valid trending data. lower than the average sample size used for business-to- consumer (B2C) market research because whole esearch participants were selected so as to organizations are being researched, rather than individuals. R suf ficient coverage of different ensure This year, the B2B market research conducted for the World regions and vertical markets to provide industry Quality Report is based on a sample of 1,700 interviews speci fic insight issues into the QA and testing from enterprises with more than 1,000 employees (26%), within each sector. organizations with more than 5000 employees (34%) and companies with more than 10000 employees (40%). The approach and sample size used for the research this year With the inclusion of product heads/CTO for the third time, enables direct comparisons of the current results to be we are able to bring in their views and insights in the space made with previous research studies conducted for the of Product and Engineering Services (P&ES) for Automotive, report, where the same question was asked. HealthCare and Life Sciences, and High-Tech Sector. During the interviews, the research questions asked of each The research sample consists mainly of senior-level IT participant were linked to the respondent’s job title and the executives as shown in Figure 33. answers he/she provided to previous questions where applicable. For this reason, the base number of respondents To ensure a robust and substantive market research study, for each survey question shown in the graphs is not always the recruited sample must be statistically representative of the full 1,700 sample size. file. the population in terms of its size and demographic pro Questionnaire and Methodology The survey questionnaire was devised by QA and Testing experts in Capgemini, Sogeti and Micro Focus (sponsors of the research study), in consultation with Coleman Parkes Research. The 41 question survey covered a range of QA and Testing subjects, enriched by qualitative data obtained from the additional in-depth interviews. The quotations shown in the report are taken from these in-depth interviews.
68 2018 –19 World Quality Report 68 Interviews by sectors Figure 31 Financial Services industry, including Healthcare and Life Sciences Capital Markets, Banking and Insurance 8% 19% Automotive Public Sector/Government 15% 8% Energy, Utilities, Telecommunications, Media and Chemicals and Entertainment 7% 13% Consumer Products and Retail/ Manufacturing Distribution and Logistics 6% 10% High-tech, including hardware Transportation vendors + Aerospace and Defence 6% 8% by job title Interviews Figure 33 27% CIO IT Directors 22% QA/Testing Manager 20% 18% VP Applications 7% CMO/CDO 6% CTO/Product Head
69 About the study 69 Interviews by Region Figure 32 170 0 To t a l 299 Western Europe 120 175 Southern EU Nordics 92 Eastern EU 345 85 North America 60 China & 50 Middle East Asia Hong Kong Japan 130 79 BeNeLux Brazil 25 Singapore 90 Australia & 150 New Zealand: UK and Ireland 32 Countries # of respondents Country # of respondents Country # of respondents Country Portugal 35 10 USA 280 New Zealand 149 32 Czech Republic Qatar 10 France Dubai 10 Hungary 30 Germany 130 UK 125 10 Abu Dhabi Finland 30 100 30 Netherlands Saudi Arabia 10 Belgium and Luxembourg Norway Sweden 85 Jordan & Bahrain 5 30 Denmark 30 Australia 80 Brazil 79 Poland 30 Hong Kong 25 Canada 65 China 60 Singapore 25 Italy 50 Ireland 25 Switzerland 20 Japan 50 Spain 35 15 UAE (excluding Dubai, Abu Dhabi)
70 2018 –19 World Quality Report 70 About the Sponsors About Capgemini and Sogeti About Micro Focus Headquartered in Newbury, United Kingdom, Micro Focus A global leader in consulting, technology services and digital is a leading global enterprise software company uniquely transformation, Capgemini is at the forefront of innovation positioned to help customers extend existing investments to address the entire breadth of clients’ opportunities in while embracing new technologies in a world of Hybrid the evolving world of cloud, digital and platforms. Building IT – from mainframe to mobile to cloud. With operations on its strong 50-year heritage and deep industry-specific expertise, Capgemini enables organizations to realize their worldwide, and over 40 years of software experience, Micro Focus helps customers solve the most complex technology business ambitions through an array of services from strategy problems through the delivery of world-class, enterprise- to operations. Capgemini is driven by the conviction that the scale solutions in key areas including Hybrid IT Management, business value of technology comes from and through people. It is a multicultural company of 200,000 team members in Enterprise DevOps, Security & Risk Management, and over 40 countries. The Group reported 2017 global revenues Predictive Analytics. of EUR 12.8 billion. For more information, visit www.microfocus.com. Sogeti is a leading provider of technology and engineering services. Sogeti delivers solutions that enable Digital Transformation and offers cutting-edge expertise in Cloud, Cybersecurity, Digital Manufacturing, Digital Assurance & Testing, and emerging technologies. Sogeti combines agility and speed of implementation with strong technology supplier partnerships, world class methodologies and its global delivery model, Rightshore®. Sogeti brings together more than 25,000 professionals in 15 countries, based in over 100 locations in Europe, USA and India. Sogeti is a wholly- owned subsidiary of Capgemini SE, listed on the Paris Stock Exchange. Learn more about us at: www.capgemini.com/testing or www.sogeti.com/testing
71 WORLD QUALITY REPORT 2017–18 Thank you Thank you Capgemini, Sogeti and Micro Focus would like to thank The 1,660 IT executives who took part in the research study *Ian Parkes, CEO and co-founder of Coleman Parkes Capgemini, Sogeti and Micro Focus would like to thank Research, is a full member of the Market Research Society. this year for their time and contribution to the report. In All research carried out by Coleman Parkes Research is accordance with the UK Market Research Society (MRS) *Ian Parkes, CEO and co-founder of Coleman Parkes The 1,700 IT executives who took part in the research study conducted in compliance with the Code of Conduct and Code of Conduct (under which this survey was carried out) Research, is a full member of the Market Research Society. this year for their time and contribution to the report. In guidelines set out by the MRS in the UK, as well as the legal the identity of the participants in the research study and accordance with the UK Market Research Society (MRS) All research carried out by Coleman Parkes Research is their responses remain confidential and are not available obligations under the Data Protection Act 1998. Code of Conduct (under which this survey was carried out) conducted in compliance with the Code of Conduct and to the sponsors. the identity of the participants in the research study and guidelines set out by the MRS in the UK, as well as the legal www.worldqualityreport.com their responses remain confidential and are not available to obligations under the Data Protection Act 1998. the sponsors. All the business leaders and subject matter experts who provided valuable insight into their respective areas of ©2017 Capgemini, Sogeti and Micro Focus. All All the business leaders and subject matter experts who expertise and market experience, including the authors of Rights Reserved. provided valuable insight into their respective areas of www.worldqualityreport.com country and industry sections and subject-matter experts expertise and market experience, including the authors of ©2018 Capgemini, Sogeti and Micro Focus. All Rights Reserved. from Capgemini, Sogeti and Micro Focus. Capgemini and Micro Focus, and their respective marks and country and industry sections and subject-matter experts from Capgemini, Sogeti and Micro Focus. logos used herein, are trademarks or registered trademarks Capgemini and Micro Focus, and their respective marks and Main Report Authors of their respective companies. All other company, product logos used herein, are trademarks or registered trademarks Mark Buenen and Govind Muthukrishnan and service names mentioned are the trademarks of their Main Report Authors of their respective companies. All other company, product respective owners and are used herein with no intention Mark Buenen and Ajay Walgude and service names mentioned are the trademarks of their ® Writer for Main Chapters is a trademark of trademark infringement. Rightshore respective owners and are used herein with no intention ® ® ® Lindsay Clark , TPI and , TMap NEXT belonging to Capgemini. TMap Writer for Main Chapters of trademark infringement. Rightshore® is a trademark ® are registered trademarks of Sogeti, part of the TPI NEXT Rahul Mitra belonging to Capgemini. TMap®, TMap NEXT®, TPI® and Writer for country pullouts Capgemini Group. TPI NEXT® are registered trademarks of Sogeti, part of the Writer for country pullouts Rahul Mitra Capgemini Group. Jayant Kumar No part of this document may be reproduced or copied in No part of this document may be reproduced or copied in Program Director any form or by any means without written permission from Program Manager any form or by any means without written permission from Julian Clarke Capgemini and Micro Focus. Balaji Narasimhan Capgemini and Micro Focus. Project Manager Program Manager Archit Revandkar Balaji Narasimhan Partner Management Project Manager Malcolm Isaacs and Christine Ewing Archit Revandkar Content Proof Reading Monica Kwiecinski Partner Management John Jeremiah (Micro Focus) Market Research Stephen Saw, Rachel Leafe and Ian Parkes Content Proof Reading (Coleman Parkes Research)* Charles Kronauer Creative Design Monica Kwiecinski Palash Naskar, Ajoy Das and Rakesh Biswas Market Research Printing and Distribution Stephen Saw, Rachel Robinson and Ian Parkes David Cole and Gerry Court (Crucial Colour) (Coleman Parkes Research)* Creative Design Partha Karmakar Printing and Distribution David Cole and Gerry Court (Crucial Colour) 74
72 Micro Focus Malcolm Isaacs ADM Solutions Marketing Manager [email protected] Christine Ewing Senior Director, Product Marketing, [email protected] Sogeti Sathish Natarajan Vice President - Digital Assurance & Testing, Sogeti, Capgemini Group [email protected] Mark Buenen Vice President, Global Leader, Digital Assurance and Testing Practice [email protected] Capgemini Ramesh Mahadevan Anand Moorthy Vice President, Testing Leader, Senior Director, Testing Leader, Financial Services, North America Continental Europe [email protected] [email protected] Ajay Walgude Shyam Narayan Head of Managed Services, Australia Vice President, Financial Services, and New Zealand Testing Leader [email protected] [email protected] Sanjeev Deshmukh Dhiraj Sinha Vice President, Testing Leader, Vice President, Financial Services- Testing, Asia Pacific North America [email protected] [email protected]
201 8 Fourth National Report on Human Exposure to Environmental Chemicals U pdated Tables, March 2018 , Volume OneMore info »
2018 Physical Activity Guidelines Advisory Committee Scientific Report To the Secretary of Health and Human ServicesMore info »