{"id":1476,"date":"2016-01-08T14:07:06","date_gmt":"2016-01-08T14:07:06","guid":{"rendered":"https:\/\/blogs.gov.scot\/digital\/?p=1476"},"modified":"2017-07-20T09:36:25","modified_gmt":"2017-07-20T09:36:25","slug":"our-approach-to-testing","status":"publish","type":"post","link":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/","title":{"rendered":"Our approach to testing"},"content":{"rendered":"<p><em>This is a post by <a href=\"https:\/\/twitter.com\/rajesh1556\">Rajesh Yarlagadda<\/a> and David Vidal, our\u00a0test engineers<\/em>.<\/p>\n<p>This post is the third\u00a0post in our series covering how we approach <a href=\"http:\/\/inside.mygov.scot\/2015\/11\/04\/how-we-approach-continuous-delivery\/\">continuous delivery<\/a>,\u00a0covering our approach to testing.<\/p>\n<p>Testing is an essential component of each and every phase of the mygov.scot development process, with quality being \u201cbaked in\u201d to the product at every stage. This blog post provides an overview of how we approach testing, and some of the tools we use. Feel free to get in touch if you\u2019d like to know a bit more about a particular technology or our reasons behind using it.<\/p>\n<p>After creating a new feature to meet a particular user need, our developers submit changes to our version control system \u2013 this can include source code, environment modifications and configuration files.<\/p>\n<p>Following a commit to the version control system, the continuous integration\u00a0server will do the following:<\/p>\n<ul>\n<li>Trigger a build<\/li>\n<li>Run automated unit tests<\/li>\n<li>Perform sonar code quality check<\/li>\n<li>Automatically deploy the successful build to a development environment<\/li>\n<\/ul>\n<p><a href=\"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png\"><img decoding=\"async\" class=\"alignnone wp-image-1475 size-full lazyload\" data-src=\"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png\" alt=\"\" width=\"700\" height=\"759\" data-srcset=\"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png 700w, https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process-277x300.png 277w, https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process-472x512.png 472w, https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process-387x420.png 387w\" data-sizes=\"(max-width: 700px) 100vw, 700px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 700px; --smush-placeholder-aspect-ratio: 700\/759;\" \/><\/a><\/p>\n<p><!--more--><\/p>\n<p>The test engineers manually promote the successful build to the integration test environment using an automated deployment job. The integration test environment is designed to run all automated tests i.e. accessibility tests, End to End (E2E) tests and browser layout tests. If a feature change completes all\u00a0automated tests with no failures then it will be promoted to the exploratory test environment.<\/p>\n<p>The exploratory test environment is designed to run session-based exploratory testing and browser testing on multiple devices using the \u2018Ghostlabs\u2019 software. On completion of successful exploratory testing, feature changes will be promoted to the performance test environment.<\/p>\n<p>The performance test environment is designed to run performance tests i.e. smoke tests, load tests, stress tests, soak tests and security tests. On completion of testing, the feature changes will be demonstrated to the product owner and other stakeholders.<\/p>\n<ul>\n<li>Smoke \u2013 check to see that there has been no significant drop in performance between builds<\/li>\n<li>Load \u2013 check that the site can handle an acceptable level of users (currently set to 1,000)<\/li>\n<li>Stress \u2013 check that the site can either handle or break gracefully for an extended number of users (currently set to 5,000 users)<\/li>\n<li>Soak \u2013 check that there are no memory leaks over a longer period (currently set to a day)<\/li>\n<\/ul>\n<p>On completion of the demo, testers record the component versions and save this as the \u2018last known good version\u2019, after which relevant artefacts are pushed to the build repository server.<\/p>\n<p>Every Monday, the mygov.scot team will create either a blue\/green off-line production environment using the \u2018last known good version\u2019. Smoke testing is carried out on the off-line environment before it is switched\u00a0over to being the live, production version.<\/p>\n<h2>Types of test<\/h2>\n<p>There are many different types of test we perform \u2013 the starting point is the user story, written for each new feature before work begins. Before\u00a0a feature is even validated against the user acceptance criteria we validate against system quality attributes \u2013 performance, accessibility and security. Once we complete these tests the next stage is exploratory testing, to validate the story.<\/p>\n<p>We look to ensure that we have:<\/p>\n<ul>\n<li>Complete functionality on a range of\u00a0different browsers. This includes Internet Explorer (versions 8 to 10 running on Windows 7), Chrome (running on Ubuntu), Safari (on Mac), native browsers on iOS devices (versions 6 to 9) and native browsers on Android (versions 2.2 \u2013 5.1).<\/li>\n<li>HTML code\u00a0in line with WCAG 2AA accessibility standards, tested via the open source tool\u00a0Pa11y<\/li>\n<li>A good level of readability, tested via ChromeVOX<\/li>\n<li>A secure site through automated security tests performed by\u00a0ZAP, which scans and validates the site against the top 10 attacks (as identified by\u00a0OWASP)<\/li>\n<li>Consistent\u00a0visual standards and UX design, tested via the\u00a0GALEN framework layout testing which\u00a0captures an image of the page if elements of the page\u00a0are not where they should be.<\/li>\n<\/ul>\n<h2>Improvements<\/h2>\n<p>Testing takes time \u2013 performance tests run in the background and can take up to a day to run per test,\u00a0manual tests take several\u00a0hours per day. With every new test added the quality of our work goes up, with every test automated or moved to an earlier stage in the process we can spot and correct issues earlier. We\u2019re working to include\u00a0performance testing as part of our continuous integration server process as this will allow us to spot\u00a0code which has the potential to slow the site down and correct it much earlier.<\/p>\n<p>Security testing is currently more reliant on manual processes\u00a0and is something we are also working to increase the automation on. We are also looking to expand the number of devices and browsers that we test on \u2013 we cover a good number already and we are looking to up that\u00a0number.<\/p>\n<p><em>Follow the team via <a href=\"https:\/\/twitter.com\/mygovscot\">@mygovscot<\/a> on Twitter for more updates. Want to comment? Let us know below!<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This is a post by Rajesh Yarlagadda and David Vidal, our\u00a0test engineers. This post is the third\u00a0post in our series covering how we approach continuous delivery,\u00a0covering our approach to testing. Testing is an essential component of each and every phase of the mygov.scot development process, with quality being \u201cbaked in\u201d to the product at every&#8230;<\/p>\n","protected":false},"author":401,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,10],"tags":[70],"class_list":["post-1476","post","type-post","status-publish","format-standard","hentry","category-digital-public-services","category-mygov-scot","tag-technology-digital-architecture"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Our approach to testing - Digital<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Our approach to testing - Digital\" \/>\n<meta property=\"og:description\" content=\"This is a post by Rajesh Yarlagadda and David Vidal, our\u00a0test engineers. This post is the third\u00a0post in our series covering how we approach continuous delivery,\u00a0covering our approach to testing. Testing is an essential component of each and every phase of the mygov.scot development process, with quality being \u201cbaked in\u201d to the product at every...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"Digital\" \/>\n<meta property=\"article:published_time\" content=\"2016-01-08T14:07:06+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-07-20T09:36:25+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png\" \/>\n\t<meta property=\"og:image:width\" content=\"700\" \/>\n\t<meta property=\"og:image:height\" content=\"759\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"craigmilligan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"craigmilligan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/\"},\"author\":{\"name\":\"craigmilligan\",\"@id\":\"https:\/\/blogs.gov.scot\/digital\/#\/schema\/person\/ef1ff8acf7c47992d04cea9ca3f9b3e9\"},\"headline\":\"Our approach to testing\",\"datePublished\":\"2016-01-08T14:07:06+00:00\",\"dateModified\":\"2017-07-20T09:36:25+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/\"},\"wordCount\":806,\"commentCount\":2,\"image\":{\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png\",\"keywords\":[\"Technology &amp; Digital Architecture\"],\"articleSection\":[\"Digital Public Services\",\"mygov.scot\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/\",\"url\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/\",\"name\":\"Our approach to testing - Digital\",\"isPartOf\":{\"@id\":\"https:\/\/blogs.gov.scot\/digital\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png\",\"datePublished\":\"2016-01-08T14:07:06+00:00\",\"dateModified\":\"2017-07-20T09:36:25+00:00\",\"author\":{\"@id\":\"https:\/\/blogs.gov.scot\/digital\/#\/schema\/person\/ef1ff8acf7c47992d04cea9ca3f9b3e9\"},\"breadcrumb\":{\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#primaryimage\",\"url\":\"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png\",\"contentUrl\":\"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png\",\"width\":700,\"height\":759},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blogs.gov.scot\/digital\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Our approach to testing\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blogs.gov.scot\/digital\/#website\",\"url\":\"https:\/\/blogs.gov.scot\/digital\/\",\"name\":\"Digital\",\"description\":\"Updates from the Scottish Government&#039;s Digital Directorate\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blogs.gov.scot\/digital\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blogs.gov.scot\/digital\/#\/schema\/person\/ef1ff8acf7c47992d04cea9ca3f9b3e9\",\"name\":\"craigmilligan\",\"url\":\"https:\/\/blogs.gov.scot\/digital\/author\/craigmilligan\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Our approach to testing - Digital","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/","og_locale":"en_GB","og_type":"article","og_title":"Our approach to testing - Digital","og_description":"This is a post by Rajesh Yarlagadda and David Vidal, our\u00a0test engineers. This post is the third\u00a0post in our series covering how we approach continuous delivery,\u00a0covering our approach to testing. Testing is an essential component of each and every phase of the mygov.scot development process, with quality being \u201cbaked in\u201d to the product at every...","og_url":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/","og_site_name":"Digital","article_published_time":"2016-01-08T14:07:06+00:00","article_modified_time":"2017-07-20T09:36:25+00:00","og_image":[{"width":700,"height":759,"url":"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png","type":"image\/png"}],"author":"craigmilligan","twitter_card":"summary_large_image","twitter_misc":{"Written by":"craigmilligan","Estimated reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#article","isPartOf":{"@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/"},"author":{"name":"craigmilligan","@id":"https:\/\/blogs.gov.scot\/digital\/#\/schema\/person\/ef1ff8acf7c47992d04cea9ca3f9b3e9"},"headline":"Our approach to testing","datePublished":"2016-01-08T14:07:06+00:00","dateModified":"2017-07-20T09:36:25+00:00","mainEntityOfPage":{"@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/"},"wordCount":806,"commentCount":2,"image":{"@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png","keywords":["Technology &amp; Digital Architecture"],"articleSection":["Digital Public Services","mygov.scot"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/","url":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/","name":"Our approach to testing - Digital","isPartOf":{"@id":"https:\/\/blogs.gov.scot\/digital\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#primaryimage"},"image":{"@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#primaryimage"},"thumbnailUrl":"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png","datePublished":"2016-01-08T14:07:06+00:00","dateModified":"2017-07-20T09:36:25+00:00","author":{"@id":"https:\/\/blogs.gov.scot\/digital\/#\/schema\/person\/ef1ff8acf7c47992d04cea9ca3f9b3e9"},"breadcrumb":{"@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#primaryimage","url":"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png","contentUrl":"https:\/\/blogs.gov.scot\/digital\/wp-content\/uploads\/sites\/5\/2017\/07\/testing-process.png","width":700,"height":759},{"@type":"BreadcrumbList","@id":"https:\/\/blogs.gov.scot\/digital\/2016\/01\/08\/our-approach-to-testing\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blogs.gov.scot\/digital\/"},{"@type":"ListItem","position":2,"name":"Our approach to testing"}]},{"@type":"WebSite","@id":"https:\/\/blogs.gov.scot\/digital\/#website","url":"https:\/\/blogs.gov.scot\/digital\/","name":"Digital","description":"Updates from the Scottish Government&#039;s Digital Directorate","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blogs.gov.scot\/digital\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/blogs.gov.scot\/digital\/#\/schema\/person\/ef1ff8acf7c47992d04cea9ca3f9b3e9","name":"craigmilligan","url":"https:\/\/blogs.gov.scot\/digital\/author\/craigmilligan\/"}]}},"_links":{"self":[{"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/posts\/1476","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/users\/401"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/comments?post=1476"}],"version-history":[{"count":0,"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/posts\/1476\/revisions"}],"wp:attachment":[{"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/media?parent=1476"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/categories?post=1476"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gov.scot\/digital\/wp-json\/wp\/v2\/tags?post=1476"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}