Gartner Symposium. I always enjoy it. Of course it’s a deluge of buzzwords and bombast, but there is substance beneath, and I enjoy thinking about the implications for teams creating software in the future.
This morning I was listening to a talk from someone in the Oil and Gas industry who has actually deployed machine-to-machine, i.e. what everyone talks about as the future of Internet-of-Things but very few have any practical experience of. It was an interesting talk. Read more…
We have just released a new whitepaper about how to set up a successful test automation project based on our experience of working with hundreds of companies to successfully deploy test automation
Test automation can deliver huge benefits in terms of time-to-market, quality, productivity, and auditability to almost any team creating or deploying software. For example, TestPlant has worked with a leading global news publisher to reduce their app update cycle from three weeks to two days, a major UK bank to reduce post-release defects by 65%, and one of the world’s top five retailers to double the number of apps they are delivering without increasing the size of their test team. These stories are not uncommon – test automation really has delivered amazing benefits to lots of companies. Read more…
A lack of environment management is one of the most common reasons for unreliable test execution. People remove devices from the test environment, change app versions without notification, change OS settings, two tests try to use the same devices at the same time, a manual tries to use the same device at the same time, a backup runs on the middle of a test run, insufficient user data is provided, and the list goes on. Read more…
- In Testing
On March 24, 2016 we co-hosted a webinar with Sharif from GameBench. To watch the full webinar recording, please visit our webinar archive.
Yesterday Sharif, Max, I had a rehearsal of the webinar and it was exciting! Max immediately started applying GameBench to various customer projects he’s working on. I love breaking down big vague problems and into clear concrete achievable steps; and that’s exactly what GameBench does for UX.
UX is always presented in a vague mysterious way. It’s not defined in technical terms (as functionality and performance are), it’s defined in terms of vague human response which as we know is subjective, volatile, and differs from person-to-person. So if asked to test UX most of us wouldn’t really know where to start. Should we hire a focus group? Read more…
Most of the major innovations in mobile devices have involved every layer of the technology stack. Cameras required hardware changes, operating system support, middleware changes, and new applications. E-mail was the same; and so was navigation. They were not simply new apps at the top layer, they were slices through the whole technology stack. Read more…
Yesterday I was rather disparaging about 5G. It’s only natural. Over the years I’ve watched the announcements of EDGE/3G/LTE and it’s always been the same story… it starts with a promise of infrastructure changes that will revolutionise business models and human engagement… and it ends with web browsing being a bit faster (which is nice). Surely 5G will be the same, it will improve bandwidth, it promises to finally improve latency (which would be really nice), but innovation is going to happen at the device/application level. Read more…
The TestPlant team have arrived in Barcelona for Mobile World Congress! It’s been a busy Monday on our stand and it’s interesting to see the much higher priority quality and testing have for digital companies compared to when we first started coming to MWC 5 years ago.
But I’ve managed to get away from our stand for a quick tour around the show and wanted to share five themes… Read more…
A few weeks ago I presented a webinar along with Paul Gerrard about “digital testing” which was based on some research by the analyst firm PAC consulting. It’s a very interesting piece of research and if you haven’t seen it you can download it from our website. One of the key highlights of this research for me was the graph below which clearly shows that the levels of test automation are still worryingly low; only 8% of companies have test automation for more than 50% of their digital processes; and 67% have less than 30%. After all these years that’s not good. Read more…
Gartner’s Magic Quadrant (MQ) for Software Test Automation was published this week, and TestPlant are proud to again be recognised as a “visionary” (you can download the full report from our website here).
Small vendors (like TestPlant) have created a mid-tier
of innovative and effective test tools
But one thing that struck me as I read the new MQ is the relative stability we are now seeing in the test tools market. Five years ago IBM, HP, and Borland completely dominated the market with a large number of very small and constantly changing vendors. Read more…
- In Testing
There is an old saying that “necessity is the mother of all invention”; and I suppose the corollary is that “the devil makes work for idle hands”. These sayings went racing through my head recently as I sat in a testing event. People were explaining what they wanted to get out of the day when a test architect said “We know we need a ‘framework’, but we don’t know what it needs to do, so we’re hoping to find out”.
“Keep It Simple, Stupid”
‘Frameworks’ are a cancer running through the testing industry. Perhaps that’s a little unfair since there are many helpful frameworks that increase productivity and reliability for their teams. But the vast majority of frameworks I’ve ever seen simply take time to create and maintain, increase the time it takes to create and maintain test scripts, and reduce the reliability of test execution. I’ve lost count of the number of times I’ve suggested someone make a very simple change to how their tests execute only to be told “it would take days to update the framework”, or that someone hasn’t automated the testing of certain features yet because “we haven’t add that to the framework”, or that a failure in the connectivity of long-running tests turned out to be a bug in the framework. Read more…
Join TestPlant CTO, Antony Edwards, and Principal at Gerrard Consulting, Paul Gerrard for a webinar – where they will discuss the strategies and challenges in digital testing. Register now.
‘Digital business’ is genuinely transforming companies in every sector, from retail and banking to manufacturing and automotive. Testing is an essential part of every digital transformation. Without the right ‘digital’ testing companies can’t get the high-quality user experiences or the frequency of releases they need to engage and retain customers/users. How can you release daily across 40+ platforms with high-quality without an efficient and effective testing process?
So TestPlant teamed up with PAC (Pierre Audoin Consultants) to understand how test teams are really being impacted by ‘digital transformation’ (is it real or just talk), the challenges they are facing, and how they are meeting those challenges. We surveyed over 200 companies across Europe, primarily CIOs and heads of testing, across a range of sectors. The full report is freely available here; but I also wanted to highlight my top three take-aways. Read more…
NOTE: Over the next few weeks we will be hosting several webinars to demonstrate how we use eggPlant Functional to test set-top boxes. These webinars will be relevant to anyone interested in testing IoT v1 ‘things’ or any other connected non-standard devices so please register now!
‘How will we test the Internet-of-Things’ is a popular topic these days at testing events. The discussion quickly ends up talking about how to test ad-hoc mesh networks of intelligent interacting devices, when self-driving cars start making moral choices in life threatening situations, and sensors in coffee cups start tweeting. Fun discussions, but a million miles from anything that any testers are really going to be working on in the next few years. It’s easy to become cynical about the Internet-of-Things (IoT) as just another buzz topic. Read more…
Imagine you went to a cafe and ordered a coffee, but the person behind the counter told you that actually they were going to make you a peppermint tea instead. You thank them for their suggestion, but confirm that you’re sure you really would like a coffee. They then explain to you that making a coffee is much harder for them, their machine isn’t working, they’ve run out of milk, and they really prefer the smell of peppermint. What would you do (this is not a trick question)? Now hold that thought… Read more…
We will be hosting several webinars over the next few weeks demonstrating how to use the eggPlant range for application-level monitoring. Please register now! Edit: This webinar was hosted in September 2015. Watch the recording here.
About 12 months ago I had a very confusing meeting with a prospective customer. It just wasn’t adding up. I’d been told that the topic was functional test automation, but it often felt more like a load testing discussion, they weren’t very interested in script creation, and their questions were very unusual. Then suddenly it all cleared up! They weren’t testers and they weren’t interested in testing an application, they were an operations team and they wanted to use eggPlant Functional to monitor their production services from the user’s perspective. A year later and we now have several customers (banking, retail, media) using eggPlant Functional this way.
But why are people using test tools for monitoring? Read more…
Omni-channel (or multi-channel) and mobile are the two key IT challenges facing digital retailers today; and all retailers are now digital retailers. We’ve been working closely with several major US and UK retailers over the past year on the testing of their omni-channel and mobile initiatives, and over the next few weeks will be hosting several webinars to show how the eggPlant range helped them deliver successfully. Please register now! Edit: The webinar associated this post was hosted in July 2015. A recording of this webinar can be watched here.
Online retail sales in the US are expected to reach $334B in 2015 (approximately 10% of all retail sales) and to continue to grow at 10% CAGR for the next five years (Forrester). Forrester also predict that in the next few years Read more…
SciFi worlds from the 1980s always included super powerful, intelligent, computers that you could talk to. By the late 90s the vision had moved on to worlds where everything is a computer and connected; maybe even us! And it’s this ubiquitous computing that I think is the most exciting and interesting part of technology. Read more…
Over the next few weeks we will be hosting several webinars to show how we used eggPlant software testing tools to address these challenges in major US and UK banks. Please register now! Edit: This webinar aired in June 2015. Watch the recording of this webinar here.
Consumer banking is now a digital business. Over 65% of people in the USA and UK do all their banking on-line and over 40% regularly use mobile banking (based on data from Pew Research Center). Earlier this year Charter Savings Bank opened as the UK’s first digital-only bank. So while “digital transformation” is one of 2015’s leading buzzwords, it seems to me that Read more…
Retail is increasingly web and mobile; on-line sales totalled $300B in the U.S. and $285B in the EU. Retailers’ web sites and apps are now absolutely mission critical. They must be user-friendly, fast, provide the right functionality, and work! Every time in any environment. My favourite data point for the on-line retail industry was published by Amazon – a 100ms increase in the response time of their website reduces revenue by 1%. Quality directly impacts revenue. Read more…
On Friday we conducted the first in a series of 4 webinars that we’re running about the challenges of mobile testing and how eggPlant tools can help you address those challenges.
We decided to run this series of 4 webinars because, like many of you, we hear a lot of presentations about the difficulties of mobile testing, but very few of these talk about the real issues that effect us in our own testing, and that we hear about from our customers. These presentations tend to focus on things like low-power mode or threading, which are genuine technical differences between mobiles and PCs, but they aren’t the issues we see impacting test teams. Read more…
Everyone talks about “the challenges of mobile testing”, but few people actually describe clearly what these challenges are, and even fewer give you tangible practical advice on how to address these challenges. Over the next 4 weeks we will be running a series of webinars clearly describing the major challenges impacting teams creating mobile apps, and show how eggPlant tools can help you address them. Please register NOW for these webinars using the links below. Edit: These webinars were hosted in April of 2015. Click the links at the end of this post to watch the webinar recordings.
But why the lack of clarity around mobile testing challenges? I believe it is because people assume that these challenges must be inherent technical differences between mobile devices and PCs, so they look for differences of this type, and end up talking about low-power mode and threading differences. But these aren’t really causing problems for the vast majority of testers. Read more…
“Mobile World Congress – 10 years in Barcelona” greeted us at the airport. Inevitably it made us think about how much ‘mobile’ has changed in the last 10 years, but it also made us think how little MWC had changed in the last 5 years. Since “convergence” around 2006 and the release of the first iPhone it feels like MWC has been searching for “the next BIG THING” without much success. And while commercially it’s always been a great show for TestPlant, some recent MWCs have certainly lacked that industry buzz that was so intoxicating in the years leading up to 2006.
But this year the buzz was back! It’s difficult to put my finger on one BIG THING, but the buzz was definitely back, and I’m glad. Here are my key take-aways Read more…
InterConnect is IBM’s new customer and partner event running for the first time in 2015. It is the consolidation of three existing IBM events – Rational ‘Innovate’ (which TestPlant has been supporting for years), ‘Pulse’ for cloud computing, and the Websphere ‘Impact’ event. So 20,0000 people in Las Vegas talking about all things IBM.
So what are they talking about and what aren’t they talking about? Here are my 4 takeaways from the event so far. Read more…
The traditional approach to performance and stress testing is:
(1) Define objective
(2) Create test
(3) Run test
(4) Analyse results
(5) Go back to #2 and repeat until clear conclusion
But modern load testing tools such as eggPlant Performance have rich ‘dynamic test control’ functionality which allows the tester to change the test at run-time. For example Read more…
The most famous and over used graph in the testing world is the one below. In everyone’s mind it simply says that the earlier you find defects in the lifecycle the cheaper they are to fix.
Most test process change project proposals start with this graph and really use it to justify the majority of their recommendations. I never find such proposals convincing and over the years have identified four reasons why not. Read more…
Almost any discussion about software development process could be improved by clearly separating the questions – who, what, and when. That may seem ridiculously obvious, but I can’t remember the last discussion I had where someone didn’t assume the answer to one of these questions from the answer to another (most commonly assuming “who” and “what” from “when”). Read more…
“Our requirements are chaos”. It’s something I’m sure you’ve all heard and many of you have said. An exclamation of frustration and self-deprecating admission. But it’s only this morning that I realised it’s also a correct statement of fact. Requirements are indeed chaotic in the mathematical sense, that is a small variation to the inputs (i.e. the requirements) can lead to a dramatic change in the outputs (i.e. the product). This is sometimes called the “butterfly effect” since theoretically a butterfly beating its wings at a certain moment could cause a hurricane on the other side of the world (weather is the most studied chaotic system). Read more…
It’s the holiday season which means it’s the retail season! I’m definitely not much of a shopper, but even I’ve been on-line buying all kinds of Arduino gadgets to give to my friends. And as I see facts such as “Black Friday on-line sales up 17%” I naturally start thinking about how many on-line retailers are doing proper load and performance testing and how many are just hoping everything is fine. I suspect that fewer than 10% are really doing load/performance testing.
It’s a bit of a mystery to me why so few on-line retailers (and other digital businesses) do proper load/performance testing. I can only assume that they don’t believe it’s really that important or sensitive. That as long as the performance is “OK” when they randomly try it one day then optimising the performance isn’t going to translate into more sales. But if this is the case then they are wrong. Very wrong. Read more…
- In Testing
One of my standard sayings these days is that “there’s no such thing as a mobile application”. What I’m getting at is that (apart from games) there are almost no mobile apps that deliver interesting functionality to users by themselves. They are all part of a distributed application that typically comprises a mobile component, a desktop and/or web client component, an administrative component, a database, etc. Nationwide, Bloomberg, and the Financial Times are all popular mobile apps that can only be fully tested in conjunction with non-mobile components. So I think we have lots of mobile software, but very few mobile-only applications (and actually very few desktop-only applications for that matter). In our multi-channel, collaborative, social world it simply must be this way. Read more…
A lot of companies I talk to are “shifting left”. This is primarily driven by the question of how test fits into agile and feels good because it aligns with the classic fact that defects are cheaper to fix the earlier you find them.
Makes sense and personally I’m a big fan of shift left; but lately it seems to me that “shift left” has taken on a very waterfall form. This concerns me since I can see our industry about to take a massive back-step in terms of quality (both in terms of effectiveness and efficiency) similar to the naive test outsourcing done in the early 2000s. It took at least four years for the industry to right itself and get a net benefit from outsourcing and the same could easily happen again; and if nothing else that just makes my job (and all those genuinely trying to advance software engineering) frustrating and boring. This waterfall form appears most commonly in phrases such as: Read more…
This week we are exhibiting at the Gartner Symposium in Barcelona! At about 6,000 attendees it’s certainly not the largest show we attend (Mobile World Congress and Dreamforce are both over 100,000 people), but in terms of understanding what’s going on in the world of enterprise IT, nothing beats it. How are retailers going multi-channel? How are banks and insurance companies engaging with customers? How are utilities shifting to continuous deployment with safety critical systems? The answers (or at least the current thoughts) are all here as the CIO/CTOs of all these companies are here. Read more…
Performance and load testing are becoming more popular as digital business drives the testing focus towards user experience. Amazon recently stated that a 100ms decrease in the response time of their website reduces revenues by 1%. That’s $700M! So this popularity is justified. But traditional tools for performance and load testing are now entirely inadequate for testing the user experience of mobile and web software and a new approach is needed. This post is all about that new approach, which we call “application-level load testing”. Read more…
Gartner, the world’s leading information technology research and advisory company, has recognized the eggPlant range as a “visionary” in its ‘Magic Quadrant (MQ) for Integrated Software Quality Suites’! We’re excited to be included as one of the top vendors in the world, and especially by the specific comments made by Gartner who highlighted the “ease-of-use” of the complete eggPlant range combined with a “consistent stream of innovation” to deliver “a full complement of functionality including industry specific solutions”.
Ease-of-use and innovation are exactly what we focus on so it’s great to be recognised for this. It’s also what our users want. Our recent survey again highlighted that ease-of-use in all its forms (user experience, productivity, maintenance, reliability, and quick to learn) is the #1 thing our users want from their tools.
Probably my favourite comment in our customer survey this year however was “I see that my requests for improvement are taken onboard”. It’s a cliche, but our product development really is driven by feedback from our demanding and passionate users. They challenge us on innovation and they demand ease-of-use. So again we’d just like to say thank you to everyone who uses eggPlant and especially those who take the time to send us feedback.
Mobile app testing has really given test departments something to think about because testing mobile apps is much better suited to product experts than technical experts. There is a real difference between testing technical parameters and testing user scenarios.
But as companies consider making the shift from a manual testing environment to complex automation tools for testing mobile apps, the resulting skills gap often requires that companies hire new automation engineers to replace manual testers.
Testers are often the people with the most product experience, so it seems crazy to think that a company would choose to lose that expertise just because they have adopted a new way of testing, but unfortunately some automated tools are actually that complex.
With eggPlant, we have developed a solution that is simple enough for both manual testers and automation testers to use with ease, making the switch to automated testing quick and painless, and ensuring that a company’s product experts are not lost. At the same time, eggPlant is powerful enough to be able to test virtually any use-case.
Over the last few months we (at TestPlant) have started to see the first “second generation” companies interested in testing HTML5-based mobile apps. By “second generation” I mean companies who have already deployed a test automation tool, fallen out of love with it, and are now looking for another. Why did they fall out of love when it all started so well? The same three reasons keep coming up.
There is a conversation I have every couple of weeks that goes something like this:
- Customer – we tried to use eggPlant but our security team won’t let it through our firewall; or we tried to use eggPlant but the connection was really slow.
- TestPlant – why are you trying to communicate across a firewall?
- Customer – well the test team is in India, but we want to have our test environment properly controlled so we have it here in UK/USA.
The mistake at this point is to start talking about protocols, ports, security, policies, IT departments, etc. There is a much better way that is simpler, faster, and gives the customer a better test experience. The key thing to realise is that two things are being combined unnecessarily:
Fear, uncertainty, and doubt is misinformation designed to give consumers a negative perception of competitive products. It exists in all competitive environments, but the term was actually invented in the technology industry, and it is here where it is still most common.
eggPlant is unfortunately often subjected to misinformation talking about “brittle image based testing”. Like all misinformation this statement deliberately confuses two very different things. So we wanted to take a few moments to set the record straight.
There are many advantages of continuous integration, while the cons only number two: initial set-up takes time, and businesses need a cutting-edge testing tool to execute continuous quality control testing in an automated fashion.
Still, the two drawbacks of continuous integration cannot compete against the myriad benefits. For example, continuous integration enables developers to immediately flip back to earlier code should a bug arise, saving time by avoiding the traditional debugging process.
For many companies, regardless of their size, location or sector, business intelligence (BI) software is now playing an increasingly important role. These highly-sophisticated analytic systems are contributing to the understanding of complex business infrastructure, maintaining business operations and driving market advantage. As businesses increase in complexity, BI tools and platforms are providing an effective way to streamline workflows and processes in order to drive efficiency and revenue growth.
And yet the prospect of implementing business intelligence tools can be overwhelming. Encompassing so many systems; whether data mining, forecasting, statistical analysis or query reporting, demands that each of these components is considered – and tested.
We see a consistent demand for eggPlant across many markets including defense, retail and finance. But one area that has really caught our eye over the last 12 months is healthcare, and medical devices. In these markets software isn’t just mission critical, it can often prove life critical. And yet despite this, between 2005 and 2009, 113 medical devices were recalled from the market – a scary thought, and a source of reputational damage and liability for medical companies. Crucially, more than a fifth of these medical and healthcare device recalls were due to poor software testing.
The challenge in these markets will continue to grow as more medical devices come to market and more software is embedded within those medical devices. We’re witnessing more lines of code, more complexity, more sophisticated user interfaces, and also an increase in medical devices designed to run on mobile devices – an area in which TestPlant has vast experience. All of this is underpinned by continuing growth in regulation and compliance.
Reminder: There’s still time to sign up for the TestPlant webinar ‘Building better healthcare applications’. This webinar, which will take place on Thursday July 19 at 12 noon EST/ 5pm BST and will last 45 minutes, will provide some vital information for healthcare IT professionals, who want understand the role testing plays in preventing IT downtime and poor application performance.
The webinar will include details on electronic health records (EHRs) and how to test these across different platforms effectively and cost-efficiently; it will review the importance of non-invasive testing of healthcare applications; and assess common software failures in medical devices and how these can be avoided.
The webinar is recommended for CIOs, IT, testing, QA and application development specialists working in media devices, healthcare sciences or the life sciences. Please click here https://www1.gotomeeting.com/register/153636880 to register.
To read more about the unique testing challenges of the medical, healthcare and life sciences and how TestPlant’s automated non-invasive solution, eggPlant, can ensure your application development lifecycle – from browser testing, data-driven testing to load testing – is in good health, click here .
A recent article on the Bank Info Security website highlighted ‘6 Top Mobile Banking Risks’ and warned that banks should stop procrastinating and take steps now to assess and avoid potential threats. The story was spurred by new research from BITS, the technology policy division of the Financial Services Roundtable, which identified the six critical areas of concern as: rapid growth; the need for new security controls; more players, more risks; privacy issues; the role of consumers; and anticipating risk.
At TestPlant we wholeheartedly agree that financial institutions must analyse and eliminate any risk before rolling out new mobile technology. But we also believe that mobile banking is exciting, valuable, useful and inevitable – and that privacy and security issues should not be allowed to hold back its development.
With record breaking growth in the mobile payment industry that is only going to get bigger it is no wonder developers are scrambling to work out the latest and greatest mobile solutions.
The use of mobile devices, like smart phones or pad devices, to make mobile payments is becoming more and more common. This is more so true in European countries and throughout Asia, but is now seeing a huge boom in the United States as well.
In the scheme of things mobile payment solutions have only just begun to truly become integrated in the available mobile devices. It’s a relatively new concept for businesses wishing to allow customers to execute these types of mobile financial transactions in exchange for services.
But as awareness and demand increase, the technology supporting this industry needs to evolve, and it needs to do so at a pace that can only be described as expeditiously.
A question that comes up from time-to-time is the need to run a script that restarts the SUT. The code referenced below demonstrates the method for scripting through the restart of a Windows system; you’d do something similar for other operating systems. The process looks like this:
- Store the current connection information to use as the argument to the connect command after the restart
- Script the restart
- Wait for the restart to finish
- Reconnect to the system
The Windows OS presents a complication that requires a couple of workarounds. The VNC server runs as a service and seems to persist into part of the shutdown process. So the connection will drop initially, but a reconnect could be made briefly while the machine was shutting down. So to ensure that a successful connection takes place after the restart, it’s a good idea to add a check for a visual cue to see that the restart is finished. If it is not, assume that you’ve reconnected during the shutdown, force a disconnect, and start the check over again. You can see this process in action in this forum post.
The world’s leading visual test automation tool eggPlant is now smarter (and more literate!) than ever. eggPlant now has an Optical Character Recognition (OCR) engine. Since 2002, eggPlant has been the preeminent image-based test automation tool. With its new OCR capabilities, eggPlant advances beyond “image-based” to become a fully “visual” automation tool. The addition of a world-class OCR engine extends eggPlant’s capabilities in dramatic new ways, enabling it to read or find virtually any text on the SUT screen without sacrificing any of the advantages of the pure image-based approach. eggPlant remains completely agnostic to the underlying technology, interacting with the SUT exactly as a seeing — and reading — human user would.
The newest version of eggPlant can fully mimic a human user to control and interact with any system non-invasively. In addition to seeing and recognizing objects on-screen purely by their appearance, eggPlant can now read text and identify elements by their text labels without any need to communicate with the underlying system, objects, or language used to construct them.
The incorporation of an OCR engine expands on eggPlant’s 100% cross-platform, technology-agnostic approach to automation, enabling scripts to locate and read text on the screen regardless of the font, color, size, style, or underlying technology used to render that text (desktop, web browser, graphic image, etc.).
The addition of OCR further enhances eggPlant’s ability to test the tough-to-automate technologies that leave users of other tools scrambling to fill in the gaps with plug-ins, add-ons, and patches. eggPlant’s visual approach is easier to learn because it parallels the actions of a human user, and works out-of-the-box with any application on any system, from desktop to web to server to mobile device.
Beauty, they say, is in the eye of the beholder, and the beauty of automation simplicity is now in the eye of the eggPlant.
Customers – v11+OCR is available in August 2011 as an upgrade to v10.If you have an urgent need and would like to get a pre-release version, please contact firstname.lastname@example.org
eggPlant evaluators – Register to receive eggPlant+OCR trial keys.
Q: I have been doing FE test automation for about 10 years and have been using a lot of different automation tools. There is no single tool available that can satisfy all the requirements. We have been switching from tool to tool to meet different requirements and technology of the GUI changes. I really hope that eggPlant will finally meet our requirements. But I still doubt about how reliable and robust is the image matching approach that may require lots of maintenance. I want to show my boss that eggPlant can really work. Please give me some info about: How eggPlant can test on Flex/Flash applications in Windows XP/Vista?
A: Since eggPlant is an image-based GUI test tool, and only requires a VNC connection over TCP/IP, it can test Flex and Flash applications in a way much like a human would–by performing clicks and other system interaction on the remote system.
Q: How easy to maintain the scripts and the images of the objects if the GUI is still being developed and keeps changing?
A: Maintaining scripts is quite straight forward, for a couple of reasons. First, eggPlant is not co-ordinate based, meaning if a button or other GUI widget is moved to another location on the screen, this UI test tool will still find it on the screen. Second, our ImageDoctor function allows you to quickly re-capture images that have changed, and if an GUI widget has more than one state, you can add a representation and create an Image Collection.[/quote]
Q: How to deal with dynamic UI objects? e.g. the movie images are kept changing in Netflix, how do you click on a movie image and test on it?
A: For that example, I would consider importing images that are already being used on the site into eggPlant and use the image collection feature as mentioned previously.
Q: How to test on different language versions of the SUT? e.g. there are more than 40 different Intl locales for Yahoo Mail. Do you need to have 40 different images for the compose button which is displayed in different language?
A: If the text is selectable, you can use eggPlant’s clipboard sharing feature to pull the text into eggPlant for comparison, otherwise, I would use eggPlant’s Text Image Generation feature. You can provide a string to match, font, size, style and background and eggPlant will generate an image automatically. This approach is used by our retail and media customers for localization testing.
A: What I would recommend is giving your eggPlant controller machine access to where the images are stored and using eggPlant’s powerful file handling to pick up a list of images and searching that for the image you want to use. Each time you run the script, it grabs the file list and if there are new images, the script will take it into account.
A: eggPlant works from a black-box, user experience perspective, so while it cannot grip the attributes of the DOM, it does have the advantage of if the DOM changes, but the user experience does not, the script will not have to be adjusted. To followup eggPlant works differently than many test automation tools it works much more like a person/manual tester would.You wouldn’t ask a manual tester to click on a particular DOM object you would ask them to “Click the movie that shows up at this part of the screen”. If that element is dynamic you can often know where to click based on other static elements on the screen maybe a label that says “Your Selected Movie:”.If validating the image is actually part of the test then you will definitely want to have those images coming in from a data base or data file. If validating the title is adequate then you can either use Copy/Paste to “read” the field or use Text Image Generation (see eggPlant Help from the menu for more information).The bottom line is that I’m confident that we can use eggPlant to automate against your web application, but it helps to think like a user rather than like a computer when testing with eggPlant.
Q: Can automated UI testing be done for Flex pure AS3 projects. I have a requirement where we need to automate testing for AS3 projects which most of the automation tools I have found do not work. If its possible with eggplant then please attach a sample project for the same.
A: Yes, eggPlant can be used to automate a Flex application. This GUI test tool is neither object based, nor coordinate-based, but instead uses images to drive any user interface as a human would. This technology-agnostic approach allows you to test anything from Flex to Silverlight. eggPlant is not resolution dependent.
Q: I just want to confirm again from you does eggPlant serve for AS3 Projects as well. Flex uses mx package which AS3 does not have. So when we tried automating AS3 projects with QTP it wasn’t possible as it used Flex automation package which is part of mx package. Can you confirm that we shall not require any Flex related dependencies regarding this. Also, I just wanted to know how to integrate and test automation of sample AS3 project so that we can propose the use of eggPlant for automation in our projects. A sample would be a great confidence builder in this regard.
A: Truly, eggPlant can (and is) used to test Flex and Flash using any version of ActionScript. Because of eggPlant’s architecture it is completely independent of the underlying code. So even if Adobe comes out with AS4 next year, eggPlant can test that too! You can review how this is achieved in the eggPlant Technology section of our website.
Check this video eggPlant testing rich internet applications.
TestPlant recently conducted a webinar: eggPlant for mobile testing and how this black-box test tool can deliver value, increase quality, and decrease time-to-market in this very dynamic and exciting area of technology. Below are some of the questions that we fielded during the webinar.
To give you some background, the demonstration was performed from a Macbook Pro using eggPlant v10. The systems under test were an iPhone 3GS running iOS 4.3 using Veency to connect, and a Google NexusOne running Android 2.3.3 using DroidVNC as the VNC server.
Q1: Several participants asked if they could install their application and script against various devices including PC’s, Tablets, iPads, iPhones, iPods, Blackberry devices, smartphones, etc. even though they use different OS’s.
A1: The answer to that question is Yes. eggPlant is a black-box tool, and through the use of our Image Collection feature and tolerant searches, many scripts can be easily made to run against any device you wish to test.
Q2:How do eggPlant handle image interchangeability?
A2: You can import your own images into eggPlant, as well as replace them on the fly through the use of our CaptureScreen command.
Q3:What are eggPlant reporting skills?
A3:While eggPlant doesn’t give you code coverage reports, you can use our log files or eggPlant Manager to generate some graphical test result reports. eggPlant can also email test results out as part of a script through our SendMail command.
Q4:What is the licensing model for eggPlant?
A4:We have two basic types of licenses, node-locked and floating licenses. Within those two license types are development and execution-only licenses. All of eggPlant’s license types allow you to run scripts, but you can only develop scripts using our development licenses.
Q5:How to access eggPlant training?
A5:eggPlant is very easy to use, so many users can simply use our tutorials, forums, and getting started training that comes with purchase to get started. If you decide you need something more, our staff would be happy to assist. However, we find that in most cases it’s not necessary .
Thanks for attending the webinar. We enjoyed having everyone join. Feel free to request a trial by clicking here.
Ensure your applications are ready for your customers with eggPlant.
Most applications are designed to be used by people. So, it is surprising how many companies believe that testing at the code-level alone represents a sufficient software quality assurance test. In reality, unless you test from the user’s perspective there is no way to be 100% certain that an application will deliver a positive customer experience. Are you failing your customers by not performing software user acceptance testing?
eggPlant offers a unique solution to the problem of software user acceptance testing and ensures that anyone who uses it can rest assured they are delivering the desired user experience.
eggPlant tests from the end-user perspective. Working through the keyboard, mouse and screen, it mimics end-user behavior. Anything someone can do working through the keyboard, mouse and screen, eggPlant can do but in an automated, consistently repeatable and dramatically accelerated fashion. For Quality Assurance teams this means that tests can be created that mirror procedures users will be expected to take following the documentation, simulate how an application will be used, verify interaction with other applications and even experiment with various unexpected ways of working with the application that a user might stumble upon.
Most importantly, using eggPlant for User Acceptance Testing ensures that the customer is the focal point of the testing. This produces better user experiences and more satisfied customers.
The Black Box Approach
eggPlant replicates the way that an end-user works. Actions are taken such as click, drag, mouse-over, typing, or keyboard commands. The results of these actions are validated by ensuring that the application responds by moving onto the next step, screen or displaying what the user requested.
This type of black box testing is not concerned with what happens inside the application only that systems and application respond appropriately to given inputs. Because black box testing most closely approximates how an end user will use an application or system it should always be included in any well conceived testing strategy.
eggPlant tests from the User’s Perspective
Acting like a “virtual user” eggPlant uses image capture and compare technology to “see” the remote screen, push buttons, enter data and use an application the way a person would.
This image-based approach enables eggPlant to test any software application with a user interface.
eggPlant is NOT dependent on what programming language an application was built with, what operating system it runs on, whether it runs in a web browser, or uses non-standard controls.
eggPlant can test the interactions of ALL the applications on the remote system. It can copy data from one application, paste it into another, or perform any of the activities a person would.
eggPlant can even test network applications from end to end because it isn’t tied to a single test system.
Test without interference
Most automated testing software can only perform tests of the application running on the local machine. In many cases the presence of the testing software can skew or completely invalidate your testing results. With eggPlant you can test your running application on an independent computer (or computers) with virtually no overhead enabling reliable and accurate testing not only of your functionality but of your performance as well.
Test without modification
Because eggPlant doesn’t rely on special hooks or builds you can test software exactly as it will be run by your end-users. In fact you can even setup to test live installations of your software over the internet all through a secure SSH connection.
Test without installing software locally
There are many types of systems where, for security policy reasons, testing software may not be installed on a computer. That does not mean that there isn’t a requirement to test those systems and this places the burden on individuals. The eggPlant software testing tool offers a solution to overcome this problem through the use of a KVM switch. By connecting a VNC enabled KVM switch to a secured system, eggPlant can work through the KVM switch to control the computer and test it without the need to install any software locally.
Software integration testing – quicker, easier and more reliable with eggPlant
The integration of multiple applications is increasingly common and with it come new challenges to creating reliable products. It is no longer sufficient to simply test the application you have created, now you must test the way that application interacts with other applications that may be used along side it. If you attempt to undertake manual testing in this new landscape you face an ever-increasing burden that will likely overwhelm you. Employing automated test software focused on testing the user experience is the only way to rise to the challenge – eggPlant software integration test tool is the solution.
Integration testing is actually composed of different types of tests, but its objective is to ensure that the interaction of two or more components produces results that satisfy functional testing requirements. Integration Testing resembles actual usage more than Unit Tests do; therefore, functional deficiencies are more likely to be detected at this level of testing than they are with Unit Testing.
An Integration Test at the UI level is often used to validate a new UI component or to validate an application’s collaboration with other utilities/applications on a system.
eggPlant’s UI-level approach to testing makes it very easy to create Integration Tests between various applications. In particular, eggPlant’s ability to test third party applications allows you to easily create scripts to validate complex, multi-application processes.
Data Driven Testing with eggPlant
Accelerate data driven test automation using eggPlant.
Today, most applications require a user to interact with an application by keying in something at some point. Knowing how the application will respond to various inputs is essential to delivering a reliable, stable and quality product to market. The problem is, with all of the possible things an individual may do, how do you find the time to test as many combinations as possible? The answer, eggPlant Data-Driven Testing.
Data-driven testing is built around the need to test how an application deals with a range of inputs. Rather than having manual testers enter endless combinations of data or building specific values into a script, the test pulls values from a data source, enters that data to the application and verifies that the application responds appropriately before repeating the test with another set of values.
The range of input that a user interface can accept can be enormous. As a software system expands, manually entering test data into the system becomes increasingly impractical. Imagine having to manually validate all of the operations on the simple calculator interface – even a data driven test will cover only a tiny fraction of all possible inputs.
An important use of data driven tests is ensuring that applications are tested for boundary conditions and invalid input. A data driven test can validate that your application responds appropriately when a number is entered that is outside of the expected range, or a string is entered in a date field, or a required field is left blank. Data driven tests are often part of model-based tests, which build up randomized tests using a wide set of input data.
A data-driven test alleviates the pains of testing with large sets of data by separating test input from the test itself. Subsequently, tests do not have to be modified for varying sets of data and can remain constant for the duration of the test cycle.
eggPlant’s scripting language, SenseTalk, provides many features that facilitate the creation of data-driven tests.The following example retrieves test data directly from a text file, but you can also write scripts that call AppleScripts or shell commands to extract data from a database.
See an example – eggPlant for Data Driven Testing
Here at TestPlant, we have been working hard to consolidate all of the information about eggPlant. We now have a dedicated website for the eggPlant community with everything in one place.
We hope you like the new look and hope that you find it easier to find the information you are looking for.
TestPlant Support Team
Barcelona 14 February 2011 : TestPlant, the UK headquartered international software company, has launched the eggPlant Mobile Testing Solution. eggPlant is the world’s leading robotic test tool for GUI testing used by many leading software companies, by significant defence, security and aerospace organisations and by global media broadcasters and on-line entertainment businesses. The mobile testing solution extends eggPlant™s reach into software running on smartphones and tablets across the major mobile operating systems. The launch is in partnership with RealVNC, the original developer and leading provider of VNC technology used by millions of computer users around the world.
Testing software or apps on mobile devices is a major pain point for app developers, for device manufacturers and for network operators. There has not been an effective in house automated solution to testing software on the full range of devices. Existing techniques are either manual, which is extremely tedious and error-prone, or electro-mechanical, which is limited, does not reproduce a real user experience test and can place IP outside the enterprise environment. eggPlant uses RealVNC’s VNC Mobile Solution to connect directly to a device. A true user experience test is created as eggPlant is capable of recognising icons, colours, text and fields on any display. A test script is automatically generated as eggPlant is trained to move around an app using its image store to spot and compare display prompts. Multiple tests become effortless and eggPlant reports test failures with helpful screen images of the exact point of failure. A automated test script can be created on one device (or emulator) to be repeated many times on other devices.
There are 4.6 billion mobile phone users in the world (ITU, 2010) and that number is growing rapidly with a dramatic switch to smartphones and now tablets. With a fourfold growth in on-line commerce transactions over the past four years (IDC) and with that trend expected to extend because of the sophistication and convenience of smart devices, there is huge pressure on developers to improve the quality of mobile apps, speed up time-to-market and reduce costs. Automation of such functional or regression testing is a solution to this problem and eggPlant is expected to take a leadership position due to its non-invasive real user experience, its ease of use and its ability to be universally deployed on virtually any device.
24 January, 2011 – London, UK and Boulder, CO, USA TestPlant, an international software company with operations in the USA and the UK, today announced that its robotic test tool product, eggPlant, has been recognized by the US Patent and Trademark Office and secured a US patent. The patent, number 7,870,504, has been granted for eggPlant™s unique method of controlling or testing a system from another computer using a communications link such as Virtual Network Computing (VNC).
This allows eggPlant to see the screen on the system under test recognizing icons, buttons, message boxes, prompts, text, backgrounds and colors. George Mackintosh, chief executive for TestPlant said: œThis patent recognizes the innovation in how eggplant works and secures our position in the market as the leader in GUI testing. By creating a visual depiction of the screen, eggPlant can intelligently search and compare while signalling test success or failure. The technique is non-invasive, another unique and valuable feature for testers in industries such as defense, security, media and medical devices. George Mackintosh continued: œThis is the first example for years that a complete and current product in the $2.4bn global test tool marketplace* has been recognized by the US Patent and Trademark Office as being sufficiently novel and innovative to deserve this distinction. It™s a great demonstration of our leadership and technical expertise in a market that many consider dominated by giants such as HP, IBM, Micro Focus and Microsoft. This is a stamp of TestPlant™s authority to describe our approach to user interface testing as truly unique.
The testing industry Testing is a crucial function in the production of software. Worldwide spending on testing is estimated by PAC to be $105bn*. The elimination of bugs requires extensive review of programming code and exhaustive manual testing of the graphic user interface (GUI). Such repetitive manual testing is tedious and prone to human error. eggPlant uses intelligent image-recognition algorithms to ˜see™ the screen on the system being tested and, by comparison with responses it has been trained to expect, pass or fail the new or updated software version. With a flood of new and diverse computing devices appearing on the market “ smart phones, tablets, wearable computers, point of sale terminals, head-up displays “ the test industry and its practitioners are faced with mounting demands to validate software performance across multiple browsers, devices, operating systems and networks. Consequently there is a surge of interest in automating as much of this process as possible. Automation in software testing helps shorten system delivery times, improve quality and reduce costs.
London, 11th January 2011 “Today we’re pleased to announce full support for Microsoft Windows 7. Since eggPlant for Windows XP was launched in April last year, we have been working hard to add more platforms. eggPlant, with its patented technology, can now run on MacOS, Linux and Microsoft Windows XP/7. eggPlant for Windows XP/7 v10.3 release also contains further fixes and improvements from customer requests. The entire development team in Colorado is grateful for your feedback.
In summary v10.3 gives you:
- Full support for Windows 7
- Ability to search all scripts in a suite
- Full find and replace capability in script editor
- Optimised VNC connectivity for load testing
- SendMail functionality now supports attachments
- General bug fixes and stability improvements
To read the comprehensive release notes and to download your free new version, go to: http://www.testplant.com/downloads.
You have questions and we have answers. Here are some of the most frequently asked questions we receive. If you have a question that is not answered here, please refer to our online support resources or contact us for assistance.
Q: Which system platforms can eggPlant test?
A: Since eggPlant interacts directly with the GUI through the use of widely available VNC software, eggPlant can test virtually ANY system. For dependable performance over typical networks, you will want a robust VNC server with network recovery capability. Vine Server, RealVNC and TightVNC have all been verified as meeting these criteria. We maintain a list of supported VNC servers for major platforms.
Q: Do I need to modify my applications in order to test them with eggPlant?
A: Absolutely not! eggPlant only sees as much of your application as a user would see — in this sense, eggPlant is a “virtual user.” eggPlant requires no modifications of your application whatsoever.
Q: Does moving icons and windows around or changing the screen resolution break my scripts?
A: No. eggPlant does not rely on UI elements remaining in the same location but instead uses advanced searching algorithms to scan the screen of the other computer for the element you want to interact with. Changing the screen resolution just changes the number of screen pixels that are used to display each image pixel; the actual images normally do not change.
Q: How does eggPlant interact with custom controls?
A: eggPlant interacts with custom controls the same way that a manual tester would, by visually matching it on the screen. Since eggPlant does not rely on the underlying code of your application, these customized controls can be changed or moved around on the screen without any impact on your script.
Q: Can eggPlant test for tool tips?
A: Yes! You can move the mouse to the location to check and then wait to see if the tool tip appears.
Q: Can I run the same script against different operating systems?
A: Yes, assuming that the flow of your application is the same on different operating systems, you can run the same test scripts against them. You will need to capture a different set of images to use for each operating system. We recommend putting each set of images in its own suite and then adding the appropriate suite as a Helper suite before running your test.
Q:Are there any limits to which environments that eggPlant can test?
A: The only limitation to the systems that eggPlant can test is that it must have a Virtual Network Computing (VNC) connection either through the use of a VNC server installed on the system-under-test, or through the use of a hardware KVM-Over-IP solution.
Q: Is there a demo version of eggPlant available?
A: Yes. Click here to request a trial license. Try it; we think youâ€™ll like it!
Q: Your licenses require a HostID, if I use my ethernet card will I still be able to use the software when working through my wireless network connection?
A: Yes, as long as that ethernet card remains part of your machine then your license will work regardless of which physical network interface you use to make network connections. If you ever replace the card (or switch machines) you can permanently transfer your license using our online license management system.
Q: Can eggPlant convert text displayed into actual values?
A: There are two ways you can accomplish this. If the text is selectable, you can use SenseTalk’s RemoteClipboard() function to get the text off the remote system. If the text is not selectable, we have an approach that you can request through support.
Q: Can eggPlant check object states?
A: If the state is visible to the user, eggPlant can check for it. For example, since a radio button which is not selected does not look the same as a radio button that is selected, you can easily compare against the two.
Q: What is the difference between an execution license and a node-locked license?
A: A node-locked license can be used to develop and execute test scripts using eggPlant. An execution license cannot be used to develop test scripts using the eggPlant UI. It can only be used to run scripts. Contact your account manager for more information and pricing.
Q: Where can I download the latest version of eggPlant?
A: The latest version of eggPlant can be found in the Downloads section of this website. You will need a valid support and maintenance contract for eggPlant v10 and you may require a new license key to use this version. Holders of maintenance contracts for prior version of eggPlant do not receive v 10 for free. If you are not an existing customer, you will need to request a trial key in order to use the trial version.
Q: Can the application that I want to test be on the same machine as eggPlant?
A: Yes! If you are running Mac OS X, you can create and run tests all on a single machine. Here is a step-by-step guide for setting it up: eggPlant Single Machine Setup.
Q: Can I have multiple instances of eggPlant testing multiple SUTs?
A: Yes. You can put multiple eggPlant licenses on a single machine. Each of these licenses can be connected to a different machine and run its own set of tests. TestPlant offers substantial discounts for multiple license purchases. Contact your account manager for more information and pricing.
Q: Why is it called eggPlant?
A: The initial development effort had the internal code name “Operation Screaming Eggplant” named after a local band. The development team liked the code name so much we went with the Mac OS X tradition and launched with our internal code name — well, part of it anyway.