- In Blog
“Mobile World Congress – 10 years in Barcelona” greeted us at the airport. Inevitably it made us think about how much ‘mobile’ has changed in the last 10 years, but it also made us think how little MWC had changed in the last 5 years. Since “convergence” around 2006 and the release of the first iPhone it feels like MWC has been searching for “the next BIG THING” without much success. And while commercially it’s always been a great show for TestPlant, some recent MWCs have certainly lacked that industry buzz that was so intoxicating in the years leading up to 2006.
But this year the buzz was back! It’s difficult to put my finger on one BIG THING, but the buzz was definitely back, and I’m glad. Here are my key take-aways:
(*) Things! As expected the “Internet of Things” is invading MWC; and perhaps freeing itself up from pure “mobile” is what helped the event. All the stands in Hall 3, and many more throughout the rest of the event, had “things” on show. LG had a full connected kitchen, a mirror with head-up display, Sony had connected speakers, Qualcomm had everything from connected lamps to robots, everyone had connected cars, everyone and their dog had a smart watch, and there were probably more health wrist-bands than mobile phones on show. So everyone was showing that they are ready for the “Internet of Things”, but actually I thought only LG showed any real vision about how IoT was going to be useful.
(*) Phones! OK that may sound a little unsurprising, but we were taken back by the number of new handset vendors. Most of the handset vendors I was working with 10 years ago are gone (Sony Ericsson, Siemens, Panasonic, Sagem, even Nokia), and reports tell us that other than Apple and Samsung the ones that remain are losing money (and Samsung aren’t looking great). So in this fairly bleak market for handset OEMs, and the lack of differentiation on smartphones these days, why are people deciding to enter this market? I can’t answer that one.
(*) Nokia? I went to the Nokia stand (really just out of habit), but of course it’s the Microsoft stand now. Wow!
(*) Security. Security was everywhere. The digitalisation of retail and banking (and others), BYOD, and the continual high-profile attacks highlighting that security concerns are real, mean that security is a real, huge, market.
(*) Near-field networks. Presumably motivated by the IoT and ubiquitous computing, there were 10s of vendors selling local ad-hoc networking hardware (especially in Hall 5). The bandwidth and latency required for the IoT are driving a fundamental re-think of local networking and that was on show this year. Interesting technologies, vital enablers, but conceptually just faster bit pipes.
(*) Payments. Mobile payments has been a hot area for about 4 years, but there was a reality about it this year that was missing in previous years. People can now point at deployed systems and significant levels of transactions, and the next generation of services based on real-world experience are starting to appear. This reality was a key ingredient in bringing back the buzz.
(*) Testing! Last but not least. From our perspective we definitely felt that quality and testing are now a top priority for the mobile software industry. 5 years ago we rarely encountered a test manager at MWC, and those responsible for delivery/development didn’t have quality as a priority. But with fast and brutal customer reaction to poor user experience, and the measurable commercial impact of this, quality and testing are now definitely a top priority for most mobile software vendors.
So that’s it. On reflection I think that the buzz has come back because MWC has allowed itself to move beyond pure “mobile” (and maybe that IS the next big thing) and there is finally reality behind the topics that have been evangelised for the last few years (e.g. payments). In any case, for the first time in a while, I’m already looking forward to next year.
I’m really excited: At Mobile World Congress yesterday we launched our brand new eggPlant Network product. You spend such a lot of time planning how a product will work and then it’s out there.
It’s been enlightening to see the questions we get asked. Many of them we thought about during the design, but real users always put an interesting twist on it.
What’s eggPlant Network about? [Ah the fundamental question, to life the universe and everything; well, from my current standpoint it is!] I wasn’t going to trouble you with the answers to most of these questions, but that one’s fundamental, so here goes: We’re aiming to solve the problem of “How do I test a [mobile] application in a real world network in an easy way.
Then we have all the usual questions: Can you control it from eggPlant Functional (our functional test tool)? Yes, next…
All easy and straightforward, until: Can you emulate a 3G network? [I hesitate for a moment, because usually I know much more about this than the questioner. The answer our sales folk would give is: YES. But I know that the problem is with the question, not the answer. It cannot be answered as stated! And, sometimes to prove it I ask if people believe that 3G (or any ‘G’) performance is the same on one side of a building to the other… Ah, now you get it. There’s no such thing as 3G performance, it’s highly variable.
As they said in “A few good men” to “Just tell me the truth” – I know that mostly “You can’t handle the truth!” well not the full truth anyway, so a simplified version will have to do. And knowing that, we have provided lots of different 3G network experiences on eggPlant Network; from good through to terrible. So I (truthfully) say: Yes
A couple of milliseconds later we move on, and the questioner never realises my quandry. But now you do: Can you handle the truth?
- In Blog
InterConnect is IBM’s new customer and partner event running for the first time in 2015. It is the consolidation of three existing IBM events – Rational ‘Innovate’ (which TestPlant has been supporting for years), ‘Pulse’ for cloud computing, and the Websphere ‘Impact’ event. So 20,0000 people in Las Vegas talking about all things IBM.
So what are they talking about and what aren’t they talking about? Here are my 4 takeaways from the event so far.
(1) Since the 1960s enterprise IT has been focussed on improving the efficiency of internal operations; and since the 1980s vendors have been selling to CIOs. A couple of years ago Gartner highlighted that IT was now being used increasingly for customer engagements and CMOs had IT budgets as big as CIOs. This realisation has been spreading around the industry since and is front-and-centre at InterConnect. IBM are talking a lot about both “systems of engagement” (for CMOs) and “systems of record” (for CIOs).
(2) These different systems are managed and updated in very different ways, “systems of engagement” are responding to AppStore feedback and are being updated monthly (or faster), whereas “systems of record” updates are measured in years. How to manage these two very different development processes? That’s a big topic being discussed here under the heading of “bi-model” development, and “hybrid” development where the same team is operating in both modes. Fortunately we can easily answer which test tools are already widely used for both – eggPlant!
(3) ‘Watson’ is clearly the big new innovation that IBM is keen to promote. IBM CEO Ginni Rometty has said that Watson will be generating $10bn in revenue per-year within the next 10 years. And it is pretty cool. The main take-away for me is that the key innovations in Watson seems less about the underlying artificial intelligence (though clearly that is there), and more about the ability for the system to execute queries articulated in natural language. So it’s less about the analytical ability of the system and more about making that analytical power available to people. It’s also interesting that IBM are leading new innovation in the area of natural language interpretation, something that so far had been Google’s domain.
(4) And finally what they are NOT talking about. No-one from IBM is talking about ‘Rational’. At the beginning of this year IBM (very quietly) re-organised the software team and ‘Rational’ as a brand and as an organisation no-longer exists. The existing tools have been moved into different (and separate) teams; e.g. UrbanCode is now part of the “Cloud” team whereas RFT is part of the “Systems” team. So RIP Rational – but long live eggPlant as we support IBM and its customers test and monitor in IoT, mobile, and systems engineering.
- In Blog
The traditional approach to performance and stress testing is:
(1) Define objective
(2) Create test
(3) Run test
(4) Analyse results
(5) Go back to #2 and repeat until clear conclusion
But modern load testing tools such as eggPlant Performance have rich ‘dynamic test control’ functionality which allows the tester to change the test at run-time. For example, increase the number of virtual users or start monitoring memory on a particular server. ‘Dynamic test control’ enables a very different approach to performance and stress testing which is far less structured. A tester can now run up the system-under-test, start the load testing tool, and “play about” with the system mixing-desk-style until they have achieved their objective and clearly understood the system behaviour. This more flexible approach to load testing is increasingly popular, with almost 50% of our customers’ testing being done this way, but is that a good thing?
‘Dynamic test control’ was primarily introduced to deal with limited test windows against production systems. Performance and stress testing, for various reasons, often has to be done against the production system, and obviously no-one will allow testers to do this while their customers (or employees) are also trying to use the system. So testers are often given a short window (usually in the middle of the night) to do their testing. In such environments a tester needs the quick flexibility that ‘dynamic test control’ gives to adapt to new information – and ‘dynamic test control’ is a very good thing.
People now use ‘dynamic test control’ in all situations; and it can be a great way to speed-up the iterations of #2 – #4 in the traditional approach described above. The problem I see is that too many people are now using the flexibility to skip #1. They don’t define their objectives, i.e. what it is they expect the system to do, and what they are trying to verify. They just run some tests against the system, look for anything “strange” or “bad”, and otherwise mark the test as passed. But if you don’t know what scenarios you need to test, and you don’t know what problems you’re looking for, then performance/stress testing is like looking for a needle in a haystack. You need to know what you’re testing, the failure criteria, and the key risk areas.
So ‘dynamic test control’ is great functionality that can speed-up performance/stress testing and help you more quickly isolate root-causes. But you need to ensure that you still clearly define your test objectives (requirements, criteria, risks) or it’s a very false level of assurance.
- In Blog
The most famous and over used graph in the testing world is the one below. In everyone’s mind it simply says that the earlier you find defects in the lifecycle the cheaper they are to fix.
Most test process change project proposals start with this graph and really use it to justify the majority of their recommendations. I never find such proposals convincing and over the years have identified four reasons why not.
(*) What the graph really says to me is that by far the most important thing to do is not let defects get into production. That is what everyone should be focussed on; and only when we’re awesome at that should we start worrying about moving from “Implementation” to “Design”. Sure, effective QA requires activities througout the lifecycle, but those activities should be focussed on preventing defects getting into production, not moving detection up the (internal) lifecycle for efficiency reasons while many defects are still being found by customers. But most proposals I see are focussed on moving detection from “Testing” to “Implementation” or “Design” while a huge number of defects are still being found in production.
(*) Most of the recommendations actually don’t follow at all from the graph. The most common one these days is “we will save money by testing earlier so we are only going to do unit testing from now on”. Huh? Why does earlier testing mean unit testing? Just because on a Waterfall or V-model it comes first? Aren’t we all a bit more educated than that these days?
(*) The graph lacks a key piece of data – the cost of finding defects. In production the cost of finding defects is $0. But finding defects at the requirements stage is rather expensive. Especially if your process applies the finding process to requirements that may never be implemented. I’m not saying that the cost of finding a defect at the requirements stage is such that it balances out the cost of fixing at the maintenance stage, but maybe it is in some environments, this is an important bit of data that’s missing.
(*) The defects that are showing up in production – where were they injected? If they are all late stage simple code bugs then doing a whole lot of requirements reviews isn’t going to help much. Few people include data about where their defects are injected.
So the graph is interesting, but I find it’s massively over-used these days, and most often it’s used when people haven’t really thought through what they are doing.
eggPlant Manager has always allowed you to have a central location for scripts, run tests concurrently and give you a high level graphical display for your results. With the release of eggPlant Manager 4.0 we have built on this core functionality to deliver an even more effective test management tool. We have added in…
- The ability to run external jobs, which includes Selenium and JUnit tests.
We understand that customers may already have lots of testing assets in other tools and languages, so we have now added in the ability to execute any arbitrary command that you can run through the command line. This means, you can use eggPlant Manager to run a full end to end test, that might incorporate all of eggPlant Functional, Selenium and JUnit tests, and then have the results of all of those pushed back into eggPlant Manager, to give you a central place for all of your results. Essentially, we are allowing you to execute any command line call from eggPlant Manager.
- Integration with eggCloud.
If you already use eggCloud, then eggPlant Manager just became a whole lot easier. You can now import all of your eggCloud connections into eggPlant Manager, and then schedule tests to run against those devices. This means that as part of executing your tests, eggPlant Manager can book the device you want to use.
- The ability to run the table tests you have created in eggPlant Functional v15.
In eggPlant Functional v15, we added in the ability to create tables so that it was possible to do keyword-driven testing. Of course, understanding that people want to schedule table tests to run whenever they want, we’ve now added in the ability to run tables from eggPlant Manager.
If you are already a TestPlant customer you can download eggPlant Manager here and contact your Account Manager for a trial key. Otherwise, fill out a form here and one of our team will be in touch shortly.
- In Testing
Like many test managers, I track some metrics, some of which are helpful some of the time. My favorite tends to be Customer-Reported Unique Defects (CRUD, aptly) which tells me how many bugs have made it into the current version of a product and been found by an unfortunate user. Recently, however, we had a software release that felt quite smooth to me, but one month after the release, the CRUD count was higher than I had expected: we had eight more CRUDs than we’d had a month into the previous release. Rather than panic (or at least in addition to panicking) my team sat down to figure out where these bugs had been hiding. What we found was two parts humbling, nine parts encouraging, and three parts liberating, which was not a half-bad mix.
Our first step was to go through the issues and divide our bugs into categories (UI, Connectivity, Memory, and so on), and then rate each bug on the following scale:
Simple: 3. We should have caught these.
Complex: 5. We might have caught these with deeper testing.
Time Intensive: 2. We could have caught this by letting our tests run longer.
Resource Intensive: 1. We could have caught it stressing the system harder.
Unexpected: 3. We could not have predicted these.
What we ended up with was a quite comforting table. We identified some extremely easy ways we can do better:
1. Run our nightly regression longer: same tests, more repetition. This would have revealed a memory leak in the last release, and now that we think of it, probably a couple of other issues over the past year. (Go, go automation!)
2. Use lots of resources. For some users, this would mean large volumes of data. In our case, it means big test suites. Some performance problems just don’t present until you tax the system.
3. Try to ward off last-minute changes. OK, maybe not so easy, but worth a friendly mention to development. Of the three bugs we identified as simple, one was introduced at the 11th hour, and there was no time to catch it.
4. Mind the low-profile functionality. This is the one I hate to admit, but two of our simple bugs involved functionality that, frankly, just doesn’t get much buzz around the office. No excuse here.
Five of our bugs were needles in a haystack– specific sequences of actions that don’t follow any predictable workflow. We might never have found them, but to stand a chance we’d have needed to do some seriously intense exploratory testing, or run more finely modularized and randomized automated tests. We’re opting for the automated tests. This will take some time, but it will be an investment.
Finally, there were three bugs (our favorite three bugs) that we absolved ourselves from in good conscience. One came from a user running in a unique environment that we couldn’t have anticipated, and we’re just happy that we can help him out now. Two bugs came from small features that weren’t tagged in the release; we simply didn’t know they were there. Civilized, productive conversation ensued.
Of course, no one would ever tell you that you shouldn’t look back on your bugs, but for me, this exercise was a reminder to do it regularly and deliberately. Fourteen bugs are fourteen great teachers.
Almost any discussion about software development process could be improved by clearly separating the questions – who, what, and when. That may seem ridiculously obvious, but I can’t remember the last discussion I had where someone didn’t assume the answer to one of these questions from the answer to another (most commonly assuming “who” and “what” from “when”). Read more…
“Our requirements are chaos”. It’s something I’m sure you’ve all heard and many of you have said. An exclamation of frustration and self-deprecating admission. But it’s only this morning that I realised it’s also a correct statement of fact. Requirements are indeed chaotic in the mathematical sense, that is a small variation to the inputs (i.e. the requirements) can lead to a dramatic change in the outputs (i.e. the product). This is sometimes called the “butterfly effect” since theoretically a butterfly beating its wings at a certain moment could cause a hurricane on the other side of the world (weather is the most studied chaotic system). Read more…
- In Blog
It’s the holiday season which means it’s the retail season! I’m definitely not much of a shopper, but even I’ve been on-line buying all kinds of Arduino gadgets to give to my friends. And as I see facts such as “Black Friday on-line sales up 17%” I naturally start thinking about how many on-line retailers are doing proper load and performance testing and how many are just hoping everything is fine. I suspect that fewer than 10% are really doing load/performance testing.
It’s a bit of a mystery to me why so few on-line retailers (and other digital businesses) do proper load/performance testing. I can only assume that they don’t believe it’s really that important or sensitive. That as long as the performance is “OK” when they randomly try it one day then optimising the performance isn’t going to translate into more sales. But if this is the case then they are wrong. Very wrong. Read more…