How We Test Software
By Jeff Kadin
Senior Manager, Software Engineering, Harman Luxury Audio
I have been an audio design engineer for most of my career working for CBS, Sony, Krell, and now Mark Levinson. My first job was with CBS, where I developed two patents, working on vinyl to Compact Disc and everything in between. I was very proud of having a system I developed selected to produce the Grateful Dead’s cassette tapes. I later spent ten years at Krell designing and writing software for over 50 audio products. These two jobs, in particular, have provided me with the perfect storm of knowledge to work at Mark Levinson.
I manage, write and test software, help design digital hardware and diagnose problems. Based in Shelton, Connecticut at the Harman Center of Excellence, I am part of a small and extremely talented group of engineers with decades of experience in mechanical, digital, and analog engineering. Much like a five-star restaurant that has several specialty chefs to create an unforgettable meal, together, we create unforgettable products.
The software we write provides the user interface and controls the hardware in Mark Levinson products. Our main responsibility is to make sure the software provides an enjoyable user experience and performs in a reliable and predictable manner. First and foremost is reliability. We design our software with a philosophy of making sure protection functions are the highest priority. Our products include many types of sensors, such as voltage, current, and temperature, and the software monitors these signals. We dedicate specific test cases to make sure the protection built into the software is working as designed. We test our products in our lab using a mix of automated test equipment and hands-on operation. We use computers, remotes, and connected components from the real world to make sure the unit is operating in the same manner that it would in the customer’s home.
Software testing can get very extensive and easily take large amounts of time. The number of test cases grows exponentially as you design cases to test every possible combination and can easily grow to tens of thousands of cases.
We need to test all functions of the unit. For example, turning on or off, but not just turning on and off, turning on via front panel, turning on via remote, turning on via APP, etc. We cannot possibly take all the time needed to test these thousands of cases so we need to reduce the number of cases in a very careful and targeted manner.
• Targeted Testing
When we do targeted testing we are testing a specific function to make sure it operates as intended without any other cases. This ensures that we know that the function works correctly before trying other cases, saving time by allowing for a correction to be made if a flaw is found before more extensive tests are performed.
• Automated Testing
We have special audio testing equipment that can generate and measure analog and digital audio signals. It can also control our products via RS232. All of our products have RS232 control and using RS232 we can make sure every product goes through the same exacting test procedure as any other. We test every published RS232 code to make sure the product responds correctly. We also add special RS232 control codes that allow us to test conditions that would normally take user intervention. Automated testing can test many cases in a short time, some faster than a human can, and is repeatable without the vagaries of a human operator.
• Advanced Stimulus Testing
Some of the safety features would be unsafe to test unless we had special embedded tests that can simulate a failure and make sure the software responds correctly. We can also change certain parameters in the code that would allow, for example, a 20 minute reaction time shortened to 2 Minutes or 2 Seconds for testing, saving a lot of time but not changing the action.
Redundancy is one way we reduce the number of test cases. Since we write the software, we can write it in a way that not only makes it more predictable but also reduces the number of test cases.
We use redundancy when we write the software by writing one function that would do the same thing for many cases. For example, if the product had six digital inputs and six analog inputs, instead of writing 12 functions, we write only two -- one for digital inputs and one for analog inputs. The functions take the current input as a parameter and performs all the necessary tasks that would be required to select that input. This means we only need to do full testing on one digital and one analog input as they all act the same. We still have to test switching between inputs, but not every input with every parameter.
I hope you’ve enjoyed this brief overview and have a better understanding of part of the process that brings these great products to you.