Chutzpah and non-ES5 JavaScript for Unit Testing is Problematic

I've been working with Chutzpah for VS.NET as a test runner for my Jasmine JavaScript unit tests and ran into some issues that had me essentially stuck and running out of debugging options. I mined Matthew Manela's samples, GitHub repo, articles, log tracing, settings files, simple test harnesses, and many other trial and error event before finally determining the root cause of my headaches.

No matter how simple it seemed of a test harness I used in JavaScript to test using Chutzpah, I kept getting errors like the following: 

Can't find variable: myVariable in file xyz....

Normally this is an indication that the referenced JavaScript file is not properly referenced and the test cannot use it. Therefore the test fails and complains that a variable can't be resolved because ultimately the file is not referenced or not referenced properly. Or so it appears on the surface.

This is the red herring...

This led me on a path of vigorous path tracing, trying different path notation for referencing, using chutzpah.json files, running from the command line, debugging, refactoring, and on and on. No matter what I'd do I couldn't apparently get the file referenced for the unit test to run.

Naturally what makes this all worse was that if I open my JS tests in the browser using the default Jasmine Test Runner, they of course pass. So I know it's only a problem with Chutzpah running my tests.

I do the most basic of tests:

expect(true).toBeTruthy();

This passes. So at least I know VS.NET and Chutzpah can actually work together. 

Here is the crux of it and I'm not sure why it dawned on me but I decided to investigate my JS code. I had begun to sprinkle in some ES6 syntax that by now was compatible in most current browsers. Things like 'class' or 'for...of' loops. The problem is Chutzpah uses the PhantomJS headless browser to run the unit tests. It is based off the WebKit layout engine, so its about like running tests in Safari.

However I was completely wrong in this assumption. From the following (albeit an older package the reference still holds for this topic): 

"Most versions of PhantomJS do not support ES5, let alone ES6. This meant that you got all sorts of errors when you tried to test ES6 features, even if you had used the Babel/6to5 transpiler."

PhantomJS today appears strictly ES5 compliant and even a single line of ES6 syntax in your JS file will cause the referenced file not to be processed and thus serving up the error that it cannot be found. It can't be found because it can't be understood by PhantomJS. Hence the red herring about not being able to understand a variable being referenced in my test because the JS class could not be used.

I had one particular JS class I thought was 100% ES5 and this made the debugging even worse. The way I finally sniffed out some ES6 syntax was to drop it in a TypeScript file and target it to transpile to ES5 JS. When I compared the files side-by-side sure enough I found some ES6 syntax in my JS files. Once I ported the ES5 compliant JS over, Chutzpah and PhantomJS both worked perfectly and my tests passed within VS.NET.

I think the lessons learned here are the following:

• PhantomJS which Chutzpah uses to run unit tests requires ES5 only JS, so make sure all of your JS is ES5 transpiled (Babel, TypeScript, etc.) and target these source files for the unit tests
• If able to write and target ES6 features and browsers and not wanting to transpile to ES5 compliant JS, consider using a different JS Test Runner than Chutzpah.