I don't know how they technically work (the unit I linked to), however I don't believe they are like a water hammer arrestor with an air chamber, which I believe can fill with water under certain circumstances. Your on/off test should be conducted by allowing the lines to fill completely, .... shut the valve off and allow enough time for any pressure transients caused by the closing valve to dissipate, then fire it up again. If the hammering is just as severe then I would be suspecting some serious line drainage/leak, or there is trapped air. There may be other causes as well, but those would probably be the first I would consider, especially given the size of that line. If you are going to replace the valve, might as well put in a Griswold DWS. This will help to control hammering from both opening and closing the valve.