I was checking everything again and I discovered a slightly loose head bolt on the exhaust valve corner of the engine. I removed the head and notice a blow through mark and the gasket eaten away at the bolt.
Does this sync with the engine problems? I can replace the head gasket easily enough if this could have been the problem. I am thinking that the cool engine was sealing "OK" but as it heated up it expanded a bit to leak, even at idle it would run ok but higher RPM would cause loss of power, etc.