Strictly subsonic suppressed, what would be the more effective cartridge say out to about 40 yards?

I'd go 300BLK just for the versatility to shoot different loads.

What barrel length?

The speed of sound at sea level is just over 1,100 fps, therefore, particularly at short ranges, the heaviest bullet you can deliver at that muzzle velocity should be the most effective, which tends toward the .45 acp at 230 grains, and .452 initial diameter. Many modern hollow point .45s will expand to nearly 1.0" which is really impressive.

However, .300 BO engineers know this and have been working on it. There is now .300BO 220 grain subsonics that expand to around .6" and some radical deep hollow point 3 or 4 petal designs that will open up to over 1.25". I don't know if those would hold together in anything other than ballistic gel, but, it looks impressive.

I think the advantage goes to the .300BO, due to ammo availability. I was unable to find any .45 ACP round specifically designed for carbine use. Theoretically, if you could get a .45 bullet in a carbine load that would weigh close to 400 grains, going 1100, you would have a real hammer, just a little under a .45-70 black powder load, but no one seems to try to make one, likely because some asshole would try to fire it in a 1911 and give themselves a case of metalface. There might be issues with allowable case pressures as well, because such a bullet will really intrude on the case, and there will be very little initial volume for powder, so such a load in .45 may not be possible.

In an AR platform gun, where there is no benefit to the smaller magazine size of the .45, I really think the .300 is the better, more useful choice.

Now, if you want to open up the category a bit, the .458 SOCOM is available in a 600 grain round nose solid running at 1000 fps...

300 BLK would be my choice.

Without question 300 blackout, especially if you ever shoot farther. Magazine reliability, simplicity of the build, level of suppression, and the ability to hit 3x harder with supers would make me use .300

45 auto may win in the price war, but that's about it.