Nyungwe Forest National Park indeed is magnificent, and indeed costs can add up.
Coming by public transport is very well possible; take a bus from Butare (I went with Anatracom, which took me more than 3 hours to Gisakura) and ask the driver to let you at where you need to be. Be alert though: with me, the entire bus was discussing whether we had reached my stop or not; people are not really sure. Another couple I met was taken all the way to Cyangugu to find out there that their bus driver did not know where to stop, contrary to his promises.
Giaskura guesthouse is a lovely place with very friendly staff. Be sure to make a reservation ahead though: you don’t want to have spent a few hours in the bus to find out there’s no bed for you.
Now ORTPN has come up with some solutions for people who don’t have their own transport, but they’re not cheap. To go from Gisakura to Uwinka (18 km), they arranged a motor taxi for me, but the price was way too high. If you want to leave Uwinka, they can make a reservation for you with a bus coming from Impala Tours coming from Cyangugu. This works excellent, but you have to pay the full Cyangugu – Butare – Kigali price, since your seat covers all that way (Rwf 4.500). Anyway, if you want to go to Kigali, this is a fair rate.