Wearing insulation that you'd be carrying otherwise, either as a primary sleeping system, or as a means to boost the performance of lighter bag is a fairly common idea.
But, since people sleep differently, it's hard for us to say whether you would be comfortable in you chosen setup.
I'd consider taking a very light down bag (e.g. 500-600g), and wearing the clothing inside this. Unfortunately, this is the wrong way around for the dew point; in mixed insulation (down and synthetic) systems, you really want the down innermost so that the dew point falls in the synthetic insulation that's better able to cope with moisture.
Pieds d'elephant (elephants' feet) have also been around for a long time in Alpine climbing circles; Alpkit have just brought them back...