There have been experiments to see if this can be done commercially so far only a few have been successful. In a domestic situation it is impractical due to the sheer volume of compostable waste needed.
First you need to decide wether to use the compost to provide all necessary heat or optimum levels of CO2 & partial heat.
If the composting component is sized to heat the greenhouse (17.5 cubic meters of compost per cubic meter of greenhouse) the amount of carbon dioxide generated will be six times that needed for optimal CO2-enriched atmospheres, and the amount of nitrogen (ammonia) released will be fifty times that needed for optimal plant growth.
When the composting component is sized on the basis of carbon dioxide the heat generated will be supplementary only meeting perhaps 15% of the energy needs. Excess nitrogen however will still be a troublesome contaminant of the system at levels roughly eight times greater than optimal. Nitrate levels are consistently too high for safe production of veg due to accumulation of nitrates in the vegetables.
If you decide to use the compost to produce all the required heat you then need to find a safe way to transfer the heat into the greenhouse without the excess carbon dioxide & nitrogen eg; heating water & pumping that through pipes/radiators within the greenhouse.
If you use the compost to provide optimum CO2 & partial heat then you have an environment for starting seedlings off but not suitable for growing plants to maturity.
If you want to read a bit more on the subject take a look at this report on a 2 year trial. It is an 8.4mb pdf file
http://www.vsb.cape.com/~nature/greencenter/pdf/compost.pdf
