I spent years trying to remember the name of this manga and find it again... I took a chance and wrote "gardener" in the tags today and FINALLY - I found one of my fave mangas once again! ^_^
Kayashimashi no Yuuga na Seikatsu