Place potatoes in a pot and cover with cold water. Cook until tender, about 25 minutes.
Meanwhile, place garlic cloves in pan with 1 cup of skim milk. Bring the milk to a boil and simmer for five minutes or until garlic is tender crisp. (This procedure removes any bitterness from the garlic.)
Discard the milk.
Add the remaining 1.5 cups milk to the helf-cooked garlic. Simmer over low heat until the garlic is very tender and can be pureed. Puree in a food processor or blender until mixture is frothy and thickened. Set aside. Once potatoes are cooked, drain well and mash them with a potato masher or ricer until potatoes are smooth. Stir in the milk-garlic mixture along with rosemary, pepper and salt. Mix well.