The downside? You can see who favorites your photos.
Edward is the ultimate creep (photo from here)
Flickr is crawling with fetishists. It is inevitable with it being the internet and the existence of Rule 34*. So I am not too surprised that there are people with 500 favorites of women in plaid skirts and nylons. While that is impressive due to the person's dedication and thoroughness, it is very creepy.
These people are very easy to spot. Usually they have no pictures if their photo stream. If they do have any of their own pictures, it's photos of their fetish. Then their favorites are nothing but the most specific niche you can think of. Women in tights, men wearing a particular color of denim, or red silk (and only silk) scarves.
Note to any Flickr fetishists out there: You can easily disguise yourself if you posted other things. If I see someone who has favorited nothing but their particular fetish, then it is creepy. If someone has a mixture of things, like some pictures of airplanes or animals, I'm more likely to think they are saving my pictures for fashion - rather than sexual - inspiration. If you're that desperate, just bookmark or save the photos. We just don't want to know.
Flickr fetishists are not a huge problem. I don't want to give that impression. Fortunately, Flickr makes it easy to ban someone from favoriting your photos, so you don't have to be reminded about THAT CREEPY GUY that keeps commenting on pictures of your shoes.
I have nothing against other people's kinks. You have a thing for women's hands? Sure, whatever. Like seeing men dress up as clowns? Knock yourself out. But I do not want to be involved in strangers' sex lives. By favoriting a picture, it is publicly announcing that you're including this photograph into your sexual routine, whatever that might be.
Everyone has a right to whack off as needed. I also have a right to not know about it. Ew.
* Rule 34 - "If something exists, porn of it exists. No exceptions."