Amanda - glad you made it on here! You've definitely got a great question that plenty of dealers need to ask from time to time. Of course there will be some highly biased responses (like mine) toward companies we are currently using.
I've been through quite a few (AutoMark [Reynolds & Reynolds purchased later], Traver [ADP purchased later], DealerOn) and have found my happy home with Dealer.com.
Aside from talking about my experiences with Dealer.com, I think I'm going to take Nick's approach and label some things for your consideration.
SEO - I think this is first and foremost on anybody's list. How does the website provider open up things for search engines to find you?
- do they change the meta-tagging for each page of your site or allow you to do it yourself?
- do they allow you to change the titles of pages?
- do they allow for custom page layouts and various font formats?...this is kind of standard
- do they allow for image tagging?
- do they allow for video uploads?
- do they allow you to put custom messages on inventory for advertising purposes?
The SEO list is endless and fluid, so you want to make sure that the company is mobile enough to change with the Google's and Yahoo's as they make changes to how they index your site.
Reports - you've got to be able to see if you're effective.
- do they monitor bounce rates?
- do they monitor bounce rates on specific pages so you know whether to change things on a page or not?
- do they monitor lead conversion rates?
- do they monitor lead conversion rates per each form so you know to fix a form?
- do they measure your inventory health: prices, comments, pictures?
- do they tell you which cars are the most viewed and which aren't being viewed at all?
- do they measure the time a customer spends on your site?
Like SEO, reports need to be fluid and constantly added to. There are always new break-throughs and ideas being derived from website analytics (reports).
Inventory - this will be the most viewed part of your site.
- do they display your inventory in a user-friendly way?
- do they allow you to embed specials in your inventory?
- do they have conversion points on the inventory (forms, phone numbers, chat points)?
- do they allow you to integrate "extras" on the inventory (upload to twitter, facebook, etc)?
- do they make it easy for a customer to find your dealership from the inventory page?
- do they allow a customer to easily reset the search parameters to start a search over again?
This is an area that can make or break you, so spend a lot of time figuring out what you want from your inventory display pages.
Leads - do they send XML leads so you can receive them easily in your CRM?
- do they provide a way to track all your leads?
- do they provide an ILM tool?
- do they provide an automated response for customers who have submitted a lead?
- can the automated response be turned off or customized?
Of course, conversion is the main goal of your website and leads are the easiest tracked.
There are many other questions to ask and some of these questions are more important than others. Some are things that could be in development with a site provider, so be sure to get an ETA if they say they're working on it.
I'd like to finish up by saying that I have no idea how knowledgeable you are Amanda, so please don't be offended if some of the things I posted are Internet 101 and you're well beyond it. Feel free to ask me any direct question you like.