• Stop being a LURKER - join our dealer community and get involved. Sign up and start a conversation.

Inventory protection Idea that doesn't work

@Cody Teegerstrom

Personally, I think this would be a good start. When a SRP is loaded for the first time, fire the captcha JS call. This may not stop all the bots but, it will stop quite a few of them. It would also help give more accurate data in GA as well. You would think that website providers would at least offer something like this but they don't!

https://developers.google.com/recaptcha/docs/invisible#programmatic_execute

View attachment 3315

Next would be cleaning up your export list in your inventory syndication dashboard. Chances are if you login and look at the sites your inventory is being exported to, 50% you've never heard of before or have never received a single lead from.


@Rick Buffkin
"it is invoked directly when the user clicks on an existing button on your site or can be invoked via a JavaScript API call."
Interesting idea.
 
You are summing they are scraping your site, they could be scraping another site (someone you use for advertising) or flat out getting a feed from vendor that is selling them.

@yagoparamo, I know brother!!! But, the solution has to start somewhere though. That's why I stated to clean up the syndication list as well earlier on in the thread.

At the end of all of it, This is the Dealers Digital Showroom. The dealers digital lot. Dealers wouldn't allow a company to come into their physical showrooms and gather all the information about their inventory and use it how ever they want to. Why allow it to happen online??? We can't fix them all but, we can at least start closing some of the gaps.
 
@Rick Buffkin This is a bit of a cat and mouse game but there are definitely ways to prevent them from scraping. The first is banning a group of IP addresses they use to scrape. You can take that a step further and prevent those pages from rendering pictures or pricing data when the page wasn't requested from another page on your site. Theres some caveats to that though.

I think the best option is to contact your dealer association and have them handle it. This service is misleading customers at your expense, which is a big no no. Surprised they're not sued into oblivion yet.
 
@jbarron Can you ask a dealer to share? I'm not asking for trade secrets but for information to help dealers solve a common problem.
Sorry for late late response. I actually spoke with the regional Ecom director for Toyota. He said that no dealership will be issued a strike against them for this website (Yay!) He also said that we should contact the Tennessee dealer association, and let their lawyers handle it. California is doing it. He was frustrated that these sites are using our information, including VIN numbers, against our will.
 
This article seemed relevant to this old (still ongoing issue) scraping problem:

http://bgr.com/2018/01/16/southwest-airlines-sues-southwest-monkey-price-drops/
That might be the case, but chasing these kind of groups down is a tedious and expensive task.

I'll add this Irish group to the quality vendor options at https://scrapinghub.com and https://www.octoparse.com/Product

PhantomJS combined with CasperJS is pretty fantastic - it runs a full, headless copy of a Webkit browser so it can operate against a real DOM, execute JavaScript properly, even grab full rendered screenshots of areas of the page but is still easy to automate.

Apart from paid applications like Visual web ripper, Mozenda, Web data extractor, etc., there are plenty of open source methodologies that enables customized web scraping operations. Such as, Selenium, Jsoup, Html Unit Driver (browser-less automation) Some of these have an inbuilt web driver / browser that could imitate even manual browsing to provide efficient web scraping.

Have a look at Why web scrape in .net, which argues the point of why you should scrape in a particular language and you should find most of these arguments will apply to your language of choice too.
 
@Alexander Lau

I don't think dealers will have the time to chase these sites business t the point is that if enough companies successfully win these type of case in court the precedent may make it easier to file a quick injunction to close a scraping site therefore making it harder for these guys to pop up and harass small biz.