远程服务器返回错误: (502) 错误的网关。

handler

求助， 爬取http://www.peugeot.com.au/find-a-dealer 的时候，post了一个4000的邮编，
从浏览器里能看到XHR显示返回的内容，但是用爬虫抓取就会返回The remote server returned an error: (502) Bad Gateway.

请问飞哥怎么回事？？
@站长苏飞

站长苏飞

这网站超级慢，你提供下抓包数据和代码，看下情况

handler

站长苏飞 发表于 2015-9-9 13:51
这网站超级慢，你提供下抓包数据和代码，看下情况

抓包数据
General

• Remote Address:
  10.140.1.110:8080
• Request URL:
  http://www.peugeot.com.au/find-a-dealer/
• Request Method:
  POST
• Status Code:
  
  200 OK
  
• Response Headersview source
  
  • Age:
    0
  • Cache-Control:
    no-store, no-cache, must-revalidate, post-check=0, pre-check=0
  • Connection:
    keep-alive
  • Content-Type:
    application/json
  • Date:
    Wed, 09 Sep 2015 05:23:56 GMT
  • Expires:
    Thu, 19 Nov 1981 08:52:00 GMT
  • Pragma:
    no-cache
  • Set-Cookie:
    sid=b0629df249f31c35a05b7d81519c10156155a597; path=/
  • Transfer-Encoding:
    chunked
  • X-Cache:
    MISS from MELSQUID110
  • X-Cache:
    MISS from SHGSQUID110
  • X-Cache-Lookup:
    MISS from SHGSQUID110:8080
  • X-Cache-Lookup:
    MISS from MELSQUID110:8080
  • X-UA-Compatible:
    IE=edge
  • X-Varnish:
    1074687279
    
• Request Headersview source
  
  • Accept:
    application/json
  • Accept-Encoding:
    gzip, deflate
  • Accept-Language:
    zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4,ja;q=0.2,pt-BR;q=0.2,pt;q=0.2
  • Cache-Control:
    no-cache
  • Content-Length:
    94
  • Content-type:
    application/x-www-form-urlencoded; charset=UTF-8
  • Cookie:
    sid=b0629df249f31c35a05b7d81519c10156155a597
  • DNT:
    1
  • Host:
    www.peugeot.com.au
  • Origin:
    http://www.peugeot.com.au
  • Pragma:
    no-cache
  • Proxy-Connection:
    keep-alive
  • Referer:
    http://www.peugeot.com.au/find-a-dealer/
  • User-Agent:
    Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36
  • X-Request:
    JSON
  • X-Requested-With:
    XMLHttpRequest
    
• Form Dataview sourceview URL encoded
  
  • type:
    address
  • geofindlat:
  • geofindlng:
  • geoloc:
  • main_search:
    4000
  • name_search:
  • range:
    300
  • ajax:
    true
    

NamePath

[tr=rgb(212, 212, 212)][/tr]




代码

[C#] 纯文本查看 复制代码

                 HttpHelper httpHelper = new HttpHelper();                HttpItem item = new HttpItem()
                {
                    Method = "post",
                    URL = "http://www.peugeot.com.au/find-a-dealer/",
                    Postdata = "type=address&geofindlat=&geofindlng=&geoloc=&main_search=4000&name_search=&range=300&ajax=true",
                    Encoding = Encoding.UTF8,
                    ProxyIp = "tkysquid110:8080",
                    Referer = "http://www.peugeot.com.au/find-a-dealer/",
                    UserAgent="Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36",
                    //ResultType="JSON"

                };

                 collectedPageStr = httpHelper.GetHtml(item).Html;

站长苏飞

tkysquid110:8080 这地址不对吧。

handler

站长苏飞 发表于 2015-9-9 14:03
tkysquid110:8080 这地址不对吧。

这是公司的代理，如果不用代理就直接返回 Unable to connect to the remote server
改成了10.140.1.110:8080， 也是(502) Bad Gateway.