淘宝式的爬虫爬取带断点续传代码,Python代码库,德仔网

代码语言

代码分类

【Python】淘宝式的爬虫爬取带断点续传

作者:奕男 / 发布于2018/3/23/ 1050

淘宝数据抓取，按照关键词抓取，带伪装协议头，

淘宝数据抓取，按照关键词抓取，带伪装协议头，
import requests
import re
import pymysql as pq

import pandas



def To_obtain():
    db = pq.connect(host='localhost', user='root', password='a83822851', db='hexun')
    curs = db.cursor()
    table = 'SHOW TABLES;'
    curs.execute(table)
    TABLE = curs.fetchall()
    print(TABLE)

    for keys in TABLE:
        rangdis = 0
        sql = "SELECT  * FROM {}".format(keys[0])
        curs.execute(sql)
        results = curs.fetchall()
        print(results)
        for i in results:
            urlid = i[0]
            url = i[2]
            Git_id(keys[0],urlid,url)

            rangdis = rangdis + 1
            if rangdis >= 300 and rangdis < 301:
                print("当前的值为"+str(rangdis))
                pass




def Git_id(keys,id,url):
    cont = [keys,id,url]
    pds = pandas.DataFrame(cont,index=False)
    pds.to_csv('nams.csv',encoding='utf-8')









To_obtain()

bbs.py

import requests
import re
import pymysql as pq

import pandas



def To_obtain():
    db = pq.connect(host='localhost', user='root', password='a83822851', db='hexun')
    curs = db.cursor()
    table = 'SHOW TABLES;'
    curs.execute(table)
    TABLE = curs.fetchall()
    print(TABLE)

    for keys in TABLE:
        rangdis = 0
        sql = "SELECT  * FROM {}".format(keys[0])
        curs.execute(sql)
        results = curs.fetchall()
        print(results)
        for i in results:
            urlid = i[0]
            url = i[2]
            Git_id(keys[0],urlid,url)

            rangdis = rangdis + 1
            if rangdis >= 300 and rangdis < 301:
                print("当前的值为"+str(rangdis))
                pass




def Git_id(keys,id,url):
    cont = [keys,id,url]
    pds = pandas.DataFrame(cont,index=False)
    pds.to_csv('nams.csv',encoding='utf-8')









To_obtain()

试试其它关键字

同语言下

可能有用的

奕男贡献的其它代码(13)