周期性抓取网页内容代码,Python代码库,德仔网

代码语言

代码分类

【Python】周期性抓取网页内容

作者:intergret / 发布于2012/3/6/ 793

1.使用sched模块可以周期性地执行指定函数 2.在周期性执行指定函数中抓取指定网页，并解析出想要的网页内容，代码中是六维论坛的在线人数

<div>coding=utf-8</div>
import time,sched,os,urllib2,re,string</div>
<div></div>
<div>初始化sched模块的scheduler类</div>
<div>第一个参数是一个可以返回时间戳的函数，第二个参数可以在定时未到达之前阻塞。</div>
<div>s = sched.scheduler(time.time,time.sleep)</div>
<div></div>
<div>被周期性调度触发的函数</div>
<div>def event_func():</div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span>req = urllib2.Request('http://bt.neu6.edu.cn/')</div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span>response = urllib2.urlopen(req)</div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span>rawdata = response.read()</div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span>response.close()</div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span></div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span>usernump = re.compile(r'总计 <em>.*?</em> 人在线')</div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span>usernummatch = usernump.findall(rawdata)</div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span>if usernummatch:</div>
<div><span class="Apple-tab-span" style="white-space:pre">		</span>currentnum=usernummatch[0]</div>
<div><span class="Apple-tab-span" style="white-space:pre">		</span>currentnum=currentnum[string.index(currentnum,'>')+1:string.rindex(currentnum,'<')]</div>
<div><span class="Apple-tab-span" style="white-space:pre">		</span>print "Current Time:",time.strftime('%Y,%m,%d,%H,%M',time.localtime(time.time())),'User num:',currentnum</div>
<div><span class="Apple-tab-span" style="white-space:pre">		</span></div>
<div><span class="Apple-tab-span" style="white-space:pre">		</span># 保存结果，供图表工具amcharts使用</div>
<div><span class="Apple-tab-span" style="white-space:pre">		</span>result=open('liuvUserNUm','a')</div>
<div><span class="Apple-tab-span" style="white-space:pre">		</span>result.write('{year: new Date('+time.strftime('%Y,%m,%d,%H,%M',time.localtime(time.time()))+'),value:'+currentnum+'},\n')</div>
<div><span class="Apple-tab-span" style="white-space:pre">		</span>result.close()</div>
<div></div>
<div>enter四个参数分别为：间隔事件、优先级（用于同时间到达的两个事件同时执行时定序）、被调用触发的函数，给他的参数（注意：一定要以tuple给如，如果只有一个参数就(xx,)）</div>
<div>def perform(inc):</div>
<div>  s.enter(inc,0,perform,(inc,))</div>
<div>  event_func()</div>
<div> </div>
<div>def mymain(inc=900):</div>
<div>  s.enter(0,0,perform,(inc,))</div>
<div>  s.run()</div>
<div></div>
if __name__ == "__main__":</div>
<div><span class="Apple-tab-span" style="white-space:pre">	</span>mymain()</div>

试试其它关键字

　抓取网页内容　

同语言下

可能有用的

intergret贡献的其它代码(4)