【Scrapy】quotes.toscrape.comのスパイダーを作ろう1/3
今回の章
セクション:Building More Advanced Spider with Scrapy
14. Scrapy Advanced Spider - Part 1
15. Scrapy Advanced Spider - Part 2
16. Scrapy Advanced Spider - Part 3
17. Scrapy Advanced Spider - Part 4
18. Scrapy Architecture
今回の目的
https://quotes.toscrape.com/の本文、著者、タグをそれぞれ取得して、Nextページにある記事も取得するSpiderを作成する。
・準備
・Xpathでデータ取集
①本文の抽出方法
②Autherの抽出方法
③タグの取得方法
・pyファイルに書き込んでSpiderを作って動かして見る
・Nextボタン以降の情報の取得方法
・CSVファイルに落とし込む
・アーキテクトについて
・item.pyを設定する。
・pipeline.pyを設定する。
・setting.pyの役割
準備
プロジェクトを始める下準備
$ scrapy startproject test_quotes
$ scrapy genspider test_quotes http://quotes.toscrape.com/
Shellで確認をする
Scrapy shellを使って動作確認をする。
response.xpath('//[@class="quote"]') でとれる確認する。
次にquotes = response.xpath('//[@class="quote"]')と式を入れる
そして、quote[0].extract() とすれば１段目を抽出する。
quote = quotes[0]としても良い。
$ scrapy shell 'http://quotes.toscrape.com/'
In [1]: response.xpath('//*[@class="quote"]')
Out[1]:
[<Selector xpath='//*[@class="quote"]' data='<div class="quote" itemscope itemtype="h'>,In [2]: quotes = response.xpath('//*[@class="quote"]')
In [3]: quote = quotes[0]
In [4]: quote
Out[4]: <Selector xpath='//*[@class="quote"]' data='<div class="quote" itemscope itemtype="h'>In [5]: quote.extract()
Out[5]: '<div class="quote" itemscope itemtype="http://schema.org/CreativeWork">\n <span class="text" itemprop="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”</span>\n <span>by <small class="author" itemprop="author">Albert Einstein</small>\n <a href="/author/Albert-Einstein">(about)</a>\n </span>\n <div class="tags">\n Tags:\n <meta class="keywords" itemprop="keywords" content="change,deep-thoughts,thinking,world"> \n \n <a class="tag" href="/tag/change/page/1/">change</a>\n \n <a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>\n \n <a class="tag" href="/tag/thinking/page/1/">thinking</a>\n \n <a class="tag" href="/tag/world/page/1/">world</a>\n \n </div>\n </div>'
・Xpathでデータ取集
①本文の抽出方法
ここからはresponse.xpathではなく、quote.xpathになる。responseは全てを検索するのに対してquoteは先ほど定義した中からだけになる。
例えば、response.xpath(.//a)でするとページ内のaタグ全て検索する。
一方でquote.xpath(.//a)とすればquote内のaタグのみを収集する。
n [1]: quotes = response.xpath('//*[@class="quote"]')
In [2]: quote = quotes[0]
In [6]: quote.xpath('.//a')
Out[6]:
[<Selector xpath='.//a' data='<a href="/author/Albert-Einstein">(about'>,
<Selector xpath='.//a' data='<a class="tag" href="/tag/change/page/1/'>,
<Selector xpath='.//a' data='<a class="tag" href="/tag/deep-thoughts/'>,
<Selector xpath='.//a' data='<a class="tag" href="/tag/thinking/page/'>,
<Selector xpath='.//a' data='<a class="tag" href="/tag/world/page/1/"'>
次に本文の抽出をする。
@class="text"の後ろに/text()をつける事でより配下の部分だけを取得できる。
['<span class="text" itemprop="text">“を省く事ができる。
In [24]: quote.xpath('.//*[@class="text"]').extract()
Out[24]: ['<span class="text" itemprop="text">“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”</span>']
quote.xpath('//*[@class="text"]/text()').extract()で本文の取得
In [25]: quote.xpath('//*[@class="text"]/text()').extract()
Out[25]:
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
'“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
'“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
'“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”',
"“Imperfection is beauty, madness is genius and it's better to be absolutely ridiculous than absolutely boring.”",
'“Try not to become a man of success. Rather become a man of value.”',
'“It is better to be hated for what you are than to be loved for what you are not.”',
"“I have not failed. I've just found 10,000 ways that won't work.”",
"“A woman is like a tea bag; you never know how strong it is until it's in hot water.”",
'“A day without sunshine is like, you know, night.”']
quote.xpath('.//*[@class="text"]/text()').extract_first()で文字のみ取得する。
In [18]: quote.xpath('.//*[@class="text"]/text()').extract_first()
Out[18]: '“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”'
.// VS //
" . "をつけると現在のノード直下にある<class>要素のみを取得する。
In [7]: quote.xpath('//*[@class="text"]/text()').extract()
Out[7]:
['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”',
'“It is our choices, Harry, that show what we truly are, far more than our abilities.”',
'“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”',
'“The person, be it gentleman or lady, who has not pleasure in a good novel, must be
In [8]: quote.xpath('.//*[@class="text"]/text()').extract()
Out[8]: ['“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”']
②Autherの抽出方法
@itemprop="author"で取得できる。
In [47]: quote.xpath('//*[@itemprop="author"]/text()').extract()
Out[47]:
['Albert Einstein',
'J.K. Rowling',
'Albert Einstein',
'Jane Austen',
'Marilyn Monroe',
'Albert Einstein',
'André Gide',
'Thomas A. Edison',
'Eleanor Roosevelt',
'Steve Martin']
@class="author"でも取得できる。
In [58]: quote.xpath('//*[@class="author"]/text()').extract()
Out[58]:
['Albert Einstein',
'J.K. Rowling',
'Albert Einstein',
'Jane Austen',
'Marilyn Monroe',
'Albert Einstein',
'André Gide',
'Thomas A. Edison',
'Eleanor Roosevelt',
'Steve Martin']
③タグの取得方法
('//*[@itemprop="keywords"]/@content').extract()で取得できる。
In [60]: quote.xpath('//*[@itemprop="keywords"]/@content').extract()
Out[60]:
['change,deep-thoughts,thinking,world',
'abilities,choices',
'inspirational,life,live,miracle,miracles',
'aliteracy,books,classic,humor',
'be-yourself,inspirational',
'adulthood,success,value',
'life,love',
'edison,failure,inspirational,paraphrased',
'misattributed-eleanor-roosevelt',
'humor,obvious,simile']
('.//*[@class="tag"]/text()')でも取得できる。
In [74]: quote.xpath('.//*[@class="tag"]/text()').extract()
Out[74]: ['change', 'deep-thoughts', 'thinking', 'world']
