0%

安装Scrapy+基本操作

发表于 更新于 分类于 阅读次数：
本文字数： 5.1k 阅读时长 ≈ 5 分钟

2018年1月7日 下午2:19

基本操作

1
2
3
4
5
# Scrapy常用命令
# 创建新项目：scrapy startproject tutorial
# 在项目目录下执行：
# 生成爬虫：scrapy genspider mydomain mydomain.com
# 执行爬虫：scrapy crawl <spider>

遇到的错误

1
2
3
4
5
    xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/Users/czh/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/53/3gvx1k5x1lqdh3ctwsyv99cw0000gn/T/pip-build-8b59p8ev/Twisted/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/53/3gvx1k5x1lqdh3ctwsyv99cw0000gn/T/pip-5dd0tubi-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/53/3gvx1k5x1lqdh3ctwsyv99cw0000gn/T/pip-build-8b59p8ev/Twisted/

解决方法

1
localhost:~ czh$ xcode-select --install

安装过程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
xcode-select: note: install requested for command line developer tools
localhost:~ czh$ pip install Scrapy
Collecting Scrapy
  Using cached Scrapy-1.5.0-py2.py3-none-any.whl
Requirement already satisfied: cssselect>=0.9 in ./anaconda3/lib/python3.6/site-packages (from Scrapy)
Requirement already satisfied: PyDispatcher>=2.0.5 in ./anaconda3/lib/python3.6/site-packages (from Scrapy)
Requirement already satisfied: lxml in ./anaconda3/lib/python3.6/site-packages (from Scrapy)
Requirement already satisfied: queuelib in ./anaconda3/lib/python3.6/site-packages (from Scrapy)
Requirement already satisfied: pyOpenSSL in ./anaconda3/lib/python3.6/site-packages (from Scrapy)
Collecting Twisted>=13.1.0 (from Scrapy)
  Using cached Twisted-17.9.0.tar.bz2
Requirement already satisfied: w3lib>=1.17.0 in ./anaconda3/lib/python3.6/site-packages (from Scrapy)
Collecting parsel>=1.1 (from Scrapy)
  Using cached parsel-1.3.1-py2.py3-none-any.whl
Requirement already satisfied: service-identity in ./anaconda3/lib/python3.6/site-packages (from Scrapy)
Requirement already satisfied: six>=1.5.2 in ./anaconda3/lib/python3.6/site-packages (from Scrapy)
Requirement already satisfied: cryptography>=1.9 in ./anaconda3/lib/python3.6/site-packages (from pyOpenSSL->Scrapy)
Requirement already satisfied: zope.interface>=4.0.2 in ./anaconda3/lib/python3.6/site-packages (from Twisted>=13.1.0->Scrapy)
Requirement already satisfied: constantly>=15.1 in ./anaconda3/lib/python3.6/site-packages (from Twisted>=13.1.0->Scrapy)
Requirement already satisfied: incremental>=16.10.1 in ./anaconda3/lib/python3.6/site-packages (from Twisted>=13.1.0->Scrapy)
Requirement already satisfied: Automat>=0.3.0 in ./anaconda3/lib/python3.6/site-packages (from Twisted>=13.1.0->Scrapy)
Requirement already satisfied: hyperlink>=17.1.1 in ./anaconda3/lib/python3.6/site-packages (from Twisted>=13.1.0->Scrapy)
Requirement already satisfied: pyasn1-modules in ./anaconda3/lib/python3.6/site-packages (from service-identity->Scrapy)
Requirement already satisfied: pyasn1 in ./anaconda3/lib/python3.6/site-packages (from service-identity->Scrapy)
Requirement already satisfied: attrs in ./anaconda3/lib/python3.6/site-packages (from service-identity->Scrapy)
Requirement already satisfied: idna>=2.1 in ./anaconda3/lib/python3.6/site-packages (from cryptography>=1.9->pyOpenSSL->Scrapy)
Requirement already satisfied: asn1crypto>=0.21.0 in ./anaconda3/lib/python3.6/site-packages (from cryptography>=1.9->pyOpenSSL->Scrapy)
Requirement already satisfied: cffi>=1.7 in ./anaconda3/lib/python3.6/site-packages (from cryptography>=1.9->pyOpenSSL->Scrapy)
Requirement already satisfied: setuptools in ./anaconda3/lib/python3.6/site-packages (from zope.interface>=4.0.2->Twisted>=13.1.0->Scrapy)
Requirement already satisfied: pycparser in ./anaconda3/lib/python3.6/site-packages (from cffi>=1.7->cryptography>=1.9->pyOpenSSL->Scrapy)
Building wheels for collected packages: Twisted
  Running setup.py bdist_wheel for Twisted ... done
  Stored in directory: /Users/czh/Library/Caches/pip/wheels/91/c7/95/0bb4d45bc4ed91375013e9b5f211ac3ebf4138d8858f84abbc
Successfully built Twisted
Installing collected packages: Twisted, parsel, Scrapy
Successfully installed Scrapy-1.5.0 Twisted-17.9.0 parsel-1.3.1