Environment Setup#

python3/request libraries/parsing libraries/databases/storage libraries/web libraries/app scraping libraries/web scraping frameworks

• python3

  • On Windows 11, you can download it directly from the Store (
  • On Linux: apt-get install python3

• Request libraries

  • requests

    pip3 install requests

  • selenium

    pip install selenium

  • chromeDriver

    1. Check the Chrome version under About
    2. Download the corresponding version from chromeDriver
    3. Add chromeDriver to the environment variables

  • phantomJS

    Newer versions of selenium no longer support phantomJS. You can use chromedriver directly instead.

    Verification:

    1
    from selenium import webdriver
    2
    from selenium.webdriver.chrome.options import Options
    3
    
    4
    chrome_options = Options()
    5
    chrome_options.add_argument('--headless')
    6
    chrome_options.add_argument('--disable-gpu')
    7
    driver = webdriver.Chrome(options=chrome_options)
    8
    driver.get("<https://dreaife.icu/>")
    9
    print(driver.current_url)

  • aiohttp

    pip install aiodns

• Parsing libraries

  • lxml

    pip install lxml

  • beautifulsoup4

    pip install beautifulsoup4

  • pyquery

    pip install pyquery

  • tesserocr

    • Install tesseract

      Windows

    • Install tesserocr

      For Windows, use pip install <name>.whl to install it

    • Verification

      

      1
      import tesserocr
      2
      from PIL import Image
      3
      
      4
      image = Image.open('G:/codeS/backOnGithub/Jupyter/spider/image.png')
      5
      print(tesserocr.image_to_text(image))

      Note: If the File “tesserocr.pyx”, line 2580, in tesserocr._tesserocr.image_to_textRuntimeError: Failed to init API, possibly an invalid tessdata path error occurs, first place tesseract’s test_data in the folder indicated by the error

• Databases

  • MySQL
  • MongoDB
  • Redis

• Storage libraries

  • PyMySQL

    pip install pymysql

  • PyMongo

    pip install pymongo

  • redis-py

    pip install redis

  • RedisDump

    Install ruby

    gem install redis-dump

• Web libraries

  • Flask

    pip install flask

  • Tornado

    pip install tornado

• App scraping libraries

  • charles

  • mitmproxy

    pip install mitmproxy

  • appium

• Web scraping frameworks

  • pyspider

    pip install pyspider

    If it does not run on Windows 11, see this post of mine

  • scrapy

  • scrapy-splash

  • scrapy-redis