Skip to main content

Static Code Analysis

black / autopep8 / yapf (Auto formatters)​

https://www.kevinpeters.net/auto-formatters-for-python

isort​

pip install isort
brew install isort

isort -rc .
isort **/*.py
isort -rc -sl **/*.py

Skipping
 # isort:skip
 # isort:skip_file

# isort.cfg
 [settings]
 line_length = 88
 multi_line_output = 3
 include_trailing_comma = True
 known_third_party = celery,django,environ,pyquery,pytz,redis,requests,rest_framework

pylint​

Pylint is a Python static code analysis tool which looks for programming errors, helps enforcing a coding standard, sniffs for code smells and offers simple refactoring suggestions.

pylint is one of the most wide-spread linters in Python. The features of pylint for sure overlaps with Flake8, but there is one feature I love: Checking for code duplication

pip install pylint

pylint --disable=all --enable=duplicate-code .

pip install pylint-django

pylint --disable=all --enable=duplicate-code --load-plugins pylint_django .

pyflakes​

autoflake​

autoflake removes unused imports and unused variables from Python code. It makes use of pyflakes to do this.

pip install autoflake
autoflake -r --in-place --remove-unused-variables .
autoflake -r --in-place --remove-unused-variables --remove-all-unused-imports **/*.py

mypy (static types)​

Mypy is an optional static type checker for Python. You can add type hints (PEP 484) to your Python programs, and use mypy to type check them statically. Find bugs in your programs without even running them!

You can mix dynamic and static typing in your programs. You can always fall back to dynamic typing when static typing is not convenient, such as for legacy code.

https://github.com/python/mypy

http://mypy-lang.org

https://medium.com/analytics-vidhya/type-annotations-in-python-3-8-3b401384403d

https://sourcery.ai/blog/python-best-practices

Black​

Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return,Blackgives you speed, determinism, and freedom frompycodestylenagging about formatting. You will save time and mental energy for more important matters.

Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.

Blackmakes code review faster by producing the smallest diffs possible.

pip install black
black <file_path>
black .

https://github.com/psf/black

Pyre type-checker​

Pyre is a performant type checker for Python compliant with PEP 484. Pyre can analyze codebases with millions of lines of code incrementally -- providing instantaneous feedback to developers as they write code.

Pyre ships withPysa, a security focused static analysis tool we've built on top of Pyre that reasons about data flows in Python applications.

https://pyre-check.org

https://github.com/facebook/pyre-check

pre-commit / precommit​

Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements. By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.

As we created more libraries and projects we recognized that sharing our pre-commit hooks across projects is painful. We copied and pasted unwieldy bash scripts from project to project and had to manually change the hooks to work for different project structures.

We believe that you should always use the best industry standard linters. Some of the best linters are written in languages that you do not use in your project or have installed on your machine. For example scss-lint is a linter for SCSS written in Ruby. If you're writing a project in node you should be able to use scss-lint as a pre-commit hook without adding a Gemfile to your project or understanding how to get scss-lint installed.

We built pre-commit to solve our hook issues. It is a multi-language package manager for pre-commit hooks. You specify a list of hooks you want and pre-commit manages the installation and execution of any hook written in any language before every commit. pre-commit is specifically designed to not require root access. If one of your developers doesn't have node installed but modifies a JavaScript file, pre-commit automatically handles downloading and building node to run eslint without root.

.pre-commit-config.yaml
exclude: '^$'
fail_fast: false
exclude: '^(?!tests/)$' #run only test folder files
files: ^API/

 repos:
     - repo: https://github.com/timothycrosley/isort
       rev: 5.6.4
       hooks:
       - id: isort
     - repo: https://github.com/pre-commit/pre-commit-hooks
       rev: v3.2.0
       hooks:
         - id: check-ast
         - id: check-byte-order-marker
         - id: check-docstring-first
         - id: check-executables-have-shebangs
         - id: check-json
         - id: debug-statements
         - id: mixed-line-ending
         - id: check-added-large-files
         - id: trailing-whitespace
         - id: end-of-file-fixer
         - id: check-yaml
    args: [--allow-multiple-documents, --unsafe]
    exclude: /templates/
         - id: requirements-txt-fixer
         - id: check-merge-conflict
         - id: check-case-conflict
         - id: detect-aws-credentials
         - id: detect-private-key
         - id: no-commit-to-branch
     - repo: https://github.com/PyCQA/flake8
       rev: 6.0.0
       hooks:
       - id: flake8
         additional_dependencies: [
             'flake8-bandit',
             'pep8-naming',
             'flake8-sfs',
             'flake8-print',
             'flake8-bugbear',
             'flake8-comprehensions',
             'flake8-debugger',
             'flake8-deprecated',
             'flake8-docstrings',
             'flake8-quotes',
             'flake8-eradicate',
         ]
     - repo: https://github.com/pycqa/pylint
       rev: pylint-2.6.0
       hooks:
       - id: pylint
     - repo: https://github.com/psf/black
       rev: stable
       hooks:
         - id: black
           language_version: python3.8
    - repo: https://github.com/igorshubovych/markdownlint-cli
      rev: v0.32.2
      hooks:
      - id: markdownlint
      - id: markdownlint-fix
    - repo: https://github.com/asottile/pyupgrade
    rev: v2.7.2
    hooks:
    - id: pyupgrade
    args: [--py36-plus]
    - repo: https://github.com/asottile/blacken-docs
    rev: v1.8.0
    hooks:
    - id: blacken-docs
    additional_dependencies: [black==20.8b1]>)

Running custom shell command​

.pre-commit-config.yaml
- repo: local
hooks:
  - id: docusaurus-mdx-checker
 name: docusaurus-mdx-checker
 entry: docusaurus-mdx-checker.sh
 language: script
 pass_filenames: false
docusaurus-mdx-checker.md
#!/bin/bash
echo "Running docusaurus-mdx-checker"
npx docusaurus-mdx-checker

# chmod +x docusaurus-mdx-checker.md

Custom Python/Shell Script for pre-commit | by shimo | Medium

Running pre-commit​

# pip install pre-commit
brew install pre-commit

# to install pre-commit as git webhook locally
# pre-commit install

pre-commit run --all-files
pre-commit autoupdate

# run pre-commit on specific files in a folder
git ls-files -- Technologies/Technologies | xargs pre-commit run --files

# bypass installed webhooks
git commit --no-verify

# isort run
isort **/*.py

# autoflake run
autoflake -r --in-place --remove-unused-variables --remove-all-unused-imports **/*.py

black .

https://medium.com/staqu-dev-logs/keeping-python-code-clean-with-pre-commit-hooks-black-flake8-and-isort-cac8b01e0ea1

https://pre-commit.com/hooks.html GitHub - igorshubovych/markdownlint-cli: MarkdownLint Command Line Interface

https://pre-commit.com/

https://github.com/pre-commit/pre-commit-hooks

flake8​

flake8 --max-line-length=88 src

  • files that contain this line are skipped: flake8: noqa

  • lines that contain a # noqa comment at the end will not issue warnings.

  • you can ignore specific errors on a line with # noqa: <error>, e.g., # noqa: E234. Multiple codes can be given, separated by comma. Thenoqatoken is case insensitive, the colon before the list of codes is required otherwise the part afternoqais ignored

  • E***/W***:pep8 errors and warnings

  • F***:PyFlakes codes (see below)

  • C9**:McCabe complexity plugin mccabe

  • N8**:Naming Conventions plugin pep8-naming

# filename - .pre-commit-config.yaml
additional_dependencies: [
            'flake8-isort==2.7.0',
            'flake8-pep3101==1.2.1',
            'flake8-polyfill==1.0.2',
            'flake8-string-format==0.2.3',
        ]

# filename - .flake8
[flake8]
# ignore = E203, E266, E501, F403, F401, F541
ignore = E231, W503
max-line-length = 119
max-complexity = 18
select = B,C,E,F,W,T4,B9,N8

Here are some of the interesting flake8 plugins:

Security​

  • flake8-bandit: Security Testing
  • flake8-bugbear: finding likely bugs and design problems in your program - usually it's silent, but when it's not you should have a look 🐻
  • flake8-requests: checks usage of the request framework

Flake8: Remove Debugging Artifacts​

Let Dead Code Die​

pip install vulture
vulture myscript.py
vulture . --exclude .history

https://github.com/jendrikseipp/vulture

Flake8: Nudging Yourself to use Good Style​

  • flake8-comprehensions: Helps you write better list/set/dict comprehensions - I love this one 😍
  • flake8-executable: Check executable permissions and shebangs. Files should either executable and have a shebang, or not be executable and not have a shebang.
  • flake8-raise: Finds improvements for raise statements
  • flake8-pytest: Use assert instead of assertEqual

Modern style Python​

Improve Flake8​

Autoformatters​

  • Prettier: HTML, CSS, JavaScript, GraphQL, and many more.
  • Clang-format: C, C++, Java, JavaScript, Objective-C, Protobuf, C#
  • Rustfmt: Rust

Bandit​

Bandit is a tool designed to find common security issues in Python code. To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.

https://pypi.org/project/bandit

mypy​

Mypy is an optional static type checker for Python that aims to combine the benefits of dynamic (or "duck") typing and static typing. Mypy combines the expressive power and convenience of Python with a powerful type system and compile-time type checking. Mypy type checks standard Python programs; run them using any Python VM with basically no runtime overhead.

http://mypy-lang.org

Code Complexity​

radon​

$ pip install radon
$ radon cc mpu/aws.py -s
mpu/aws.py
F 85:0 s3_download - B (6)
F 16:0 list_files - A (3)
F 165:0 _s3_path_split - A (2)
F 46:0 s3_read - A (1)
F 141:0 s3_upload - A (1)
C 77:0 ExistsStrategy - A (1)

The first letter shows the type of block (F for function, C for class). Then radon gives theline number, thenameof the class/function, agrade(A, B, C, D, E, or F), and the actualcomplexity as a number. Typically, a complexity below 10 is ok.The most complex part of scipy has a complexity of 61.

Besides radon, there are various other packages and Flake8 plugins:

https://towardsdatascience.com/static-code-analysis-for-python-bdce10b8d287

Linting & formatting - https://realpython.com/python-pep8

Pep8 Naming Convention for files​

  • modules (filenames) should have short, all-lowercase names, and they can contain underscores;
  • packages (directories) should have short, all-lowercase names, preferably without underscores;
  • classes should use the CapWords convention.

https://www.youtube.com/watch?v=4klj8UYPZxY&ab_channel=freeCodeCampTalks

Others​

GitHub - Instagram/Fixit: Advanced Python linting framework with auto-fixes and hierarchical configuration that makes it easy to write custom in-repo lint rules.

Fixit: linting framework with auto-fixes - Fixit documentation