diff options
author | Johannes 'josch' Schauer <josch@debian.org> | 2020-02-01 01:11:55 +0100 |
---|---|---|
committer | Johannes 'josch' Schauer <josch@debian.org> | 2020-02-01 01:11:55 +0100 |
commit | 61b98ca52e8d48a6ad3b4baed5feb4b38ee53804 (patch) | |
tree | 9f7ab24aabb0a3b25d125df81fff058ab5e63fe9 /docs/dev | |
parent | 3e49246c2e44159486ea66fed3757cdb4e4d0c50 (diff) |
New upstream version 0.16.0+dfsg1
Diffstat (limited to 'docs/dev')
-rw-r--r-- | docs/dev/contribution_guide.rst | 180 | ||||
-rw-r--r-- | docs/dev/csv_table.txt | 6 | ||||
-rw-r--r-- | docs/dev/engine_overview.rst | 267 | ||||
-rw-r--r-- | docs/dev/hello.dot | 3 | ||||
-rw-r--r-- | docs/dev/index.rst | 15 | ||||
-rw-r--r-- | docs/dev/makefile.rst | 221 | ||||
-rw-r--r-- | docs/dev/plugins.rst | 54 | ||||
-rw-r--r-- | docs/dev/quickstart.rst | 132 | ||||
-rw-r--r-- | docs/dev/reST.rst | 1428 | ||||
-rw-r--r-- | docs/dev/search_api.rst | 120 | ||||
-rw-r--r-- | docs/dev/svg_image.svg | 10 | ||||
-rw-r--r-- | docs/dev/translation.rst | 71 |
12 files changed, 2507 insertions, 0 deletions
diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst new file mode 100644 index 0000000..459dfb4 --- /dev/null +++ b/docs/dev/contribution_guide.rst @@ -0,0 +1,180 @@ +.. _how to contribute: + +================= +How to contribute +================= + +Prime directives: Privacy, Hackability +====================================== + +Searx has two prime directives, **privacy-by-design and hackability** . The +hackability comes in three levels: + +- support of search engines +- plugins to alter search behaviour +- hacking searx itself + +Note the lack of "world domination" among the directives. Searx has no +intention of wide mass-adoption, rounded corners, etc. The prime directive +"privacy" deserves a separate chapter, as it's quite uncommon unfortunately. + +Privacy-by-design +----------------- + +Searx was born out of the need for a **privacy-respecting** search tool which +can be extended easily to maximize both, its search and its privacy protecting +capabilities. + +A few widely used features work differently or turned off by default or not +implemented at all **as a consequence of privacy-by-design**. + +If a feature reduces the privacy preserving aspects of searx, it should be +switched off by default or should not implemented at all. There are plenty of +search engines already providing such features. If a feature reduces the +protection of searx, users must be informed about the effect of choosing to +enable it. Features that protect privacy but differ from the expectations of +the user should also be explained. + +Also, if you think that something works weird with searx, it's might be because +of the tool you use is designed in a way to interfere with the privacy respect. +Submitting a bugreport to the vendor of the tool that misbehaves might be a good +feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST`` +requests in various browsers). + +Remember the other prime directive of searx is to be hackable, so if the above +privacy concerns do not fancy you, simply fork it. + + *Happy hacking.* + +Code +==== + +.. _PEP8: https://www.python.org/dev/peps/pep-0008/ +.. _Conventional Commits: https://www.conventionalcommits.org/ +.. _Git Commit Good Practice: https://wiki.openstack.org/wiki/GitCommitMessages +.. _Structural split of changes: + https://wiki.openstack.org/wiki/GitCommitMessages#Structural_split_of_changes +.. _gitmoji: https://gitmoji.carloscuesta.me/ +.. _Semantic PR: https://github.com/zeke/semantic-pull-requests + +.. sidebar:: Create good commits! + + - `Structural split of changes`_ + - `Conventional Commits`_ + - `Git Commit Good Practice`_ + - some like to use: gitmoji_ + - not yet active: `Semantic PR`_ + +In order to submit a patch, please follow the steps below: + +- Follow coding conventions. + + - PEP8_ standards apply, except the convention of line length + - Maximum line length is 120 characters + +- The cardinal rule for creating good commits is to ensure there is only one + *logical change* per commit / read `Structural split of changes`_ + +- Check if your code breaks existing tests. If so, update the tests or fix your + code. + +- If your code can be unit-tested, add unit tests. + +- Add yourself to the :origin:`AUTHORS.rst` file. + +- Choose meaning full commit messages, read `Conventional Commits`_ + + .. code:: + + <type>[optional scope]: <description> + + [optional body] + + [optional footer(s)] + +- Create a pull request. + +For more help on getting started with searx development, see :ref:`devquickstart`. + + +Translation +=========== + +Translation currently takes place on :ref:`transifex <translation>`. + +.. caution:: + + Please, do not update translation files in the repo. + + +.. _contrib docs: + +Documentation +============= + +.. _Sphinx: http://www.sphinx-doc.org +.. _reST: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html + +.. sidebar:: The reST sources + + has been moved from ``gh-branch`` into ``master`` (:origin:`docs`). + +The documentation is built using Sphinx_. So in order to be able to generate +the required files, you have to install it on your system. Much easier, use +our :ref:`makefile`. + +Here is an example which makes a complete rebuild: + +.. code:: sh + + $ make docs-clean docs + ... + The HTML pages are in dist/docs. + +.. _make docs-live: + +live build +---------- + +.. sidebar:: docs-clean + + It is recommended to assert a complete rebuild before deploying (use + ``docs-clean``). + +Live build is like WYSIWYG. If you want to edit the documentation, its +recommended to use. The Makefile target ``docs-live`` builds the docs, opens +URL in your favorite browser and rebuilds every time a reST file has been +changed. + +.. code:: sh + + $ make docs-live + ... + The HTML pages are in dist/docs. + ... Serving on http://0.0.0.0:8080 + ... Start watching changes + + +.. _deploy on github.io: + +deploy on github.io +------------------- + +To deploy documentation at :docs:`github.io <.>` use Makefile target +:ref:`make gh-pages`, which will builds the documentation, clones searx into a sub +folder ``gh-pages``, cleans it, copies the doc build into and runs all the +needed git add, commit and push: + +.. code:: sh + + $ make docs-clean gh-pages + ... + SPHINX docs --> file://<...>/dist/docs + The HTML pages are in dist/docs. + ... + Cloning into 'gh-pages' ... + ... + cd gh-pages; git checkout gh-pages >/dev/null + Switched to a new branch 'gh-pages' + ... + doc available at --> https://asciimoo.github.io/searx diff --git a/docs/dev/csv_table.txt b/docs/dev/csv_table.txt new file mode 100644 index 0000000..8a14541 --- /dev/null +++ b/docs/dev/csv_table.txt @@ -0,0 +1,6 @@ +stub col row 1, column, "loremLorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy +eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam +voluptua." +stub col row 1, "At vero eos et accusam et justo duo dolores et ea rebum. Stet clita +kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.", column +stub col row 1, column, column diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst new file mode 100644 index 0000000..449c837 --- /dev/null +++ b/docs/dev/engine_overview.rst @@ -0,0 +1,267 @@ + +.. _engines-dev: + +=============== +Engine overview +=============== + +.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine + +searx is a metasearch-engine_, so it uses different search engines to provide +better results. + +Because there is no general search API which could be used for every search +engine, an adapter has to be built between searx and the external search +engines. Adapters are stored under the folder :origin:`searx/engines`. + +.. contents:: + :depth: 3 + :backlinks: entry + +general engine configuration +============================ + +It is required to tell searx the type of results the engine provides. The +arguments can be set in the engine file or in the settings file +(normally ``settings.yml``). The arguments in the settings file override +the ones in the engine file. + +It does not matter if an option is stored in the engine file or in the +settings. However, the standard way is the following: + +.. _engine file: + +engine file +----------- + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +categories list pages, in which the engine is working +paging boolean support multible pages +language_support boolean support language choosing +time_range_support boolean support search time range +offline boolean engine runs offline +======================= =========== =========================================== + +.. _engine settings: + +settings.yml +------------ + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +name string name of search-engine +engine string name of searx-engine + (filename without ``.py``) +shortcut string shortcut of search-engine +timeout string specific timeout for search-engine +======================= =========== =========================================== + + +overrides +--------- + +A few of the options have default values in the engine, but are often +overwritten by the settings. If ``None`` is assigned to an option in the engine +file, it has to be redefined in the settings, otherwise searx will not start +with that engine. + +The naming of overrides is arbitrary. But the recommended overrides are the +following: + +======================= =========== =========================================== +argument type information +======================= =========== =========================================== +base_url string base-url, can be overwritten to use same + engine on other URL +number_of_results int maximum number of results per request +language string ISO code of language and country like en_US +api_key string api-key if required by engine +======================= =========== =========================================== + +example code +------------ + +.. code:: python + + # engine dependent config + categories = ['general'] + paging = True + language_support = True + + +making a request +================ + +To perform a search an URL have to be specified. In addition to specifying an +URL, arguments can be passed to the query. + +passed arguments +---------------- + +These arguments can be used to construct the search query. Furthermore, +parameters with default value can be redefined for special purposes. + +====================== ============ ======================================================================== +argument type default-value, information +====================== ============ ======================================================================== +url string ``''`` +method string ``'GET'`` +headers set ``{}`` +data set ``{}`` +cookies set ``{}`` +verify boolean ``True`` +headers.User-Agent string a random User-Agent +category string current category, like ``'general'`` +started datetime current date-time +pageno int current pagenumber +language string specific language code like ``'en_US'``, or ``'all'`` if unspecified +====================== ============ ======================================================================== + +parsed arguments +---------------- + +The function ``def request(query, params):`` always returns the ``params`` +variable. Inside searx, the following paramters can be used to specify a search +request: + +============ =========== ========================================================= +argument type information +============ =========== ========================================================= +url string requested url +method string HTTP request method +headers set HTTP header information +data set HTTP data information (parsed if ``method != 'GET'``) +cookies set HTTP cookies +verify boolean Performing SSL-Validity check +============ =========== ========================================================= + + +example code +------------ + +.. code:: python + + # search-url + base_url = 'https://example.com/' + search_string = 'search?{query}&page={page}' + + # do search-request + def request(query, params): + search_path = search_string.format( + query=urlencode({'q': query}), + page=params['pageno']) + + params['url'] = base_url + search_path + + return params + + +returned results +================ + +Searx is able to return results of different media-types. Currently the +following media-types are supported: + +- default_ +- images_ +- videos_ +- torrent_ +- map_ + +To set another media-type as default, the parameter ``template`` must be set to +the desired type. + +default +------- + +========================= ===================================================== +result-parameter information +========================= ===================================================== +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, time of publish +========================= ===================================================== + +images +------ + +To use this template, the parameter: + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``images.html`` +url string, url to the result site +title string, title of the result *(partly implemented)* +content *(partly implemented)* +publishedDate :py:class:`datetime.datetime`, + time of publish *(partly implemented)* +img\_src string, url to the result image +thumbnail\_src string, url to a small-preview image +========================= ===================================================== + +videos +------ + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``videos.html`` +url string, url of the result +title string, title of the result +content *(not implemented yet)* +publishedDate :py:class:`datetime.datetime`, time of publish +thumbnail string, url to a small-preview image +========================= ===================================================== + +torrent +------- + +.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme + +========================= ===================================================== +result-parameter information +========================= ===================================================== +template is set to ``torrent.html`` +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, + time of publish *(not implemented yet)* +seed int, number of seeder +leech int, number of leecher +filesize int, size of file in bytes +files int, number of files +magnetlink string, magnetlink_ of the result +torrentfile string, torrentfile of the result +========================= ===================================================== + + +map +--- + +========================= ===================================================== +result-parameter information +========================= ===================================================== +url string, url of the result +title string, title of the result +content string, general result-text +publishedDate :py:class:`datetime.datetime`, time of publish +latitude latitude of result (in decimal format) +longitude longitude of result (in decimal format) +boundingbox boundingbox of result (array of 4. values + ``[lat-min, lat-max, lon-min, lon-max]``) +geojson geojson of result (http://geojson.org) +osm.type type of osm-object (if OSM-Result) +osm.id id of osm-object (if OSM-Result) +address.name name of object +address.road street name of object +address.house_number house number of object +address.locality city, place of object +address.postcode postcode of object +address.country country of object +========================= ===================================================== diff --git a/docs/dev/hello.dot b/docs/dev/hello.dot new file mode 100644 index 0000000..504621d --- /dev/null +++ b/docs/dev/hello.dot @@ -0,0 +1,3 @@ +graph G { + Hello -- World +} diff --git a/docs/dev/index.rst b/docs/dev/index.rst new file mode 100644 index 0000000..cb913a8 --- /dev/null +++ b/docs/dev/index.rst @@ -0,0 +1,15 @@ +======================= +Developer documentation +======================= + +.. toctree:: + :maxdepth: 1 + + quickstart + contribution_guide + engine_overview + search_api + plugins + translation + makefile + reST diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst new file mode 100644 index 0000000..f595700 --- /dev/null +++ b/docs/dev/makefile.rst @@ -0,0 +1,221 @@ +.. _makefile: + +================ +Makefile Targets +================ + +.. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction + +.. sidebar:: build environment + + Before looking deeper at the targets, first read about :ref:`makefile setup` + and :ref:`make pyenv`. + +With the aim to simplify development cycles, started with :pull:`1756` a +``Makefile`` based boilerplate was added. If you are not familiar with +Makefiles, we recommend to read gnu-make_ introduction. + +The usage is simple, just type ``make {target-name}`` to *build* a target. +Calling the ``help`` target gives a first overview:: + + $ make help + test - run developer tests + docs - build documentation + docs-live - autobuild HTML documentation while editing + run - run developer instance + install - developer install (./local) + uninstall - uninstall (./local) + gh-pages - build docs & deploy on gh-pages branch + clean - drop builds and environments + ... + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + + +.. _makefile setup: + +Setup +===== + +.. _git stash: https://git-scm.com/docs/git-stash + +The main setup is done in the :origin:`Makefile`:: + + export GIT_URL=https://github.com/asciimoo/searx + export SEARX_URL=https://searx.me + export DOCS_URL=https://asciimoo.github.io/searx + +.. sidebar:: fork & upstream + + Commit changes in your (local) branch, fork or whatever, but do not push them + upstream / `git stash`_ is your friend. + +:GIT_URL: Changes this, to point to your searx fork. + +:SEARX_URL: Changes this, to point to your searx instance. + +:DOCS_URL: If you host your own (branded) documentation, change this URL. + +.. _make pyenv: + +Python environment +================== + +.. sidebar:: activate environment + + ``source ./local/py3/bin/activate`` + +With Makefile we do no longer need to build up the virualenv manually (as +described in the :ref:`devquickstart` guide). Jump into your git working tree +and release a ``make pyenv``: + +.. code:: sh + + $ cd ~/searx-clone + $ make pyenv + PYENV usage: source ./local/py3/bin/activate + ... + +With target ``pyenv`` a development environment (aka virtualenv) was build up in +``./local/py3/``. To make a *developer install* of searx (:origin:`setup.py`) +into this environment, use make target ``install``: + +.. code:: sh + + $ make install + PYENV usage: source ./local/py3/bin/activate + PYENV using virtualenv from ./local/py3 + PYENV install . + +You have never to think about intermediate targets like ``pyenv`` or +``install``, the ``Makefile`` chains them as requisites. Just run your main +target. + +.. sidebar:: drop environment + + To get rid of the existing environment before re-build use :ref:`clean target + <make clean>` first. + +If you think, something goes wrong with your ./local environment or you change +the :origin:`setup.py` file (or the requirements listed in +:origin:`requirements-dev.txt` and :origin:`requirements.txt`), you have to call +:ref:`make clean`. + + +.. _make run: + +``make run`` +============ + +To get up a running a developer instance simply call ``make run``. This enables +*debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py`` +instance, disables *debug* option again and opens the URL in your favorite WEB +browser (:man:`xdg-open`): + +.. code:: sh + + $ make run + PYENV usage: source ./local/py3/bin/activate + PYENV install . + ./local/py3/bin/python ./searx/webapp.py + ... + INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit) + ... + +.. _make clean: + +``make clean`` +============== + +Drop all intermediate files, all builds, but keep sources untouched. Includes +target ``pyclean`` which drops ./local environment. Before calling ``make +clean`` stop all processes using :ref:`make pyenv`. + +.. code:: sh + + $ make clean + CLEAN pyclean + CLEAN clean + +.. _make docs: + +``make docs docs-live docs-clean`` +================================== + +We describe the usage of the ``doc*`` targets in the :ref:`How to contribute / +Documentation <contrib docs>` section. If you want to edit the documentation +read our :ref:`make docs-live` section. If you are working in your own brand, +adjust your :ref:`Makefile setup <makefile setup>`. + + +.. _make gh-pages: + +``make gh-pages`` +================= + +To deploy on github.io first adjust your :ref:`Makefile setup <makefile +setup>`. For any further read :ref:`deploy on github.io`. + +.. _make test: + +``make test`` +============= + +Runs a series of tests: ``test.pep8``, ``test.unit``, ``test.robot`` and does +additional :ref:`pylint checks <make pylint>`. You can run tests selective, +e.g.: + +.. code:: sh + + $ make test.pep8 test.unit + . ./local/py3/bin/activate; ./manage.sh pep8_check + [!] Running pep8 check + . ./local/py3/bin/activate; ./manage.sh unit_tests + [!] Running unit tests + +.. _make pylint: + +``make pylint`` +=============== + +.. _Pylint: https://www.pylint.org/ + +Before commiting its recommend to do some (more) linting. Pylint_ is known as +one of the best source-code, bug and quality checker for the Python programming +language. Pylint_ is not yet a quality gate within our searx project (like +:ref:`test.pep8 <make test>` it is), but Pylint_ can help to improve code +quality anyway. The pylint profile we use at searx project is found in +project's root folder :origin:`.pylintrc`. + +Code quality is a ongoing process. Don't try to fix all messages from Pylint, +run Pylint and check if your changed lines are bringing up new messages. If so, +fix it. By this, code quality gets incremental better and if there comes the +day, the linting is balanced out, we might decide to add Pylint as a quality +gate. + + +``make pybuild`` +================ + +.. _PyPi: https://pypi.org/ +.. _twine: https://twine.readthedocs.io/en/latest/ + +Build Python packages in ``./dist/py``. + +.. code:: sh + + $ make pybuild + ... + BUILD pybuild + running sdist + running egg_info + ... + $ ls ./dist/py/ + searx-0.15.0-py3-none-any.whl searx-0.15.0.tar.gz + +To upload packages to PyPi_, there is also a ``upload-pypi`` target. It needs +twine_ to be installed. Since you are not the owner of :pypi:`searx` you will +never need the latter. diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst new file mode 100644 index 0000000..2bf44f1 --- /dev/null +++ b/docs/dev/plugins.rst @@ -0,0 +1,54 @@ +.. _dev plugin: + +======= +Plugins +======= + +.. sidebar:: Further reading .. + + - :ref:`plugins generic` + +Plugins can extend or replace functionality of various components of searx. + +Example plugin +============== + +.. code:: python + + name = 'Example plugin' + description = 'This plugin extends the suggestions with the word "example"' + default_on = False # disabled by default + + js_dependencies = tuple() # optional, list of static js files + css_dependencies = tuple() # optional, list of static css files + + + # attach callback to the post search hook + # request: flask request object + # ctx: the whole local context of the post search hook + def post_search(request, ctx): + ctx['search'].suggestions.add('example') + return True + +Plugin entry points +=================== + +Entry points (hooks) define when a plugin runs. Right now only three hooks are +implemented. So feel free to implement a hook if it fits the behaviour of your +plugin. + +Pre search hook +--------------- + +Runs BEFORE the search request. Function to implement: ``pre_search`` + +Post search hook +---------------- + +Runs AFTER the search request. Function to implement: ``post_search`` + +Result hook +----------- + +Runs when a new result is added to the result list. Function to implement: +``on_result`` diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst new file mode 100644 index 0000000..e40772b --- /dev/null +++ b/docs/dev/quickstart.rst @@ -0,0 +1,132 @@ +.. _devquickstart: + +====================== +Development Quickstart +====================== + +.. sidebar:: :ref:`makefile` + + For additional developer purpose there are :ref:`makefile`. + +This quickstart guide gets your environment set up with searx. Furthermore, it +gives a short introduction to the ``manage.sh`` script. + +How to setup your development environment +========================================= + +.. sidebar:: :ref:`make pyenv <make pyenv>` + + Alternatively use the :ref:`make pyenv`. + +First, clone the source code of searx to the desired folder. In this case the +source is cloned to ``~/myprojects/searx``. Then create and activate the +searx-ve virtualenv and install the required packages using ``manage.sh``. + +.. code:: sh + + cd ~/myprojects + git clone https://github.com/asciimoo/searx.git + cd searx + virtualenv searx-ve + . ./searx-ve/bin/activate + ./manage.sh update_dev_packages + + +How to run tests +================ + +.. sidebar:: :ref:`make test.unit <make test>` + + Alternatively use the ``test.pep8``, ``test.unit``, ``test.robot`` targets. + +Tests can be run using the ``manage.sh`` script. Following tests and checks are +available: + +- Unit tests +- Selenium tests +- PEP8 validation +- Unit test coverage check + +For example unit tests are run with the command below: + +.. code:: sh + + ./manage.sh unit_tests + +For further test options, please consult the help of the ``manage.sh`` script or +read :ref:`make test`. + + +How to compile styles and javascript +==================================== + +.. _less: http://lesscss.org/ +.. _NodeJS: https://nodejs.org + +How to build styles +------------------- + +Less_ is required to build the styles of searx. Less_ can be installed using +either NodeJS_ or Apt. + +.. code:: sh + + sudo -H apt-get install nodejs + sudo -H npm install -g less + +OR + +.. code:: sh + + sudo -H apt-get install node-less + +After satisfying the requirements styles can be build using ``manage.sh`` + +.. code:: sh + + ./manage.sh styles + + +How to build the source of the oscar theme +========================================== + +.. _grunt: https://gruntjs.com/ + +Grunt_ must be installed in order to build the javascript sources. It depends on +NodeJS, so first Node has to be installed. + +.. code:: sh + + sudo -H apt-get install nodejs + sudo -H npm install -g grunt-cli + +After installing grunt, the files can be built using the following command: + +.. code:: sh + + ./manage.sh grunt_build + + +Tips for debugging/development +============================== + +.. sidebar:: :ref:`make run` + + Makefile target ``run`` already enables debug option for your developer + session / see :ref:`make run`. + +Turn on debug logging + Whether you are working on a new engine or trying to eliminate a bug, it is + always a good idea to turn on debug logging. When debug logging is enabled a + stack trace appears, instead of the cryptic ``Internal Server Error`` + message. It can be turned on by setting ``debug: False`` to ``debug: True`` in + :origin:`settings.yml <searx/settings.yml>`. + +.. sidebar:: :ref:`make test` + + Alternatively use the :ref:`make test` targets. + +Run ``./manage.sh tests`` before creating a PR. + Failing build on Travis is common because of PEP8 checks. So a new commit + must be created containing these format fixes. This phase can be skipped if + ``./manage.sh tests`` is run locally before creating a PR. diff --git a/docs/dev/reST.rst b/docs/dev/reST.rst new file mode 100644 index 0000000..4dc1279 --- /dev/null +++ b/docs/dev/reST.rst @@ -0,0 +1,1428 @@ +.. _reST primer: + +=========== +reST primer +=========== + +.. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at searx encourage our + contributors to follow principles like KISS_ and readability_. + +We at searx are using reStructuredText (aka reST_) markup for all kind of +documentation, with the builders from the Sphinx_ project a HTML output is +generated and deployed at :docs:`github.io <.>`. For build prerequisites read +:ref:`docs build`. + +The source files of Searx's documentation are located at :origin:`docs`. Sphinx +assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs-live +<make docs-live>` to build HTML while editing. + +.. sidebar:: Further reading + + - Sphinx-Primer_ + - `Sphinx markup constructs`_ + - reST_, docutils_, `docutils FAQ`_ + - Sphinx_, `sphinx-doc FAQ`_ + - `sphinx config`_, doctree_ + - `sphinx cross references`_ + - linuxdoc_ + - intersphinx_ + - sphinx-jinja_ + - `Sphinx's autodoc`_ + - `Sphinx's Python domain`_, `Sphinx's C domain`_ + - SVG_, ImageMagick_ + - DOT_, `Graphviz's dot`_, Graphviz_ + + +.. contents:: Contents + :depth: 3 + :local: + :backlinks: entry + +Sphinx_ and reST_ have their place in the python ecosystem. Over that reST is +used in popular projects, e.g the Linux kernel documentation `[kernel doc]`_. + +.. _[kernel doc]: https://www.kernel.org/doc/html/latest/doc-guide/sphinx.html + +.. sidebar:: Content matters + + The readability_ of the reST sources has its value, therefore we recommend to + make sparse usage of reST markup / .. content matters! + +**reST** is a plaintext markup language, its markup is *mostly* intuitive and +you will not need to learn much to produce well formed articles with. I use the +word *mostly*: like everything in live, reST has its advantages and +disadvantages, some markups feel a bit grumpy (especially if you are used to +other plaintext markups). + +Soft skills +=========== + +Before going any deeper into the markup let's face on some **soft skills** a +trained author brings with, to reach a well feedback from readers: + +- Documentation is dedicated to an audience and answers questions from the + audience point of view. +- Don't detail things which are general knowledge from the audience point of + view. +- Limit the subject, use cross links for any further reading. + +To be more concrete what a *point of view* means. In the (:origin:`docs`) +folder we have three sections (and the *blog* folder), each dedicate to a +different group of audience. + +User's POV: :origin:`docs/user` + A typical user knows about search engines and might have heard about + meta crawlers and privacy. + +Admin's POV: :origin:`docs/admin` + A typical Admin knows about setting up services on a linux system, but he does + not know all the pros and cons of a searx setup. + +Developer's POV: :origin:`docs/dev` + Depending on the readability_ of code, a typical developer is able to read and + understand source code. Describe what a item aims to do (e.g. a function). + If the chronological order matters, describe it. Name the *out-of-limits + conditions* and all the side effects a external developer will not know. + +.. _reST inline markup: + +Basic inline markup +=================== + +.. sidebar:: Inline markup + + - :ref:`reST roles` + - :ref:`reST smart ref` + +Basic inline markup is done with asterisks and backquotes. If asterisks or +backquotes appear in running text and could be confused with inline markup +delimiters, they have to be escaped with a backslash (``\*pointer``). + +.. table:: basic inline markup + :widths: 4 3 7 + + ================================================ ==================== ======================== + description rendered markup + ================================================ ==================== ======================== + one asterisk for emphasis *italics* ``*italics*`` + two asterisks for strong emphasis **boldface** ``**boldface**`` + backquotes for code samples and literals ``foo()`` ````foo()```` + quote asterisks or backquotes \*foo is a pointer ``\*foo is a pointer`` + ================================================ ==================== ======================== + +.. _reST basic structure: + +Basic article structure +======================= + +The basic structure of an article makes use of heading adornments to markup +chapter, sections and subsections. + +.. _reST template: + +reST template +------------- + +reST template for an simple article: + +.. code:: reST + + .. _doc refname: + + ============== + Document title + ============== + + Lorem ipsum dolor sit amet, consectetur adipisici elit .. Further read + :ref:`chapter refname`. + + .. _chapter refname: + + Chapter + ======= + + Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut + aliquid ex ea commodi consequat ... + + .. _section refname: + + Section + ------- + + lorem .. + + .. _subsection refname: + + Subsection + ~~~~~~~~~~ + + lorem .. + + +Headings +-------- + +#. title - with overline for document title: + + .. code:: reST + + ============== + Document title + ============== + + +#. chapter - with anchor named ``anchor name``: + + .. code:: reST + + .. _anchor name: + + Chapter + ======= + +#. section + + .. code:: reST + + Section + ------- + +#. subsection + + .. code:: reST + + Subsection + ~~~~~~~~~~ + + + +Anchors & Links +=============== + +.. _reST anchor: + +Anchors +------- + +.. _ref role: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref + +To refer a point in the documentation a anchor is needed. The :ref:`reST +template <reST template>` shows an example where a chapter titled *"Chapters"* +gets an anchor named ``chapter title``. Another example from *this* document, +where the anchor named ``reST anchor``: + +.. code:: reST + + .. _reST anchor: + + Anchors + ------- + + To refer a point in the documentation a anchor is needed ... + +To refer anchors use the `ref role`_ markup: + +.. code:: reST + + Visit chapter :ref:`reST anchor`. Or set hyperlink text manualy :ref:`foo + bar <reST anchor>`. + +.. admonition:: ``:ref:`` role + :class: rst-example + + Visist chapter :ref:`reST anchor`. Or set hyperlink text manualy :ref:`foo + bar <reST anchor>`. + +.. _reST ordinary ref: + +Link ordinary URL +----------------- + +If you need to reference external URLs use *named* hyperlinks to maintain +readability of reST sources. Here is a example taken from *this* article: + +.. code:: reST + + .. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + <https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html>`__. + +.. admonition:: Named hyperlink + :class: rst-example + + With the *named* hyperlink `Sphinx Field Lists`_, the raw text is much more + readable. + + And this shows the alternative (less readable) hyperlink markup `Sphinx Field + Lists + <https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html>`__. + + +.. _reST smart ref: + +Smart refs +---------- + +With the power of sphinx.ext.extlinks_ and intersphinx_ referencing external +content becomes smart. + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + refer ... rendered example markup + ========================== ================================== ==================================== + :rst:role:`rfc` :rfc:`822` ``:rfc:`822``` + :rst:role:`pep` :pep:`8` ``:pep:`8``` + sphinx.ext.extlinks_ + -------------------------------------------------------------------------------------------------- + project's wiki article :wiki:`Searx-instances` ``:wiki:`Searx-instances``` + to docs public URL :docs:`dev/reST.html` ``:docs:`dev/reST.html``` + files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst``` + pull request :pull:`1756` ``:pull:`1756``` + patch :patch:`af2cae6` ``:patch:`af2cae6``` + PyPi package :pypi:`searx` ``:pypi:`searx``` + manual page man :man:`bash` ``:man:`bash``` + intersphinx_ + -------------------------------------------------------------------------------------------------- + external anchor :ref:`python:and` ``:ref:`python:and``` + external doc anchor :doc:`jinja:templates` ``:doc:`jinja:templates``` + python code object :py:obj:`datetime.datetime` ``:py:obj:`datetime.datetime``` + flask code object :py:obj:`flask.Flask` ``:py:obj:`flask.Flask``` + ========================== ================================== ==================================== + + +Intersphinx is configured in :origin:`docs/conf.py`: + +.. code:: python + + intersphinx_mapping = { + "python": ("https://docs.python.org/3/", None), + "flask": ("https://flask.palletsprojects.com/", None), + "jinja": ("https://jinja.palletsprojects.com/", None), + "linuxdoc" : ("https://return42.github.io/linuxdoc/", None), + "sphinx" : ("https://www.sphinx-doc.org/en/master/", None), + } + + +To list all anchors of the inventory (e.g. ``python``) use: + +.. code:: sh + + $ python -m sphinx.ext.intersphinx https://docs.python.org/3/objects.inv + +Literal blocks +============== + +The simplest form of :duref:`literal-blocks` is a indented block introduced by +two colons (``::``). For highlighting use :dudir:`highlight` or :ref:`reST +code` directive. To include literals from external files use directive +:dudir:`literalinclude`. + +.. _reST literal: + +``::`` +------ + +.. code:: reST + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + +.. admonition:: Literal block + :class: rst-example + + :: + + Literal block + + Lorem ipsum dolor:: + + Literal block + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore :: + + Literal block + + +.. _reST code: + +``code-block`` +-------------- + +.. _pygments: https://pygments.org/languages/ + +.. sidebar:: Syntax highlighting + + is handled by pygments_. + +The :rst:dir:`code-block` directive is a variant of the :dudir:`code` directive +with additional options. To learn more about code literals visit +:ref:`sphinx:code-examples`. + +.. code-block:: reST + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +.. code-block:: reST + +.. admonition:: Code block + :class: rst-example + + The URL ``/stats`` handle is shown in :ref:`stats-handle` + + .. code-block:: Python + :caption: python code block + :name: stats-handle + + @app.route('/stats', methods=['GET']) + def stats(): + """Render engine statistics page.""" + stats = get_engines_stats() + return render( + 'stats.html' + , stats = stats ) + +Unicode substitution +==================== + +The :dudir:`unicode directive <unicode-character-codes>` converts Unicode +character codes (numerical values) to characters. This directive can only be +used within a substitution definition. + +.. code-block:: reST + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + +.. admonition:: Unicode + :class: rst-example + + .. |copy| unicode:: 0xA9 .. copyright sign + .. |(TM)| unicode:: U+2122 + + Trademark |(TM)| and copyright |copy| glyphs. + + +.. _reST roles: + +Roles +===== + +.. sidebar:: Further reading + + - `Sphinx Roles`_ + - :doc:`sphinx:usage/restructuredtext/domains` + + +A *custom interpreted text role* (:duref:`ref <roles>`) is an inline piece of +explicit markup. It signifies that that the enclosed text should be interpreted +in a specific way. + +The general markup is one of: + +.. code:: reST + + :rolename:`ref-name` + :rolename:`ref text <ref-name>` + +.. table:: smart refs with sphinx.ext.extlinks_ and intersphinx_ + :widths: 4 3 7 + + ========================== ================================== ==================================== + role rendered example markup + ========================== ================================== ==================================== + :rst:role:`guilabel` :guilabel:`&Cancel` ``:guilabel:`&Cancel``` + :rst:role:`kbd` :kbd:`C-x C-f` ``:kbd:`C-x C-f``` + :rst:role:`menuselection` :menuselection:`Open --> File` ``:menuselection:`Open --> File``` + :rst:role:`download` :download:`this file <reST.rst>` ``:download:`this file <reST.rst>``` + math_ :math:`a^2 + b^2 = c^2` ``:math:`a^2 + b^2 = c^2``` + :rst:role:`ref` :ref:`svg image example` ``:ref:`svg image example``` + :rst:role:`command` :command:`ls -la` ``:command:`ls -la``` + :durole:`emphasis` :emphasis:`italic` ``:emphasis:`italic``` + :durole:`strong` :strong:`bold` ``:strong:`bold``` + :durole:`literal` :literal:`foo()` ``:literal:`foo()``` + :durole:`subscript` H\ :sub:`2`\ O ``H\ :sub:`2`\ O`` + :durole:`superscript` E = mc\ :sup:`2` ``E = mc\ :sup:`2``` + :durole:`title-reference` :title:`Time` ``:title:`Time``` + ========================== ================================== ==================================== + +Figures & Images +================ + +.. sidebar:: Image processing + + With the directives from :ref:`linuxdoc <linuxdoc:kfigure>` the build process + is flexible. To get best results in the generated output format, install + ImageMagick_ and Graphviz_. + +Searx's sphinx setup includes: :ref:`linuxdoc:kfigure`. Scaleable here means; +scaleable in sense of the build process. Normally in absence of a converter +tool, the build process will break. From the authors POV it’s annoying to care +about the build process when handling with images, especially since he has no +access to the build process. With :ref:`linuxdoc:kfigure` the build process +continues and scales output quality in dependence of installed image processors. + +If you want to add an image, you should use the ``kernel-figure`` (inheritance +of :dudir:`figure`) and ``kernel-image`` (inheritance of :dudir:`image`) +directives. E.g. to insert a figure with a scaleable image format use SVG +(:ref:`svg image example`): + +.. code:: reST + + .. _svg image example: + + .. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image + + To refer the figure, a caption block is needed: :ref:`svg image example`. + +.. _svg image example: + +.. kernel-figure:: svg_image.svg + :alt: SVG image example + + Simple SVG image. + +To refer the figure, a caption block is needed: :ref:`svg image example`. + +DOT files (aka Graphviz) +------------------------ + +With :ref:`linuxdoc:kernel-figure` reST support for **DOT** formatted files is +given. + +- `Graphviz's dot`_ +- DOT_ +- Graphviz_ + +A simple example is shown in :ref:`dot file example`: + +.. code:: reST + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +.. admonition:: hello.dot + :class: rst-example + + .. _dot file example: + + .. kernel-figure:: hello.dot + :alt: hello world + + DOT's hello world example + +``kernel-render`` DOT +--------------------- + +Embed *render* markups (or languages) like Graphviz's **DOT** is provided by the +:ref:`linuxdoc:kernel-render` directive. A simple example of embedded DOT_ is +shown in figure :ref:`dot render example`: + +.. code:: reST + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +Please note :ref:`build tools <linuxdoc:kfigure_build_tools>`. If Graphviz_ is +installed, you will see an vector image. If not, the raw markup is inserted as +*literal-block*. + +.. admonition:: kernel-render DOT + :class: rst-example + + .. _dot render example: + + .. kernel-render:: DOT + :alt: digraph + :caption: Embedded DOT (Graphviz) code + + digraph foo { + "bar" -> "baz"; + } + + Attribute ``caption`` is needed, if you want to refer the figure: :ref:`dot + render example`. + +``kernel-render`` SVG +--------------------- + +A simple example of embedded SVG_ is shown in figure :ref:`svg render example`: + +.. code:: reST + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow +.. + + .. code:: xml + + <?xml version="1.0" encoding="UTF-8"?> + <svg xmlns="http://www.w3.org/2000/svg" version="1.1" + baseProfile="full" width="70px" height="40px" + viewBox="0 0 700 400" + > + <line x1="180" y1="370" + x2="500" y2="50" + stroke="black" stroke-width="15px" + /> + <polygon points="585 0 525 25 585 50" + transform="rotate(135 525 25)" + /> + </svg> + +.. admonition:: kernel-render SVG + :class: rst-example + + .. _svg render example: + + .. kernel-render:: SVG + :caption: Embedded **SVG** markup + :alt: so-nw-arrow + + <?xml version="1.0" encoding="UTF-8"?> + <svg xmlns="http://www.w3.org/2000/svg" version="1.1" + baseProfile="full" width="70px" height="40px" + viewBox="0 0 700 400" + > + <line x1="180" y1="370" + x2="500" y2="50" + stroke="black" stroke-width="15px" + /> + <polygon points="585 0 525 25 585 50" + transform="rotate(135 525 25)" + /> + </svg> + + + + +.. _reST lists: + +List markups +============ + +Bullet list +----------- + +List markup (:duref:`ref <bullet-lists>`) is simple: + +.. code:: reST + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + +.. admonition:: bullet list + :class: rst-example + + - This is a bulleted list. + + 1. Nested lists are possible, but be aware that they must be separated from + the parent list items by blank line + 2. Second item of nested list + + - It has two items, the second + item uses two lines. + + #. This is a numbered list. + #. It has two items too. + + +Horizontal list +--------------- + +The :rst:dir:`.. hlist:: <hlist>` transforms a bullet list into a more compact +list. + +.. code:: reST + + .. hlist:: + + - first list item + - second list item + - third list item + ... + +.. admonition:: hlist + :class: rst-example + + .. hlist:: + + - first list item + - second list item + - third list item + - next list item + - next list item xxxx + - next list item yyyy + - next list item zzzz + + +Definition list +--------------- + +.. sidebar:: Note .. + + - the term cannot have more than one line of text + + - there is **no blank line between term and definition block** // this + distinguishes definition lists (:duref:`ref <definition-lists>`) from block + quotes (:duref:`ref <block-quotes>`). + +Each definition list (:duref:`ref <definition-lists>`) item contains a term, +optional classifiers and a definition. A term is a simple one-line word or +phrase. Optional classifiers may follow the term on the same line, each after +an inline ' : ' (**space, colon, space**). A definition is a block indented +relative to the term, and may contain multiple paragraphs and other body +elements. There may be no blank line between a term line and a definition block +(*this distinguishes definition lists from block quotes*). Blank lines are +required before the first and after the last definition list item, but are +optional in-between. + +Definition lists are created as follows: + +.. code:: reST + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref <block-quotes>`) and not as term's definition! + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + Definition 4. + +.. admonition:: definition list + :class: rst-example + + term 1 (up to a line of text) + Definition 1. + + See the typo : this line is not a term! + + And this is not term's definition. **There is a blank line** in between + the line above and this paragraph. That's why this paragraph is taken as + **block quote** (:duref:`ref <block-quotes>`) and not as term's definition! + + + term 2 + Definition 2, paragraph 1. + + Definition 2, paragraph 2. + + term 3 : classifier + Definition 3. + + term 4 : classifier one : classifier two + + +Quoted paragraphs +----------------- + +Quoted paragraphs (:duref:`ref <block-quotes>`) are created by just indenting +them more than the surrounding paragraphs. Line blocks (:duref:`ref +<line-blocks>`) are a way of preserving line breaks: + +.. code:: reST + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. admonition:: Quoted paragraph and line block + :class: rst-example + + normal paragraph ... + lorem ipsum. + + Quoted paragraph ... + lorem ipsum. + + | These lines are + | broken exactly like in + | the source file. + + +.. _reST field list: + +Field Lists +----------- + +.. _Sphinx Field Lists: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/field-lists.html + +.. sidebar:: bibliographic fields + + First lines fields are bibliographic fields, see `Sphinx Field Lists`_. + +Field lists are used as part of an extension syntax, such as options for +directives, or database-like records meant for further processing. Field lists +are mappings from field names to field bodies. They marked up like this: + +.. code:: reST + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + +.. admonition:: Field List + :class: rst-example + + :fieldname: Field content + :foo: first paragraph in field foo + + second paragraph in field foo + + :bar: Field content + + +They are commonly used in Python documentation: + +.. code:: python + + def my_function(my_arg, my_other_arg): + """A function just for me. + + :param my_arg: The first of my arguments. + :param my_other_arg: The second of my arguments. + + :returns: A message (just for me, of course). + """ + +Further list blocks +------------------- + +- field lists (:duref:`ref <field-lists>`, with caveats noted in + :ref:`reST field list`) +- option lists (:duref:`ref <option-lists>`) +- quoted literal blocks (:duref:`ref <quoted-literal-blocks>`) +- doctest blocks (:duref:`ref <doctest-blocks>`) + + +Admonitions +=========== + +Sidebar +------- + +Sidebar is an eye catcher, often used for admonitions pointing further stuff or +site effects. Here is the source of the sidebar :ref:`on top of this page <reST +primer>`. + +.. code:: reST + + .. sidebar:: KISS_ and readability_ + + Instead of defining more and more roles, we at searx encourage our + contributors to follow principles like KISS_ and readability_. + +Generic admonition +------------------ + +The generic :dudir:`admonition <admonitions>` needs a title: + +.. code:: reST + + .. admonition:: generic admonition title + + lorem ipsum .. + + +.. admonition:: generic admonition title + + lorem ipsum .. + + +Specific admonitions +-------------------- + +Specific admonitions: :dudir:`hint`, :dudir:`note`, :dudir:`tip` :dudir:`attention`, +:dudir:`caution`, :dudir:`danger`, :dudir:`error`, , :dudir:`important`, and +:dudir:`warning` . + +.. code:: reST + + .. hint:: + + lorem ipsum .. + + .. note:: + + lorem ipsum .. + + .. warning:: + + lorem ipsum .. + + +.. hint:: + + lorem ipsum .. + +.. note:: + + lorem ipsum .. + +.. tip:: + + lorem ipsum .. + +.. attention:: + + lorem ipsum .. + +.. caution:: + + lorem ipsum .. + +.. danger:: + + lorem ipsum .. + +.. important:: + + lorem ipsum .. + +.. error:: + + lorem ipsum .. + +.. warning:: + + lorem ipsum .. + + +Tables +====== + +.. sidebar:: Nested tables + + Nested tables are ugly! Not all builder support nested tables, don't use + them! + +ASCII-art tables like :ref:`reST simple table` and :ref:`reST grid table` might +be comfortable for readers of the text-files, but they have huge disadvantages +in the creation and modifying. First, they are hard to edit. Think about +adding a row or a column to a ASCII-art table or adding a paragraph in a cell, +it is a nightmare on big tables. + + +.. sidebar:: List tables + + For meaningful patch and diff use :ref:`reST flat table`. + +Second the diff of modifying ASCII-art tables is not meaningful, e.g. widening a +cell generates a diff in which also changes are included, which are only +ascribable to the ASCII-art. Anyway, if you prefer ASCII-art for any reason, +here are some helpers: + +* `Emacs Table Mode`_ +* `Online Tables Generator`_ + +.. _reST simple table: + +Simple tables +------------- + +:duref:`Simple tables <simple-tables>` allow *colspan* but not *rowspan*. If +your table need some metadata (e.g. a title) you need to add the ``.. table:: +directive`` :dudir:`(ref) <table>` in front and place the table in its body: + +.. code:: reST + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + +.. admonition:: Simple ASCII table + :class: rst-example + + .. table:: foo gate truth table + :widths: grid + :align: left + + ====== ====== ====== + Inputs Output + ------------- ------ + A B A or B + ====== ====== ====== + False + -------------------- + True + -------------------- + True False True + (foo) + ------ ------ ------ + False True + (foo) + ====== ============= + + + +.. _reST grid table: + +Grid tables +----------- + +:duref:`Grid tables <grid-tables>` allow colspan *colspan* and *rowspan*: + +.. code:: reST + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + +.. admonition:: ASCII grid table + :class: rst-example + + .. table:: grid table example + :widths: 1 1 5 + + +------------+------------+-----------+ + | Header 1 | Header 2 | Header 3 | + +============+============+===========+ + | body row 1 | column 2 | column 3 | + +------------+------------+-----------+ + | body row 2 | Cells may span columns.| + +------------+------------+-----------+ + | body row 3 | Cells may | - Cells | + +------------+ span rows. | - contain | + | body row 4 | | - blocks. | + +------------+------------+-----------+ + + +.. _reST flat table: + +flat-table +---------- + +The ``flat-table`` is a further developed variant of the :ref:`list tables +<linuxdoc:list-table-directives>`. It is a double-stage list similar to the +:dudir:`list-table` with some additional features: + +column-span: ``cspan`` + with the role ``cspan`` a cell can be extended through additional columns + +row-span: ``rspan`` + with the role ``rspan`` a cell can be extended through additional rows + +auto-span: + spans rightmost cell of a table row over the missing cells on the right side + of that table-row. With Option ``:fill-cells:`` this behavior can changed + from *auto span* to *auto fill*, which automatically inserts (empty) cells + instead of spanning the last cell. + +options: + :header-rows: [int] count of header rows + :stub-columns: [int] count of stub columns + :widths: [[int] [int] ... ] widths of columns + :fill-cells: instead of auto-span missing cells, insert missing cells + +roles: + :cspan: [int] additional columns (*morecols*) + :rspan: [int] additional rows (*morerows*) + +The example below shows how to use this markup. The first level of the staged +list is the *table-row*. In the *table-row* there is only one markup allowed, +the list of the cells in this *table-row*. Exception are *comments* ( ``..`` ) +and *targets* (e.g. a ref to :ref:`row 2 of table's body <row body 2>`). + +.. code:: reST + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to rigth end + + * - row 6 + - cell 6.1 + - .. + + +.. admonition:: List table + :class: rst-example + + .. flat-table:: ``flat-table`` example + :header-rows: 2 + :stub-columns: 1 + :widths: 1 1 1 1 2 + + * - :rspan:`1` head / stub + - :cspan:`3` head 1.1-4 + + * - head 2.1 + - head 2.2 + - head 2.3 + - head 2.4 + + * .. row body 1 / this is a comment + + - row 1 + - :rspan:`2` cell 1-3.1 + - cell 1.2 + - cell 1.3 + - cell 1.4 + + * .. Comments and targets are allowed on *table-row* stage. + .. _`row body 2`: + + - row 2 + - cell 2.2 + - :rspan:`1` :cspan:`1` + cell 2.3 with a span over + + * col 3-4 & + * row 2-3 + + * - row 3 + - cell 3.2 + + * - row 4 + - cell 4.1 + - cell 4.2 + - cell 4.3 + - cell 4.4 + + * - row 5 + - cell 5.1 with automatic span to rigth end + + * - row 6 + - cell 6.1 + - .. + + +CSV table +--------- + +CSV table might be the choice if you want to include CSV-data from a outstanding +(build) process into your documentation. + +.. code:: reST + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 2 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Content of file ``csv_table.txt``: + +.. literalinclude:: csv_table.txt + +.. admonition:: CSV table + :class: rst-example + + .. csv-table:: CSV table example + :header: .. , Column 1, Column 2 + :widths: 3 5 5 + :stub-columns: 1 + :file: csv_table.txt + +Templating +========== + +.. sidebar:: Build environment + + All *generic-doc* tasks are running in the :ref:`build environment <make + pyenv>`. + +Templating is suitable for documentation which is created generic at the build +time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`build +environment <make pyenv>` (with searx modules installed). We use this e.g. to +build chapter: :ref:`engines generic`. Below the jinja directive from the +:origin:`docs/admin/engines.rst` is shown: + +.. literalinclude:: ../admin/engines.rst + :language: reST + :start-after: .. _configured engines: + +The context for the template is selected in the line ``.. jinja:: webapp``. In +sphinx's build configuration (:origin:`docs/conf.py`) the ``webapp`` context +points to the name space of the python module: ``webapp``. + +.. code:: py + + from searx import webapp + jinja_contexts = { + 'webapp': dict(**webapp.__dict__) + } + + +Tabbed views +============ + +.. _sphinx-tabs: https://github.com/djungelorm/sphinx-tabs +.. _basic-tabs: https://github.com/djungelorm/sphinx-tabs#basic-tabs +.. _group-tabs: https://github.com/djungelorm/sphinx-tabs#group-tabs +.. _code-tabs: https://github.com/djungelorm/sphinx-tabs#code-tabs + +With `sphinx-tabs`_ extension we have *tabbed views*. To provide installation +instructions with one tab per distribution we use the `group-tabs`_ directive, +others are basic-tabs_ and code-tabs_. Below a *group-tab* example from +:ref:`docs build` is shown: + +.. literalinclude:: ../admin/buildhosts.rst + :language: reST + :start-after: .. _system requirements: + :end-before: .. _system requirements END: + + +.. _math: + +Math equations +============== + +.. _Mathematics: https://en.wikibooks.org/wiki/LaTeX/Mathematics +.. _amsmath user guide: + http://vesta.informatik.rwth-aachen.de/ftp/pub/mirror/ctan/macros/latex/required/amsmath/amsldoc.pdf + +.. sidebar:: About LaTeX + + - `amsmath user guide`_ + - Mathematics_ + - :ref:`docs build` + +The input language for mathematics is LaTeX markup using the :ctan:`amsmath` +package. + +To embed LaTeX markup in reST documents, use role :rst:role:`:math: <math>` for +inline and directive :rst:dir:`.. math:: <math>` for block markup. + +.. code:: reST + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + +.. admonition:: LaTeX math equation + :class: rst-example + + In :math:numref:`schroedinger general` the time-dependent Schrödinger equation + is shown. + + .. math:: + :label: schroedinger general + + \mathrm{i}\hbar\dfrac{\partial}{\partial t} |\,\psi (t) \rangle = + \hat{H} |\,\psi (t) \rangle. + + +The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac`` +(*displaystyle*) used in a inline markup or another fraction. + +.. code:: reST + + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + ``\dfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + +.. admonition:: Line spacing + :class: rst-example + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\tfrac{\tfrac{1}{x}+\tfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy + eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam + voluptua. ... + ``\tfrac`` **inline example** :math:`\dfrac{\dfrac{1}{x}+\dfrac{1}{y}}{y-z}` + At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd + gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. + + +.. _KISS: https://en.wikipedia.org/wiki/KISS_principle + +.. _readability: https://docs.python-guide.org/writing/style/ +.. _Sphinx-Primer: + http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +.. _reST: https://docutils.sourceforge.io/rst.html +.. _Sphinx Roles: + https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html +.. _Sphinx: http://www.sphinx-doc.org +.. _`sphinx-doc FAQ`: http://www.sphinx-doc.org/en/stable/faq.html +.. _Sphinx markup constructs: + http://www.sphinx-doc.org/en/stable/markup/index.html +.. _`sphinx cross references`: + http://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations +.. _sphinx.ext.extlinks: + https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html +.. _intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html +.. _sphinx config: http://www.sphinx-doc.org/en/stable/config.html +.. _Sphinx's autodoc: http://www.sphinx-doc.org/en/stable/ext/autodoc.html +.. _Sphinx's Python domain: + http://www.sphinx-doc.org/en/stable/domains.html#the-python-domain +.. _Sphinx's C domain: + http://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs +.. _doctree: + http://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases +.. _docutils: http://docutils.sourceforge.net/docs/index.html +.. _docutils FAQ: http://docutils.sourceforge.net/FAQ.html +.. _linuxdoc: https://return42.github.io/linuxdoc +.. _jinja: https://jinja.palletsprojects.com/ +.. _sphinx-jinja: https://github.com/tardyp/sphinx-jinja +.. _SVG: https://www.w3.org/TR/SVG11/expanded-toc.html +.. _DOT: https://graphviz.gitlab.io/_pages/doc/info/lang.html +.. _`Graphviz's dot`: https://graphviz.gitlab.io/_pages/pdf/dotguide.pdf +.. _Graphviz: https://graphviz.gitlab.io +.. _ImageMagick: https://www.imagemagick.org + +.. _`Emacs Table Mode`: https://www.emacswiki.org/emacs/TableMode +.. _`Online Tables Generator`: http://www.tablesgenerator.com/text_tables +.. _`OASIS XML Exchange Table Model`: https://www.oasis-open.org/specs/tm9901.html diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst new file mode 100644 index 0000000..922548f --- /dev/null +++ b/docs/dev/search_api.rst @@ -0,0 +1,120 @@ +.. _search API: + +========== +Search API +========== + +The search supports both ``GET`` and ``POST``. + +Furthermore, two enpoints ``/`` and ``/search`` are available for querying. + + +``GET /`` + +``GET /search`` + +Parameters +========== + +.. sidebar:: Further reading .. + + - :ref:`engines-dev` + - :ref:`settings.yml` + - :ref:`engines generic` + +``q`` : required + The search query. This string is passed to external search services. Thus, + searx supports syntax of each search service. For example, ``site:github.com + searx`` is a valid query for Google. However, if simply the query above is + passed to any search engine which does not filter its results based on this + syntax, you might not get the results you wanted. + + See more at :ref:`search-syntax` + +``categories`` : optional + Comma separated list, specifies the active search categories + +``engines`` : optional + Comma separated list, specifies the active search engines. + +``lang`` : default ``all`` + Code of the language. + +``pageno`` : default ``1`` + Search page number. + +``time_range`` : optional + [ ``day``, ``month``, ``year`` ] + + Time range of search for engines which support it. See if an engine supports + time range search in the preferences page of an instance. + +``format`` : optional + [ ``json``, ``csv``, ``rss`` ] + + Output format of results. + +``results_on_new_tab`` : default ``0`` + [ ``0``, ``1`` ] + + Open search results on new tab. + +``image_proxy`` : default ``False`` + [ ``True``, ``False`` ] + + Proxy image results through searx. + +``autocomplete`` : default *empty* + [ ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia`` ] + + Service which completes words as you type. + +``safesearch`` : default ``None`` + [ ``0``, ``1``, ``None`` ] + + Filter search results of engines which support safe search. See if an engine + supports safe search in the preferences page of an instance. + +``theme`` : default ``oscar`` + [ ``oscar``, ``simple``, ``legacy``, ``pix-art``, ``courgette`` ] + + Theme of instance. + + Please note, available themes depend on an instance. It is possible that an + instance administrator deleted, created or renamed themes on his/her instance. + See the available options in the preferences page of the instance. + +``oscar-style`` : default ``logicodev`` + [ ``pointhi``, ``logicodev`` ] + + Style of Oscar theme. It is only parsed if the theme of an instance is + ``oscar``. + + Please note, available styles depend on an instance. It is possible that an + instance administrator deleted, created or renamed styles on his/her + instance. See the available options in the preferences page of the instance. + +``enabled_plugins`` : optional + List of enabled plugins. + + :default: ``HTTPS_rewrite``, ``Self_Informations``, + ``Search_on_category_select``, ``Tracker_URL_remover`` + + :values: [ ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, + ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, + ``Search_on_category_select`` ] + +``disabled_plugins``: optional + List of disabled plugins. + + :default: ``DOAI_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys`` + :values: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, + ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, + ``Search_on_category_select`` + +``enabled_engines`` : optional : *all* :origin:`engines <searx/engines>` + List of enabled engines. + +``disabled_engines`` : optional : *all* :origin:`engines <searx/engines>` + List of disabled engines. + diff --git a/docs/dev/svg_image.svg b/docs/dev/svg_image.svg new file mode 100644 index 0000000..5405f85 --- /dev/null +++ b/docs/dev/svg_image.svg @@ -0,0 +1,10 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- originate: https://commons.wikimedia.org/wiki/File:Variable_Resistor.svg --> +<svg xmlns="http://www.w3.org/2000/svg" + version="1.1" baseProfile="full" + width="70px" height="40px" viewBox="0 0 700 400"> + <line x1="0" y1="200" x2="700" y2="200" stroke="black" stroke-width="20px"/> + <rect x="100" y="100" width="500" height="200" fill="white" stroke="black" stroke-width="20px"/> + <line x1="180" y1="370" x2="500" y2="50" stroke="black" stroke-width="15px"/> + <polygon points="585 0 525 25 585 50" transform="rotate(135 525 25)"/> +</svg> diff --git a/docs/dev/translation.rst b/docs/dev/translation.rst new file mode 100644 index 0000000..86c4c84 --- /dev/null +++ b/docs/dev/translation.rst @@ -0,0 +1,71 @@ +.. _translation: + +=========== +Translation +=========== + +.. _searx@transifex: https://www.transifex.com/asciimoo/searx/ + +Translation currently takes place on `searx@transifex`_ + +Requirements +============ + +* Transifex account +* Installed CLI tool of Transifex + +Init Transifex project +====================== + +After installing ``transifex`` using pip, run the following command to +initialize the project. + +.. code:: sh + + tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/ + + +After ``$HOME/.transifexrc`` is created, get a Transifex API key and insert it +into the configuration file. + +Create a configuration file for ``tx`` named ``$HOME/.tx/config``. + +.. code:: ini + + [main] + host = https://www.transifex.com + [searx.messagespo] + file_filter = searx/translations/<lang>/LC_MESSAGES/messages.po + source_file = messages.pot + source_lang = en + type = PO + + +Then run ``tx set``: + +.. code:: shell + + tx set --auto-local -r searx.messagespo 'searx/translations/<lang>/LC_MESSAGES/messages.po' \ + --source-lang en --type PO --source-file messages.pot --execute + + +Update translations +=================== + +To retrieve the latest translations, pull it from Transifex. + +.. code:: sh + + tx pull -a + +Then check the new languages. If strings translated are not enough, delete those +folders, because those should not be compiled. Call the command below to compile +the ``.po`` files. + +.. code:: shell + + pybabel compile -d searx/translations + + +After the compilation is finished commit the ``.po`` and ``.mo`` files and +create a PR. |