diff options
Diffstat (limited to 'docs')
50 files changed, 3953 insertions, 1218 deletions
diff --git a/docs/_themes/searx/static/searx.css b/docs/_themes/searx/static/searx.css index d6a664f..ae742a4 100644 --- a/docs/_themes/searx/static/searx.css +++ b/docs/_themes/searx/static/searx.css @@ -33,7 +33,7 @@ p.sidebar-title, .sidebar p { /* admonitions */ -div.admonition, div.topic { +div.admonition, div.topic, div.toctree-wrapper { background-color: #fafafa; margin: 8px 0px; padding: 1em; @@ -42,6 +42,16 @@ div.admonition, div.topic { border-right: none; border-bottom: none; border-left: 5pt solid #ccc; + list-style-type: disclosure-closed; +} + +div.toctree-wrapper p.caption { + font-weight: normal; + font-size: 24px; + margin: 0 0 10px 0; + padding: 0; + line-height: 1; + display: inline; } p.admonition-title:after { diff --git a/docs/admin/arch_public.dot b/docs/admin/arch_public.dot index a46c96d..0d82607 100644 --- a/docs/admin/arch_public.dot +++ b/docs/admin/arch_public.dot @@ -4,11 +4,11 @@ digraph G { edge [fontname="Sans"]; browser [label="Browser", shape=Mdiamond]; - rp [label="Reverse Proxy", href="url to configure reverse proxy"]; - filtron [label="Filtron", href="https://github.com/asciimoo/filtron"]; - morty [label="Morty", href="https://github.com/asciimoo/morty"]; + rp [label="Reverse Proxy", href="https://searx.github.io/searx/utils/filtron.sh.html#public-reverse-proxy"]; + filtron [label="Filtron", href="https://searx.github.io/searx/utils/filtron.sh.html"]; + morty [label="Morty", href="https://searx.github.io/searx/utils/morty.sh.html"]; static [label="Static files", href="url to configure static files"]; - uwsgi [label="uwsgi", href="url to configure uwsgi"] + uwsgi [label="uwsgi", href="https://searx.github.io/searx/utils/searx.sh.html"] searx1 [label="Searx #1"]; searx2 [label="Searx #2"]; searx3 [label="Searx #3"]; diff --git a/docs/admin/architecture.rst b/docs/admin/architecture.rst index 7064a29..464e765 100644 --- a/docs/admin/architecture.rst +++ b/docs/admin/architecture.rst @@ -4,17 +4,21 @@ Architecture ============ -.. sidebar:: Needs work! +.. sidebar:: Further reading - This article needs some work / Searx is a collaborative effort. If you have - any contribution, feel welcome to send us your :pull:`PR <../pulls>`, see - :ref:`how to contribute`. + - Reverse Proxy: :ref:`Apache <apache searx site>` & :ref:`nginx <nginx searx + site>` + - Filtron: :ref:`searx filtron` + - Morty: :ref:`searx morty` + - uWSGI: :ref:`searx uwsgi` + - Searx: :ref:`installation basic` Herein you will find some hints and suggestions about typical architectures of searx infrastructures. We start with a contribution from :pull:`@dalf <1776#issuecomment-567917320>`. -It shows a *reference* setup for public searx instances. +It shows a *reference* setup for public searx instances which can build up and +maintained by the scripts from our :ref:`toolboxing`. .. _arch public: diff --git a/docs/admin/buildhosts.rst b/docs/admin/buildhosts.rst index 5260da0..e23327b 100644 --- a/docs/admin/buildhosts.rst +++ b/docs/admin/buildhosts.rst @@ -9,8 +9,27 @@ Buildhosts If you have any contribution send us your :pull:`PR <../pulls>`, see :ref:`how to contribute`. +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + To get best results from build, its recommend to install additional packages -on build hosts. +on build hosts (see :ref:`searx.sh`).:: + + sudo -H ./utils/searx.sh install buildhost + +This will install packages needed by searx: + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START distro-packages + :end-before: END distro-packages + +and packages needed to build docuemtation and run tests: + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START build-packages + :end-before: END build-packages .. _docs build: @@ -30,16 +49,25 @@ Build docs - dvisvgm_ Most of the sphinx requirements are installed from :origin:`setup.py` and the -docs can be build from scratch with ``make docs``. For better math and image -processing additional packages are needed. The XeTeX_ needed not only for PDF -creation, its also needed for :ref:`math` when HTML output is build. +docs can be build from scratch with ``make docs.html``. For better math and +image processing additional packages are needed. The XeTeX_ needed not only for +PDF creation, its also needed for :ref:`math` when HTML output is build. To be able to do :ref:`sphinx:math-support` without CDNs, the math are rendered -as images (``sphinx.ext.imgmath`` extension). If your docs build (``make -docs``) shows warnings like this:: +as images (``sphinx.ext.imgmath`` extension). + +Here is the extract from the :origin:`docs/conf.py` file, setting math renderer +to ``imgmath``: + +.. literalinclude:: ../conf.py + :language: python + :start-after: # sphinx.ext.imgmath setup + :end-before: # sphinx.ext.imgmath setup END + +If your docs build (``make docs.html``) shows warnings like this:: WARNING: dot(1) not found, for better output quality install \ - graphviz from http://www.graphviz.org + graphviz from https://www.graphviz.org .. WARNING: LaTeX command 'latex' cannot be run (needed for math \ display), check the imgmath_latex setting @@ -47,8 +75,6 @@ docs``) shows warnings like this:: you need to install additional packages on your build host, to get better HTML output. -.. _system requirements: - .. tabs:: .. group-tab:: Ubuntu / debian @@ -92,12 +118,38 @@ For PDF output you also need: $ sudo dnf install \ texlive-collection-fontsrecommended texlive-collection-latex \ - dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts + dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts \ + ImageMagick -.. _system requirements END: +.. _sh lint: -.. literalinclude:: ../conf.py - :language: python - :start-after: # sphinx.ext.imgmath setup - :end-before: # sphinx.ext.imgmath setup END +Lint shell scripts +================== + +.. _ShellCheck: https://github.com/koalaman/shellcheck + +To lint shell scripts, we use ShellCheck_ - A shell script static analysis tool. + +.. SNIP sh lint requirements + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo apt install shellcheck + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo pacman -S shellcheck + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo dnf install ShellCheck +.. SNAP sh lint requirements diff --git a/docs/admin/engines.rst b/docs/admin/engines.rst index 4b78c8c..3ad2063 100644 --- a/docs/admin/engines.rst +++ b/docs/admin/engines.rst @@ -1,14 +1,28 @@ -.. _engines generic: - ======= Engines ======= +Special Engine Settings +======================= + .. sidebar:: Further reading .. - :ref:`settings engine` - - :ref:`engine settings` - - :ref:`engine file` + - :ref:`engine settings` & :ref:`engine file` + +.. toctree:: + :maxdepth: 1 + + engines/recoll.rst + + +.. _engines generic: + +General Engine Settings +======================= + +Explanation of the :ref:`general engine configuration` shown in the table +:ref:`configured engines`. ============= =========== ==================== ============ :ref:`engine settings` :ref:`engine file` @@ -19,20 +33,20 @@ Engine .. Paging support **P** ------------------------- -------------------- ------------ Shortcut **S** Language support **L** Timeout **TO** Time range support **TR** -Disabled **D** Offline **O** +Disabled **D** Engine type **ET** ------------- ----------- -------------------- ------------ Safe search **SS** ------------- ----------- --------------------------------- Weigth **W** ------------- ----------- --------------------------------- Disabled **D** +------------- ----------- --------------------------------- +Show errors **DE** ============= =========== ================================= -Configuration defaults (at built time): - .. _configured engines: -.. jinja:: webapp +.. jinja:: searx .. flat-table:: Engines configured at built time (defaults) :header-rows: 1 @@ -48,9 +62,10 @@ Configuration defaults (at built time): - SS - D - TR - - O - - W - - D + - ET + - W + - D + - DE {% for name, mod in engines.items() %} @@ -64,8 +79,10 @@ Configuration defaults (at built time): - {{(mod.safesearch and "y") or ""}} - {{(mod.disabled and "y") or ""}} - {{(mod.time_range_support and "y") or ""}} - - {{(mod.offline and "y") or ""}} + - {{mod.engine_type or ""}} - {{mod.weight or 1 }} - {{(mod.disabled and "y") or ""}} + - {{(mod.display_error_messages and "y") or ""}} {% endfor %} + diff --git a/docs/admin/engines/recoll.rst b/docs/admin/engines/recoll.rst new file mode 100644 index 0000000..cba2e81 --- /dev/null +++ b/docs/admin/engines/recoll.rst @@ -0,0 +1,50 @@ +.. _engine recoll: + +====== +Recoll +====== + +.. sidebar:: info + + - `Recoll <https://www.lesbonscomptes.com/recoll/>`_ + - `recoll-webui <https://framagit.org/medoc92/recollwebui.git>`_ + +Recoll_ is a desktop full-text search tool based on Xapian. By itself Recoll_ +does not offer web or API access, this can be achieved using recoll-webui_ + + + +Configuration +============= + +You must configure the following settings: + +``base_url``: + Location where recoll-webui can be reached. + +``mount_prefix``: + Location where the file hierarchy is mounted on your *local* filesystem. + +``dl_prefix``: + Location where the file hierarchy as indexed by recoll can be reached. + +``search_dir``: + Part of the indexed file hierarchy to be search, if empty the full domain is + searched. + + +Example +======= + +Scenario: + +#. Recoll indexes a local filesystem mounted in ``/export/documents/reference``, +#. the Recoll search inteface can be reached at https://recoll.example.org/ and +#. the contents of this filesystem can be reached though https://download.example.org/reference + +.. code:: yaml + + base_url: https://recoll.example.org/ + mount_prefix: /export/documents + dl_prefix: https://download.example.org + search_dir: '' diff --git a/docs/admin/filtron.rst b/docs/admin/filtron.rst index 07dcb9b..41c4a31 100644 --- a/docs/admin/filtron.rst +++ b/docs/admin/filtron.rst @@ -1,18 +1,51 @@ + +.. _searx filtron: + ========================== How to protect an instance ========================== -Searx depens on external search services. To avoid the abuse of these services +.. sidebar:: further reading + + - :ref:`filtron.sh` + - :ref:`nginx searx site` + + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. _filtron: https://github.com/asciimoo/filtron + +Searx depends on external search services. To avoid the abuse of these services it is advised to limit the number of requests processed by searx. -An application firewall, ``filtron`` solves exactly this problem. Information -on how to install it can be found at the `project page of filtron -<https://github.com/asciimoo/filtron>`__. +An application firewall, filtron_ solves exactly this problem. Filtron is just +a middleware between your web server (nginx, apache, ...) and searx, we describe +such infratructures in chapter: :ref:`architecture`. +filtron & go +============ + +.. _Go: https://golang.org/ +.. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md + +Filtron needs Go_ installed. If Go_ is preinstalled, filtron_ is simply +installed by ``go get`` package management (see `filtron README`_). If you use +filtron as middleware, a more isolated setup is recommended. To simplify such +an installation and the maintenance of, use our script :ref:`filtron.sh`. + +.. _Sample configuration of filtron: + Sample configuration of filtron =============================== +.. sidebar:: Tooling box + + - :origin:`/etc/filtron/rules.json <utils/templates/etc/filtron/rules.json>` + An example configuration can be find below. This configuration limits the access of: @@ -24,105 +57,104 @@ of: .. code:: json - [{ - "name":"search request", - "filters":[ - "Param:q", - "Path=^(/|/search)$" - ], - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "subrules":[ - { - "name":"roboagent limit", - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "filters":[ - "Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - }, - { - "name":"botlimit", - "limit":0, - "stop":true, - "filters":[ - "Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - }, - { - "name":"IP limit", - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "stop":true, - "aggregations":[ - "Header:X-Forwarded-For" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - }, - { - "name":"rss/json limit", - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "stop":true, - "filters":[ - "Param:format=(csv|json|rss)" - ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } - ] - }, - { - "name":"useragent limit", - "interval":"<time-interval-in-sec (int)>", - "limit":"<max-request-number-in-interval (int)>", - "aggregations":[ - "Header:User-Agent" + [ + { + "name": "search request", + "filters": [ + "Param:q", + "Path=^(/|/search)$" ], - "actions":[ - { - "name":"block", - "params":{ - "message":"Rate limit exceeded" - } - } + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "subrules": [ + { + "name": "missing Accept-Language", + "filters": ["!Header:Accept-Language"], + "limit": "<max-request-number-in-interval (int)>", + "stop": true, + "actions": [ + {"name":"log"}, + {"name": "block", + "params": {"message": "Rate limit exceeded"}} + ] + }, + { + "name": "suspiciously Connection=close header", + "filters": ["Header:Connection=close"], + "limit": "<max-request-number-in-interval (int)>", + "stop": true, + "actions": [ + {"name":"log"}, + {"name": "block", + "params": {"message": "Rate limit exceeded"}} + ] + }, + { + "name": "IP limit", + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "stop": true, + "aggregations": [ + "Header:X-Forwarded-For" + ], + "actions": [ + { "name": "log"}, + { "name": "block", + "params": { + "message": "Rate limit exceeded" + } + } + ] + }, + { + "name": "rss/json limit", + "filters": [ + "Param:format=(csv|json|rss)" + ], + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "stop": true, + "actions": [ + { "name": "log"}, + { "name": "block", + "params": { + "message": "Rate limit exceeded" + } + } + ] + }, + { + "name": "useragent limit", + "interval": "<time-interval-in-sec (int)>", + "limit": "<max-request-number-in-interval (int)>", + "aggregations": [ + "Header:User-Agent" + ], + "actions": [ + { "name": "log"}, + { "name": "block", + "params": { + "message": "Rate limit exceeded" + } + } + ] + } ] - } - ] - }] + } + ] +.. _filtron route request: Route request through filtron ============================= +.. sidebar:: further reading + + - :ref:`filtron.sh overview` + - :ref:`installation nginx` + - :ref:`installation apache` + Filtron can be started using the following command: .. code:: sh @@ -136,13 +168,24 @@ Use it along with ``nginx`` with the following example configuration. .. code:: nginx - location / { - proxy_set_header Host $http_host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Scheme $scheme; - proxy_pass http://127.0.0.1:4004/; + # https://example.org/searx + + location /searx { + proxy_pass http://127.0.0.1:4004/; + + proxy_set_header Host $host; + proxy_set_header Connection $http_connection; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Script-Name /searx; } + location /searx/static { + /usr/local/searx/searx-src/searx/static; + } + + Requests are coming from port 4004 going through filtron and then forwarded to -port 8888 where a searx is being run. +port 8888 where a searx is being run. For a complete setup see: :ref:`nginx +searx site`. diff --git a/docs/admin/index.rst b/docs/admin/index.rst index b3c7f51..c708c4f 100644 --- a/docs/admin/index.rst +++ b/docs/admin/index.rst @@ -3,9 +3,16 @@ Administrator documentation =========================== .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Contents installation + installation-searx + installation-uwsgi + installation-nginx + installation-apache + installation-docker + update-searx settings api architecture diff --git a/docs/admin/installation-apache.rst b/docs/admin/installation-apache.rst new file mode 100644 index 0000000..311b5c2 --- /dev/null +++ b/docs/admin/installation-apache.rst @@ -0,0 +1,515 @@ +.. _installation apache: + +=================== +Install with apache +=================== + +.. _Apache: https://httpd.apache.org/ +.. _Apache Debian: + https://cwiki.apache.org/confluence/display/HTTPD/DistrosDefaultLayout#DistrosDefaultLayout-Debian,Ubuntu(Apachehttpd2.x): +.. _README.Debian: + https://salsa.debian.org/apache-team/apache2/raw/master/debian/apache2.README.Debian +.. _Apache Arch Linux: + https://wiki.archlinux.org/index.php/Apache_HTTP_Server +.. _Apache Fedora: + https://docs.fedoraproject.org/en-US/quick-docs/getting-started-with-apache-http-server/index.html +.. _Apache directives: + https://httpd.apache.org/docs/trunk/mod/directives.html +.. _Getting Started: + https://httpd.apache.org/docs/current/en/getting-started.html +.. _Terms Used to Describe Directives: + https://httpd.apache.org/docs/current/en/mod/directive-dict.html +.. _Configuration Files: + https://httpd.apache.org/docs/current/en/configuring.html +.. _ProxyPreserveHost: https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypreservehost +.. _LoadModule: + https://httpd.apache.org/docs/2.4/mod/mod_so.html#loadmodule +.. _DocumentRoot: + https://httpd.apache.org/docs/trunk/mod/core.html#documentroot +.. _Location: + https://httpd.apache.org/docs/trunk/mod/core.html#location +.. _uWSGI Apache support: + https://uwsgi-docs.readthedocs.io/en/latest/Apache.html +.. _mod_proxy_uwsgi: + https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-proxy-uwsgi + +.. sidebar:: further read + + - `Apache Arch Linux`_ + - `Apache Debian`_ and `README.Debian`_ + - `Apache Fedora`_ + - `Apache directives`_ + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +---- + +**Install** :ref:`apache searx site` using :ref:`filtron.sh <filtron.sh overview>` + +.. code:: bash + + $ sudo -H ./utils/filtron.sh apache install + +**Install** :ref:`apache searx site` using :ref:`morty.sh <morty.sh overview>` + +.. code:: bash + + $ sudo -H ./utils/morty.sh apache install + +---- + +The apache HTTP server +====================== + +If Apache_ is not installed, install it now. If apache_ is new to you, the +`Getting Started`_, `Configuration Files`_ and `Terms Used to Describe +Directives`_ documentation gives first orientation. There is also a list of +`Apache directives`_ *to keep in the pocket*. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H apt-get install apache2 + + .. group-tab:: Arch Linux + + .. code:: sh + + sudo -H pacman -S apache + sudo -H systemctl enable httpd + sudo -H systemctl start http + + .. group-tab:: Fedora / RHEL + + .. code:: sh + + sudo -H dnf install httpd + sudo -H systemctl enable httpd + sudo -H systemctl start httpd + +Now at http://localhost you should see any kind of *Welcome* or *Test* page. +How this default intro site is configured, depends on the linux distribution +(compare `Apache directives`_). + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + less /etc/apache2/sites-enabled/000-default.conf + + In this file, there is a line setting the `DocumentRoot`_ directive: + + .. code:: apache + + DocumentRoot /var/www/html + + And the *welcome* page is the HTML file at ``/var/www/html/index.html``. + + .. group-tab:: Arch Linux + + .. code:: sh + + less /etc/httpd/conf/httpd.conf + + In this file, there is a line setting the `DocumentRoot`_ directive: + + .. code:: apache + + DocumentRoot "/srv/http" + <Directory "/srv/http"> + Options Indexes FollowSymLinks + AllowOverride None + Require all granted + </Directory> + + The *welcome* page of Arch Linux is a page showing directory located at + ``DocumentRoot``. This is *directory* page is generated by the Module + `mod_autoindex <https://httpd.apache.org/docs/2.4/mod/mod_autoindex.html>`_: + + .. code:: apache + + LoadModule autoindex_module modules/mod_autoindex.so + ... + Include conf/extra/httpd-autoindex.conf + + .. group-tab:: Fedora / RHEL + + .. code:: sh + + less /etc/httpd/conf/httpd.conf + + In this file, there is a line setting the ``DocumentRoot`` directive: + + .. code:: apache + + DocumentRoot "/var/www/html" + ... + <Directory "/var/www"> + AllowOverride None + # Allow open access: + Require all granted + </Directory> + + On fresh installations, the ``/var/www`` is empty and the *default + welcome page* is shown, the configuration is located at:: + + less /etc/httpd/conf.d/welcome.conf + +.. _apache searx site: + +Apache Reverse Proxy +==================== + +.. sidebar:: public to the internet? + + If your searx instance is public, stop here and first install :ref:`filtron + reverse proxy <filtron.sh>` and :ref:`result proxy morty <morty.sh>`, see + :ref:`installation scripts`. If already done, follow setup: *searx via + filtron plus morty*. + +To setup a Apache revers proxy you have to enable the *headers* and *proxy* +modules and create a `Location`_ configuration for the searx site. In most +distributions you have to un-comment the lines in the main configuration file, +except in :ref:`The Debian Layout`. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + In the Apache setup, enable headers and proxy modules: + + .. code:: sh + + sudo -H a2enmod headers + sudo -H a2enmod proxy + sudo -H a2enmod proxy_http + + In :ref:`The Debian Layout` you create a ``searx.conf`` with the + ``<Location /searx >`` directive and save this file in the *sites + available* folder at ``/etc/apache2/sites-available``. To enable the + ``searx.conf`` use :man:`a2ensite`: + + .. code:: sh + + sudo -H a2ensite searx.conf + + .. group-tab:: Arch Linux + + In the ``/etc/httpd/conf/httpd.conf`` file, activate headers and proxy + modules (LoadModule_): + + .. code:: apache + + FIXME needs test + + LoadModule headers_module modules/mod_headers.so + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_http_module modules/mod_proxy_http.so + + .. group-tab:: Fedora / RHEL + + In the ``/etc/httpd/conf/httpd.conf`` file, activate headers and proxy + modules (LoadModule_): + + .. code:: apache + + FIXME needs test + + LoadModule headers_module modules/mod_headers.so + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_http_module modules/mod_proxy_http.so + +With ProxyPreserveHost_ the incoming Host HTTP request header is passed to the +proxied host. + +.. _apache searx via filtron plus morty: + +.. tabs:: + + .. group-tab:: searx via filtron plus morty + + Use this setup, if your instance is public to the internet, compare + figure: :ref:`architecture <arch public>` and :ref:`installation scripts`. + + 1. Configure a reverse proxy for :ref:`filtron <filtron.sh>`, listening on + *localhost 4004* (:ref:`filtron route request`): + + .. code:: apache + + <Location /searx > + + # SetEnvIf Request_URI "/searx" dontlog + # CustomLog /dev/null combined env=dontlog + + Require all granted + + Order deny,allow + Deny from all + #Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1 + Allow from all + + ProxyPreserveHost On + ProxyPass http://127.0.0.1:4004 + RequestHeader set X-Script-Name /searx + + </Location> + + 2. Configure reverse proxy for :ref:`morty <searx morty>`, listening on + *localhost 3000* + + .. code:: apache + + ProxyPreserveHost On + + <Location /morty > + + # SetEnvIf Request_URI "/morty" dontlog + # CustomLog /dev/null combined env=dontlog + + Require all granted + + Order deny,allow + Deny from all + #Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1 + Allow from all + + ProxyPass http://127.0.0.1:3000 + RequestHeader set X-Script-Name /morty + + </Location> + + For a fully result proxification add :ref:`morty's <searx morty>` **public + URL** to your :origin:`searx/settings.yml`: + + .. code:: yaml + + result_proxy: + # replace example.org with your server's public name + url : https://example.org/morty + key : !!binary "insert_your_morty_proxy_key_here" + + server: + image_proxy : True + +uWSGI support +============= + +Be warned, with this setup, your instance isn't :ref:`protected <searx +filtron>`, nevertheless it is good enough for intranet usage. In modern Linux +distributions, the `mod_proxy_uwsgi`_ is compiled into the *normal* apache +package and you need to install only the :ref:`uWSGI <searx uwsgi>` package: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H apt-get install uwsgi + + # Ubuntu =< 18.04 + sudo -H apt-get install libapache2-mod-proxy-uwsgi + + .. group-tab:: Arch Linux + + .. code:: sh + + sudo -H pacman -S uwsgi + + .. group-tab:: Fedora / RHEL + + .. code:: sh + + sudo -H dnf install uwsgi + +The next example shows a configuration using the `uWSGI Apache support`_ via +unix sockets and `mod_proxy_uwsgi`_. + +For socket communication, you have to activate ``socket = +/run/uwsgi/app/searx/socket`` and comment out the ``http = 127.0.0.1:8888`` +configuration in your :ref:`uwsgi ini file <uwsgi configuration>`. If not +already exists, create a folder for the unix sockets, which can be used by the +searx account (see :ref:`create searx user`): + +.. code:: bash + + sudo -H mkdir -p /run/uwsgi/app/searx/ + sudo -H chown -R searx:searx /run/uwsgi/app/searx/ + +If the server is public; to limit access to your intranet replace ``Allow from +all`` directive and replace ``192.168.0.0/16`` with your subnet IP/class. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: apache + + LoadModule headers_module /usr/lib/apache2/mod_headers.so + LoadModule proxy_module /usr/lib/apache2/modules/mod_proxy.so + LoadModule proxy_uwsgi_module /usr/lib/apache2/modules/mod_proxy_uwsgi.so + + # SetEnvIf Request_URI /searx dontlog + # CustomLog /dev/null combined env=dontlog + + <Location /searx> + + Require all granted + Order deny,allow + Deny from all + # Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1 + Allow from all + + ProxyPreserveHost On + ProxyPass unix:/run/uwsgi/app/searx/socket|uwsgi://uwsgi-uds-searx/ + + </Location> + + .. group-tab:: Arch Linux + + .. code:: apache + + FIXME needs test + + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so + + # SetEnvIf Request_URI /searx dontlog + # CustomLog /dev/null combined env=dontlog + + <Location /searx> + + Require all granted + Order deny,allow + Deny from all + # Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1 + Allow from all + + ProxyPreserveHost On + ProxyPass unix:/run/uwsgi/app/searx/socket|uwsgi://uwsgi-uds-searx/ + + </Location> + + .. group-tab:: Fedora / RHEL + + .. code:: apache + + FIXME needs test + + LoadModule proxy_module modules/mod_proxy.so + LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so + <IfModule proxy_uwsgi_module> + + # SetEnvIf Request_URI /searx dontlog + # CustomLog /dev/null combined env=dontlog + + <Location /searx> + + Require all granted + Order deny,allow + Deny from all + # Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1 + Allow from all + + ProxyPreserveHost On + ProxyPass unix:/run/uwsgi/app/searx/socket|uwsgi://uwsgi-uds-searx/ + + </Location> + + </IfModule> + + .. group-tab:: old mod_wsgi + + We show this only for historical reasons, DON'T USE `mod_uwsgi + <https://uwsgi-docs.readthedocs.io/en/latest/Apache.html#mod-uwsgi>`_. + ANYMORE! + + .. code:: apache + + <IfModule mod_uwsgi.c> + + # SetEnvIf Request_URI "/searx" dontlog + # CustomLog /dev/null combined env=dontlog + + <Location /searx > + + Require all granted + + Options FollowSymLinks Indexes + SetHandler uwsgi-handler + uWSGISocket /run/uwsgi/app/searx/socket + + Order deny,allow + Deny from all + # Allow from fd00::/8 192.168.0.0/16 fe80::/10 127.0.0.0/8 ::1 + Allow from all + + </Location> + + </IfModule> + +.. _restart apache: + +Restart service +=============== + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H systemctl restart apache2 + sudo -H service uwsgi restart searx + + .. group-tab:: Arch Linux + + .. code:: sh + + sudo -H systemctl restart httpd + sudo -H systemctl restart uwsgi@searx + + .. group-tab:: Fedora / RHEL + + .. code:: sh + + sudo -H systemctl restart httpd + sudo -H touch /etc/uwsgi.d/searx.ini + + +disable logs +============ + +For better privacy you can disable Apache logs. In the examples above activate +one of the lines and `restart apache`_:: + + + # SetEnvIf Request_URI "/searx" dontlog + # CustomLog /dev/null combined env=dontlog + +The ``CustomLog`` directive disable logs for the whole (virtual) server, use it +when the URL of the service does not have a path component (``/searx``) / is +located at root (``/``). + +.. _The Debian Layout: + +The Debian Layout +================= + +Be aware that the Debian layout is quite different from the standard Apache +configuration. For details look at the README.Debian_ +(``/usr/share/doc/apache2/README.Debian.gz``). Some commands you should know on +Debian: + +* :man:`apache2ctl`: Apache HTTP server control interface +* :man:`a2enmod`, :man:`a2dismod`: switch on/off modules +* :man:`a2enconf`, :man:`a2disconf`: switch on/off configurations +* :man:`a2ensite`, :man:`a2dissite`: switch on/off sites diff --git a/docs/admin/installation-docker.rst b/docs/admin/installation-docker.rst new file mode 100644 index 0000000..13d21bc --- /dev/null +++ b/docs/admin/installation-docker.rst @@ -0,0 +1,60 @@ +.. _installation docker: + +=================== +Docker installation +=================== + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +---- + +Docker image searx/searx +======================== + + +The docker image is `searx/searx <https://hub.docker.com/r/searx/searx>`_ (based on `github.com/searx/searx <https://github.com/searx/searx>`_). + +Make sure you have `installed Docker <https://docs.docker.com/get-docker/>`_. For instance, you can deploy a local instance: + +.. code:: sh + + export PORT=80 + docker pull searx/searx + docker run --rm -d -v ${PWD}/searx:/etc/searx -p $PORT:8080 -e BASE_URL=http://localhost:$PORT/ searx/searx + +Go to ``http://localhost:$PORT``. + +Inside ``${PWD}/searx``, you will find ``settings.yml`` and ``uwsgi.ini``. +You can modify these files according to your needs and restart the Docker image. + + +Command line +------------ + + +.. code:: sh + + docker run --rm -it searx/searx -h + +.. program-output:: ../dockerfiles/docker-entrypoint.sh help + + +Build the image +--------------- + +It's also possible to build searx from the embedded Dockerfile. + +.. code:: sh + + git clone https://github.com/searx/searx.git + cd searx + make docker.build + + +Public instance +=============== + +If you intend to create a public instance using Docker, see https://github.com/searx/searx-docker diff --git a/docs/admin/installation-nginx.rst b/docs/admin/installation-nginx.rst new file mode 100644 index 0000000..97966c8 --- /dev/null +++ b/docs/admin/installation-nginx.rst @@ -0,0 +1,383 @@ +.. _installation nginx: + +================== +Install with nginx +================== + +.. _nginx: + https://docs.nginx.com/nginx/admin-guide/ +.. _nginx server configuration: + https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#setting-up-virtual-servers +.. _nginx beginners guide: + https://nginx.org/en/docs/beginners_guide.html +.. _Getting Started wiki: + https://www.nginx.com/resources/wiki/start/ +.. _uWSGI support from nginx: + https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html +.. _uwsgi_params: + https://uwsgi-docs.readthedocs.io/en/latest/Nginx.html#configuring-nginx +.. _SCRIPT_NAME: + https://werkzeug.palletsprojects.com/en/1.0.x/wsgi/#werkzeug.wsgi.get_script_name + +.. sidebar:: further reading + + - nginx_ + - `nginx beginners guide`_ + - `nginx server configuration`_ + - `Getting Started wiki`_ + - `uWSGI support from nginx`_ + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +---- + +**Install** :ref:`nginx searx site` using :ref:`filtron.sh <filtron.sh overview>` + +.. code:: bash + + $ sudo -H ./utils/filtron.sh nginx install + +**Install** :ref:`nginx searx site` using :ref:`morty.sh <morty.sh overview>` + +.. code:: bash + + $ sudo -H ./utils/morty.sh nginx install + +---- + + +The nginx HTTP server +===================== + +If nginx_ is not installed (uwsgi will not work with the package nginx-light), +install it now. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H apt-get install nginx + + .. group-tab:: Arch Linux + + .. code-block:: sh + + sudo -H pacman -S nginx-mainline + sudo -H systemctl enable nginx + sudo -H systemctl start nginx + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + sudo -H dnf install nginx + sudo -H systemctl enable nginx + sudo -H systemctl start nginx + +Now at http://localhost you should see a *Welcome to nginx!* page, on Fedora you +see a *Fedora Webserver - Test Page*. The test page comes from the default +`nginx server configuration`_. How this default intro site is configured, +depends on the linux distribution: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + less /etc/nginx/nginx.conf + + there is a line including site configurations from: + + .. code:: nginx + + include /etc/nginx/sites-enabled/*; + + .. group-tab:: Arch Linux + + .. code-block:: sh + + less /etc/nginx/nginx.conf + + in there is a configuration section named ``server``: + + .. code-block:: nginx + + server { + listen 80; + server_name localhost; + # ... + } + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + less /etc/nginx/nginx.conf + + there is a line including site configurations from: + + .. code:: nginx + + include /etc/nginx/conf.d/*.conf; + +.. _nginx searx site: + +A nginx searx site +================== + +.. sidebar:: public to the internet? + + If your searx instance is public, stop here and first install :ref:`filtron + reverse proxy <filtron.sh>` and :ref:`result proxy morty <morty.sh>`, see + :ref:`installation scripts`. If already done, follow setup: *searx via + filtron plus morty*. + +Now you have to create a configuration for the searx site. If nginx_ is new to +you, the `nginx beginners guide`_ is a good starting point and the `Getting +Started wiki`_ is always a good resource *to keep in the pocket*. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + Create configuration at ``/etc/nginx/sites-available/searx`` and place a + symlink to sites-enabled: + + .. code:: sh + + sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx + + .. group-tab:: Arch Linux + + In the ``/etc/nginx/nginx.conf`` file, replace the configuration section + named ``server``. + + .. group-tab:: Fedora / RHEL + + Create configuration at ``/etc/nginx/conf.d/searx`` and place a + symlink to sites-enabled: + +.. _nginx searx via filtron plus morty: + +.. tabs:: + + .. group-tab:: searx via filtron plus morty + + Use this setup, if your instance is public to the internet, compare + figure: :ref:`architecture <arch public>` and :ref:`installation scripts`. + + 1. Configure a reverse proxy for :ref:`filtron <filtron.sh>`, listening on + *localhost 4004* (:ref:`filtron route request`): + + .. code:: nginx + + # https://example.org/searx + + location /searx { + proxy_pass http://127.0.0.1:4004/; + + proxy_set_header Host $host; + proxy_set_header Connection $http_connection; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Script-Name /searx; + } + + location /searx/static/ { + alias /usr/local/searx/searx-src/searx/static/; + } + + + 2. Configure reverse proxy for :ref:`morty <searx morty>`, listening on + *localhost 3000*: + + .. code:: nginx + + # https://example.org/morty + + location /morty { + proxy_pass http://127.0.0.1:3000/; + + proxy_set_header Host $host; + proxy_set_header Connection $http_connection; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Scheme $scheme; + } + + For a fully result proxification add :ref:`morty's <searx morty>` **public + URL** to your :origin:`searx/settings.yml`: + + .. code:: yaml + + result_proxy: + # replace example.org with your server's public name + url : https://example.org/morty + key : !!binary "insert_your_morty_proxy_key_here" + + server: + image_proxy : True + + + .. group-tab:: proxy or uWSGI + + Be warned, with this setup, your instance isn't :ref:`protected <searx + filtron>`. Nevertheless it is good enough for intranet usage and it is a + excellent example of; *how different services can be set up*. The next + example shows a reverse proxy configuration wrapping the :ref:`searx-uWSGI + application <uwsgi configuration>`, listening on ``http = + 127.0.0.1:8888``. + + .. code:: nginx + + # https://hostname.local/ + + location / { + proxy_pass http://127.0.0.1:8888; + + proxy_set_header Host $host; + proxy_set_header Connection $http_connection; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Scheme $scheme; + proxy_buffering off; + } + + Alternatively you can use the `uWSGI support from nginx`_ via unix + sockets. For socket communication, you have to activate ``socket = + /run/uwsgi/app/searx/socket`` and comment out the ``http = + 127.0.0.1:8888`` configuration in your :ref:`uwsgi ini file <uwsgi + configuration>`. + + The example shows a nginx virtual ``server`` configuration, listening on + port 80 (IPv4 and IPv6 http://[::]:80). The uWSGI app is configured at + location ``/`` by importing the `uwsgi_params`_ and passing requests to + the uWSGI socket (``uwsgi_pass``). The ``server``\'s root points to the + :ref:`searx-src clone <searx-src>` and wraps directly the + :origin:`searx/static/` content at ``location /static``. + + .. code:: nginx + + server { + # replace hostname.local with your server's name + server_name hostname.local; + + listen 80; + listen [::]:80; + + location / { + include uwsgi_params; + uwsgi_pass unix:/run/uwsgi/app/searx/socket; + } + + root /usr/local/searx/searx-src/searx; + location /static { } + } + + If not already exists, create a folder for the unix sockets, which can be + used by the searx account: + + .. code:: bash + + mkdir -p /run/uwsgi/app/searx/ + sudo -H chown -R searx:searx /run/uwsgi/app/searx/ + + .. group-tab:: \.\. at subdir URL + + Be warned, with these setups, your instance isn't :ref:`protected <searx + filtron>`. The examples are just here to demonstrate how to export the + searx application from a subdirectory URL ``https://example.org/searx/``. + + .. code:: nginx + + # https://hostname.local/searx + + location /searx { + proxy_pass http://127.0.0.1:8888; + + proxy_set_header Host $host; + proxy_set_header Connection $http_connection; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Script-Name /searx; + proxy_buffering off; + } + + location /searx/static/ { + alias /usr/local/searx/searx-src/searx/static/; + } + + The ``X-Script-Name /searx`` is needed by the searx implementation to + calculate relative URLs correct. The next example shows a uWSGI + configuration. Since there are no HTTP headers in a (u)WSGI protocol, the + value is shipped via the SCRIPT_NAME_ in the WSGI environment. + + .. code:: nginx + + # https://hostname.local/searx + + location /searx { + uwsgi_param SCRIPT_NAME /searx; + include uwsgi_params; + uwsgi_pass unix:/run/uwsgi/app/searx/socket; + } + + location /searx/static/ { + alias /usr/local/searx/searx-src/searx/; + } + + For searx to work correctly the ``base_url`` must be set in the + :origin:`searx/settings.yml`. + + .. code:: yaml + + server: + # replace example.org with your server's public name + base_url : https://example.org/searx/ + + +Restart service: + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code:: sh + + sudo -H systemctl restart nginx + sudo -H service uwsgi restart searx + + .. group-tab:: Arch Linux + + .. code:: sh + + sudo -H systemctl restart nginx + sudo -H systemctl restart uwsgi@searx + + .. group-tab:: Fedora + + .. code:: sh + + sudo -H systemctl restart nginx + sudo -H touch /etc/uwsgi.d/searx.ini + + +Disable logs +============ + +For better privacy you can disable nginx logs in ``/etc/nginx/nginx.conf``. + +.. code:: nginx + + http { + # ... + access_log /dev/null; + error_log /dev/null; + # ... + } diff --git a/docs/admin/installation-searx.rst b/docs/admin/installation-searx.rst new file mode 100644 index 0000000..adea016 --- /dev/null +++ b/docs/admin/installation-searx.rst @@ -0,0 +1,120 @@ +.. _installation basic: + +========================= +Step by step installation +========================= + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +Step by step installation with virtualenv. For Ubuntu, be sure to have enable +universe repository. + +.. _install packages: + +Install packages +================ + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START distro-packages + :end-before: END distro-packages + +.. hint:: + + This installs also the packages needed by :ref:`searx uwsgi` + +.. _create searx user: + +Create user +=========== + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START create user + :end-before: END create user + +.. _searx-src: + +install searx & dependencies +============================ + +Start a interactive shell from new created user and clone searx: + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START clone searx + :end-before: END clone searx + +In the same shell create *virtualenv*: + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START create virtualenv + :end-before: END create virtualenv + +To install searx's dependencies, exit the searx *bash* session you opened above +and restart a new. Before install, first check if your *virtualenv* was sourced +from the login (*~/.profile*): + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START manage.sh update_packages + :end-before: END manage.sh update_packages + +.. tip:: + + Open a second terminal for the configuration tasks and leave the ``(searx)$`` + terminal open for the tasks below. + + +.. _use_default_settings.yml: + +Configuration +============= + +.. sidebar:: ``use_default_settings: True`` + + - :ref:`settings global` + - :ref:`settings location` + - :ref:`settings use_default_settings` + - :origin:`/etc/searx/settings.yml <utils/templates/etc/searx/use_default_settings.yml>` + +To create a initial ``/etc/searx/settings.yml`` you can start with a copy of the +file :origin:`utils/templates/etc/searx/use_default_settings.yml`. This setup +:ref:`use default settings <settings use_default_settings>` from +:origin:`searx/settings.yml` and is recommended since :pull:`2291` is merged. + +For a *minimal setup*, configure like shown below – replace ``searx@$(uname +-n)`` with a name of your choice, set ``ultrasecretkey`` -- *and/or* edit +``/etc/searx/settings.yml`` to your needs. + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START searx config + :end-before: END searx config + +.. tabs:: + + .. group-tab:: Use default settings + + .. literalinclude:: ../../utils/templates/etc/searx/use_default_settings.yml + :language: yaml + + .. group-tab:: searx/settings.yml + + .. literalinclude:: ../../searx/settings.yml + :language: yaml + + +Check +===== + +To check your searx setup, optional enable debugging and start the *webapp*. +Searx looks at the exported environment ``$SEARX_SETTINGS_PATH`` for a +configuration file. + +.. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START check searx installation + :end-before: END check searx installation + +If everything works fine, hit ``[CTRL-C]`` to stop the *webapp* and disable the +debug option in ``settings.yml``. You can now exit searx user bash (enter exit +command twice). At this point searx is not demonized; uwsgi allows this. + diff --git a/docs/admin/installation-uwsgi.rst b/docs/admin/installation-uwsgi.rst new file mode 100644 index 0000000..7b48297 --- /dev/null +++ b/docs/admin/installation-uwsgi.rst @@ -0,0 +1,150 @@ +.. _searx uwsgi: + +===== +uwsgi +===== + +.. sidebar:: further reading + + - `systemd.unit`_ + - `uWSGI Emperor`_ + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + + +.. _systemd.unit: https://www.freedesktop.org/software/systemd/man/systemd.unit.html +.. _One service per app in systemd: + https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html#one-service-per-app-in-systemd +.. _uWSGI Emperor: + https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html +.. _uwsgi ini file: + https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#ini-files +.. _systemd unit template: + http://0pointer.de/blog/projects/instances.html + + +Origin uWSGI +============ + +How uWSGI is implemented by distributors is different. uWSGI itself +recommend two methods + +`systemd.unit`_ template files as described here `One service per app in systemd`_. + + There is one `systemd unit template`_ and one `uwsgi ini file`_ per uWSGI-app + placed at dedicated locations. Take archlinux and a searx.ini as example:: + + unit template --> /usr/lib/systemd/system/uwsgi@.service + uwsgi ini files --> /etc/uwsgi/searx.ini + + The searx app can be maintained as know from common systemd units:: + + systemctl enable uwsgi@searx + systemctl start uwsgi@searx + systemctl restart uwsgi@searx + systemctl stop uwsgi@searx + +The `uWSGI Emperor`_ mode which fits for maintaining a large range of uwsgi apps. + + The Emperor mode is a special uWSGI instance that will monitor specific + events. The Emperor mode (service) is started by a (common, not template) + systemd unit. The Emperor service will scan specific directories for `uwsgi + ini file`_\s (also know as *vassals*). If a *vassal* is added, removed or the + timestamp is modified, a corresponding action takes place: a new uWSGI + instance is started, reload or stopped. Take Fedora and a searx.ini as + example:: + + to start a new searx instance create --> /etc/uwsgi.d/searx.ini + to reload the instance edit timestamp --> touch /etc/uwsgi.d/searx.ini + to stop instance remove ini --> rm /etc/uwsgi.d/searx.ini + +Distributors +============ + +The `uWSGI Emperor`_ mode and `systemd unit template`_ is what the distributors +mostly offer their users, even if they differ in the way they implement both +modes and their defaults. Another point they might differ is the packaging of +plugins (if so, compare :ref:`install packages`) and what the default python +interpreter is (python2 vs. python3). + +Fedora starts a Emperor by default, while archlinux does not start any uwsgi +service by default. Worth to know; debian (ubuntu) follow a complete different +approach. *debian*: your are familiar with the apache infrastructure? .. they +do similar for the uWSGI infrastructure (with less comfort), the folders are:: + + /etc/uwsgi/apps-available/ + /etc/uwsgi/apps-enabled/ + +The `uwsgi ini file`_ is enabled by a symbolic link:: + + ln -s /etc/uwsgi/apps-available/searx.ini /etc/uwsgi/apps-enabled/ + +From debian's documentation (``/usr/share/doc/uwsgi/README.Debian.gz``): You +could control specific instance(s) by issuing:: + + service uwsgi <command> <confname> <confname> ... + + sudo -H service uwsgi start searx + sudo -H service uwsgi stop searx + +My experience is, that this command is a bit buggy. + +.. _uwsgi configuration: + +Alltogether +=========== + +Create the configuration ini-file according to your distribution (see below) and +restart the uwsgi application. + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START searx uwsgi-description ubuntu-20.04 + :end-before: END searx uwsgi-description ubuntu-20.04 + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Arch Linux + + .. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START searx uwsgi-description arch + :end-before: END searx uwsgi-description arch + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Fedora / RHEL + + .. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START searx uwsgi-description fedora + :end-before: END searx uwsgi-description fedora + + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START searx uwsgi-appini ubuntu-20.04 + :end-before: END searx uwsgi-appini ubuntu-20.04 + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Arch Linux + + .. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START searx uwsgi-appini arch + :end-before: END searx uwsgi-appini arch + + .. hotfix: a bug group-tab need this comment + + .. group-tab:: Fedora / RHEL + + .. kernel-include:: $DOCS_BUILD/includes/searx.rst + :start-after: START searx uwsgi-appini fedora + :end-before: END searx uwsgi-appini fedora diff --git a/docs/admin/installation.rst b/docs/admin/installation.rst index 15800fc..9bb102c 100644 --- a/docs/admin/installation.rst +++ b/docs/admin/installation.rst @@ -4,346 +4,83 @@ Installation ============ -.. contents:: - :depth: 3 +*You're spoilt for choice*, choose your preferred method of installation. -Basic installation -================== +- :ref:`installation docker` +- :ref:`installation scripts` +- :ref:`installation basic` -Step by step installation for Debian/Ubuntu with virtualenv. For Ubuntu, be sure -to have enable universe repository. +The :ref:`installation basic` is good enough for intranet usage and it is a +excellent illustration of *how a searx instance is build up*. If you place your +instance public to the internet you should really consider to install a +:ref:`filtron reverse proxy <filtron.sh>` and for privacy a :ref:`result proxy +<morty.sh>` is mandatory. -Install packages: +Therefore, if you do not have any special preferences, its recommend to use the +:ref:`installation docker` or the `Installation scripts`_ from our :ref:`tooling +box <toolboxing>` as described below. -.. code:: sh +.. _installation scripts: - $ sudo -H apt-get install \ - git build-essential libxslt-dev \ - python-dev python-virtualenv python-babel \ - zlib1g-dev libffi-dev libssl-dev +Installation scripts +==================== -Install searx: +.. sidebar:: Update OS first! -.. code:: sh + To avoid unwanted side effects, update your OS before installing searx. - cd /usr/local - sudo -H git clone https://github.com/asciimoo/searx.git - sudo -H useradd searx -d /usr/local/searx - sudo -H chown searx:searx -R /usr/local/searx +The following will install a setup as shown in :ref:`architecture`. First you +need to get a clone. The clone is only needed for the installation procedure +and some maintenance tasks (alternatively you can create your own fork). -Install dependencies in a virtualenv: +For the installation procedure, use a *sudoer* login to run the scripts. If you +install from ``root``, take into account that the scripts are creating a +``searx``, a ``filtron`` and a ``morty`` user. In the installation procedure +these new created users do need read access to the clone of searx, which is not +the case if you clone into a folder below ``/root``. -.. code:: sh +.. code:: bash - cd /usr/local/searx - sudo -H -u searx -i + $ cd ~/Downloads + $ git clone https://github.com/searx/searx searx + $ cd searx -.. code:: sh +.. sidebar:: further read - (searx)$ virtualenv searx-ve - (searx)$ . ./searx-ve/bin/activate - (searx)$ ./manage.sh update_packages + - :ref:`toolboxing` + - :ref:`update searx` + - :ref:`inspect searx` -Configuration -============== +**Install** :ref:`searx service <searx.sh>` -.. code:: sh +This installs searx as described in :ref:`installation basic`. - sed -i -e "s/ultrasecretkey/`openssl rand -hex 16`/g" searx/settings.yml +.. code:: bash -Edit searx/settings.yml if necessary. + $ sudo -H ./utils/searx.sh install all -Check -===== +**Install** :ref:`filtron reverse proxy <filtron.sh>` -Start searx: +.. code:: bash -.. code:: sh + $ sudo -H ./utils/filtron.sh install all - python searx/webapp.py +**Install** :ref:`result proxy <morty.sh>` -Go to http://localhost:8888 +.. code:: bash -If everything works fine, disable the debug option in settings.yml: + $ sudo -H ./utils/morty.sh install all -.. code:: sh +If all services are running fine, you can add it to your HTTP server: - sed -i -e "s/debug : True/debug : False/g" searx/settings.yml +- :ref:`installation apache` +- :ref:`installation nginx` -At this point searx is not demonized ; uwsgi allows this. +.. _git stash: https://git-scm.com/docs/git-stash -You can exit the virtualenv and the searx user bash (enter exit command -twice). +.. tip:: -uwsgi -===== - -Install packages: - -.. code:: sh - - sudo -H apt-get install \ - uwsgi uwsgi-plugin-python - -Create the configuration file ``/etc/uwsgi/apps-available/searx.ini`` with this -content: - -.. code:: ini - - [uwsgi] - # Who will run the code - uid = searx - gid = searx - - # disable logging for privacy - disable-logging = true - - # Number of workers (usually CPU count) - workers = 4 - - # The right granted on the created socket - chmod-socket = 666 - - # Plugin to use and interpretor config - single-interpreter = true - master = true - plugin = python - lazy-apps = true - enable-threads = true - - # Module to import - module = searx.webapp - - # Support running the module from a webserver subdirectory. - route-run = fixpathinfo: - - # Virtualenv and python path - virtualenv = /usr/local/searx/searx-ve/ - pythonpath = /usr/local/searx/ - chdir = /usr/local/searx/searx/ - -Activate the uwsgi application and restart: - -.. code:: sh - - cd /etc/uwsgi/apps-enabled - ln -s ../apps-available/searx.ini - /etc/init.d/uwsgi restart - -Web server -========== - -with nginx ----------- - -If nginx is not installed (uwsgi will not work with the package -nginx-light): - -.. code:: sh - - sudo -H apt-get install nginx - -Hosted at / -~~~~~~~~~~~ - -Create the configuration file ``/etc/nginx/sites-available/searx`` with this -content: - -.. code:: nginx - - server { - listen 80; - server_name searx.example.com; - root /usr/local/searx/searx; - - location /static { - } - - location / { - include uwsgi_params; - uwsgi_pass unix:/run/uwsgi/app/searx/socket; - } - } - -Create a symlink to sites-enabled: - -.. code:: sh - - sudo -H ln -s /etc/nginx/sites-available/searx /etc/nginx/sites-enabled/searx - -Restart service: - -.. code:: sh - - sudo -H service nginx restart - sudo -H service uwsgi restart - -from subdirectory URL (/searx) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Add this configuration in the server config file -``/etc/nginx/sites-enabled/default``: - -.. code:: nginx - - location /searx/static { - alias /usr/local/searx/searx/static; - } - - location /searx { - uwsgi_param SCRIPT_NAME /searx; - include uwsgi_params; - uwsgi_pass unix:/run/uwsgi/app/searx/socket; - } - - -**OR** using reverse proxy (Please, note that reverse proxy advised to be used -in case of single-user or low-traffic instances.) - -.. code:: nginx - - location /searx/static { - alias /usr/local/searx/searx/static; - } - - location /searx { - proxy_pass http://127.0.0.1:8888; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Scheme $scheme; - proxy_set_header X-Script-Name /searx; - proxy_buffering off; - } - - -Enable ``base_url`` in ``searx/settings.yml`` - -.. code:: yaml - - base_url : http://your.domain.tld/searx/ - -Restart service: - -.. code:: sh - - sudo -H service nginx restart - sudo -H service uwsgi restart - -disable logs -^^^^^^^^^^^^ - -for better privacy you can disable nginx logs about searx. - -how to proceed: below ``uwsgi_pass`` in ``/etc/nginx/sites-available/default`` -add: - -.. code:: nginx - - access_log /dev/null; - error_log /dev/null; - -Restart service: - -.. code:: sh - - sudo -H service nginx restart - -with apache ------------ - -Add wsgi mod: - -.. code:: sh - - sudo -H apt-get install libapache2-mod-uwsgi - sudo -H a2enmod uwsgi - -Add this configuration in the file ``/etc/apache2/apache2.conf``: - -.. code:: apache - - <Location /> - Options FollowSymLinks Indexes - SetHandler uwsgi-handler - uWSGISocket /run/uwsgi/app/searx/socket - </Location> - -Note that if your instance of searx is not at the root, you should change -``<Location />`` by the location of your instance, like ``<Location /searx>``. - -Restart Apache: - -.. code:: sh - - sudo -H /etc/init.d/apache2 restart - -disable logs -~~~~~~~~~~~~ - -For better privacy you can disable Apache logs. - -.. warning:: - - You can only disable logs for the whole (virtual) server not for a specific - path. - -Go back to ``/etc/apache2/apache2.conf`` and above ``<Location />`` add: - -.. code:: apache - - CustomLog /dev/null combined - -Restart Apache: - -.. code:: sh - - sudo -H /etc/init.d/apache2 restart - -How to update -============= - -.. code:: sh - - cd /usr/local/searx - sudo -H -u searx -i - -.. code:: sh - - (searx)$ . ./searx-ve/bin/activate - (searx)$ git stash - (searx)$ git pull origin master - (searx)$ git stash apply - (searx)$ ./manage.sh update_packages - -.. code:: sh - - sudo -H service uwsgi restart - -Docker -====== - -Make sure you have installed Docker. For instance, you can deploy searx like this: - -.. code:: sh - - docker pull wonderfall/searx - docker run -d --name searx -p $PORT:8888 wonderfall/searx - -Go to ``http://localhost:$PORT``. - -See https://hub.docker.com/r/wonderfall/searx/ for more informations. It's also -possible to build searx from the embedded Dockerfile. - -.. code:: sh - - git clone https://github.com/asciimoo/searx.git - cd searx - docker build -t whatever/searx . - -References -========== - -* https://about.okhin.fr/posts/Searx/ with some additions - -* How to: `Setup searx in a couple of hours with a free SSL certificate - <https://www.reddit.com/r/privacytoolsIO/comments/366kvn/how_to_setup_your_own_privacy_respecting_search/>`__ + About script's installation options have a look at chapter :ref:`toolboxing + setup`. How to brand your instance see chapter :ref:`settings global`. To + *stash* your instance's setup, `git stash`_ your clone's :origin:`Makefile` + and :origin:`.config.sh` file . diff --git a/docs/admin/morty.rst b/docs/admin/morty.rst index 7d7b344..5468c9c 100644 --- a/docs/admin/morty.rst +++ b/docs/admin/morty.rst @@ -1,7 +1,14 @@ + +.. _searx morty: + ========================= How to setup result proxy ========================= +.. sidebar:: further reading + + - :ref:`morty.sh` + .. _morty: https://github.com/asciimoo/morty .. _morty's README: https://github.com/asciimoo/morty @@ -9,15 +16,22 @@ By default searx can only act as an image proxy for result images, but it is possible to proxify all the result URLs with an external service, morty_. To use this feature, morty has to be installed and activated in searx's -``settings.yml``. - -Add the following snippet to your ``settings.yml`` and restart searx: +``settings.yml``. Add the following snippet to your ``settings.yml`` and +restart searx: .. code:: yaml result_proxy: url : http://127.0.0.1:3000/ - key : your_morty_proxy_key + key : !!binary "insert_your_morty_proxy_key_here" + +Note that the example above (``http://127.0.0.1:3000``) is only for single-user +instances without a HTTP proxy. If your morty service is public, the url is the +address of the reverse proxy (e.g ``https://example.org/morty``). + +For more information about *result proxy* have a look at *"searx via filtron +plus morty"* in the :ref:`nginx <nginx searx via filtron plus morty>` and +:ref:`apache <apache searx via filtron plus morty>` sections. ``url`` Is the address of the running morty service. diff --git a/docs/admin/plugins.rst b/docs/admin/plugins.rst index 4ed9066..d97b3da 100644 --- a/docs/admin/plugins.rst +++ b/docs/admin/plugins.rst @@ -14,7 +14,7 @@ Configuration defaults (at built time): .. _configured plugins: -.. jinja:: webapp +.. jinja:: searx .. flat-table:: Plugins configured at built time (defaults) :header-rows: 1 diff --git a/docs/admin/settings.rst b/docs/admin/settings.rst index 0bfdcc6..7cf055d 100644 --- a/docs/admin/settings.rst +++ b/docs/admin/settings.rst @@ -4,110 +4,193 @@ ``settings.yml`` ================ +This page describe the options possibilities of the :origin:`searx/settings.yml` +file. + .. sidebar:: Further reading .. + - :ref:`use_default_settings.yml` - :ref:`search API` -This page describe the options possibilities of the settings.yml file. +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +.. _settings location: + +settings.yml location +===================== + +First, searx will try to load settings.yml from these locations: + +1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable. +2. ``/etc/searx/settings.yml`` + +If these files don't exist (or are empty or can't be read), searx uses the +:origin:`searx/settings.yml` file. + .. _settings global: Global Settings =============== +``general:`` +------------ + .. code:: yaml - server: - port : 8888 - secret_key : "ultrasecretkey" # change this! - debug : False # debug mode, only for development - request_timeout : 2.0 # seconds - base_url : False # set custom base_url (or False) - themes_path : "" # custom ui themes path - default_theme : oscar # ui theme - useragent_suffix : "" # suffix of searx_useragent, could contain - # informations like admins email address - image_proxy : False # proxying image results through searx - default_locale : "" # default interface locale + general: + debug : False # Debug mode, only for development + instance_name : "searx" # displayed name + git_url: https://github.com/searx/searx + git_branch: master + issue_url: https://github.com/searx/searx/issues + docs_url: https://searx.github.io/searx + public_instances: https://searx.space + contact_url: False # mailto:contact@example.com + wiki_url: https://github.com/searx/searx/wiki + twitter_url: https://twitter.com/Searx_engine - # uncomment below section if you want to use a proxy +``debug`` : + Allow a more detailed log if you run searx directly. Display *detailed* error + messages in the browser too, so this must be deactivated in production. - #outgoing_proxies : - # http : http://127.0.0.1:8080 - # https: http://127.0.0.1:8080 +``contact_url``: + Contact ``mailto:`` address or WEB form. - # uncomment below section only if you have more than one network interface - # which can be the source of outgoing search requests +``git_url`` and ``git_branch``: + Changes this, to point to your searx fork (branch). - #source_ips: - # - 1.1.1.1 - # - 1.1.1.2 +``docs_url`` + If you host your own documentation, change this URL. - locales: - en : English - de : Deutsch - he : Hebrew - hu : Magyar - fr : Français - es : Español - it : Italiano - nl : Nederlands - ja : 日本語 (Japanese) - tr : Türkçe - ru : Russian - ro : Romanian +``wiki_url``: + Link to your wiki (or ``False``) +``twitter_url``: + Link to your tweets (or ``False``) -``port`` : - Port number of the searx web application if you run it directly using ``python - searx/webapp.py``. Doesn't apply to searx running on Apache or Nginx. -``secret_key`` : - Used for cryptography purpose. +``server:`` +----------- -``debug`` : - Allow a more detailed log if you run searx directly. Display *detailed* error - messages in the browser too, so this must be deactivated in production. +.. code:: yaml -``request_timeout`` : - Global timeout of the requests made to others engines in seconds. A bigger - timeout will allow to wait for answers from slow engines, but in consequence - will slow searx reactivity (the result page may take the time specified in the - timeout to load) + server: + port : 8888 + bind_address : "127.0.0.1" # address to listen on + secret_key : "ultrasecretkey" # change this! + base_url : False # set custom base_url (or False) + image_proxy : False # proxying image results through searx + default_locale : "" # default interface locale + default_theme : oscar # ui theme + default_http_headers: + X-Content-Type-Options : nosniff + X-XSS-Protection : 1; mode=block + X-Download-Options : noopen + X-Robots-Tag : noindex, nofollow + Referrer-Policy : no-referrer + +``port`` & ``bind_address``: + Port number and *bind address* of the searx web application if you run it + directly using ``python searx/webapp.py``. Doesn't apply to searx running on + Apache or Nginx. + +``secret_key`` : + Used for cryptography purpose. ``base_url`` : The base URL where searx is deployed. Used to create correct inbound links. -``themes_path`` : - Path to where the themes are located. If you didn't develop anything, leave it - blank. - -``default_theme`` : - Name of the theme you want to use by default on you searx instance. - -``useragent_suffix`` : - Suffix to the user-agent searx uses to send requests to others engines. If an - engine wish to block you, a contact info here may be useful to avoid that. - ``image_proxy`` : Allow your instance of searx of being able to proxy images. Uses memory space. ``default_locale`` : - Aearx interface language. If blank, the locale is detected by using the + Searx interface language. If blank, the locale is detected by using the browser language. If it doesn't work, or you are deploying a language specific instance of searx, a locale can be defined using an ISO language code, like ``fr``, ``en``, ``de``. -.. _requests proxies: http://docs.python-requests.org/en/latest/user/advanced/#proxies -.. _PR SOCKS support: https://github.com/kennethreitz/requests/pull/478 +``default_theme`` : + Name of the theme you want to use by default on your searx instance. + +.. _HTTP headers: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers + +``default_http_headers``: + Set additional HTTP headers, see `#755 <https://github.com/searx/searx/issues/715>`__ + +``outgoing:`` +------------- + +.. code:: yaml + + outgoing: # communication with search engines + request_timeout : 2.0 # default timeout in seconds, can be override by engine + # max_request_timeout: 10.0 # the maximum timeout in seconds + useragent_suffix : "" # informations like an email address to the administrator + pool_connections : 100 # Number of different hosts + pool_maxsize : 10 # Number of simultaneous requests by host + # uncomment below section if you want to use a proxy + # proxies: + # http: + # - http://proxy1:8080 + # - http://proxy2:8080 + # https: + # - http://proxy1:8080 + # - http://proxy2:8080 + # uncomment below section only if you have more than one network interface + # which can be the source of outgoing search requests + # source_ips: + # - 1.1.1.1 + # - 1.1.1.2 + + +``request_timeout`` : + Global timeout of the requests made to others engines in seconds. A bigger + timeout will allow to wait for answers from slow engines, but in consequence + will slow searx reactivity (the result page may take the time specified in the + timeout to load). Can be override by :ref:`settings engine` + +``useragent_suffix`` : + Suffix to the user-agent searx uses to send requests to others engines. If an + engine wish to block you, a contact info here may be useful to avoid that. + +.. _requests proxies: https://requests.readthedocs.io/en/latest/user/advanced/#proxies +.. _PySocks: https://pypi.org/project/PySocks/ + +``proxies`` : + Define one or more proxies you wish to use, see `requests proxies`_. + If there are more than one proxy for one protocol (http, https), + requests to the engines are distributed in a round-robin fashion. -``outgoing_proxies`` : - Define a proxy you wish to use, see `requests proxies`_. SOCKS proxies are - not supported / see `PR SOCKS support`. + - Proxy: `see <https://2.python-requests.org/en/latest/user/advanced/#proxies>`__. + - SOCKS proxies are also supported: `see <https://2.python-requests.org/en/latest/user/advanced/#socks>`__ ``source_ips`` : - If you use multiple nework interfaces, define from which IP the requests must - be made. + If you use multiple network interfaces, define from which IP the requests must + be made. This parameter is ignored when ``proxies`` is set. + + +``locales:`` +------------ + +.. code:: yaml + + locales: + en : English + de : Deutsch + he : Hebrew + hu : Magyar + fr : Français + es : Español + it : Italiano + nl : Nederlands + ja : 日本語 (Japanese) + tr : Türkçe + ru : Russian + ro : Romanian ``locales`` : Locales codes and their names. Available translations of searx interface. @@ -133,9 +216,18 @@ Engine settings api_key : 'apikey' disabled : True language : en_US + #proxies: + # http: + # - http://proxy1:8080 + # - http://proxy2:8080 + # https: + # - http://proxy1:8080 + # - http://proxy2:8080 + # - socks5://user:password@proxy3:1080 + # - socks5h://user:password@proxy4:1080 ``name`` : - Name that will be used accross searx to define this engine. In settings, on + Name that will be used across searx to define this engine. In settings, on the result page... ``engine`` : @@ -146,7 +238,7 @@ Engine settings Code used to execute bang requests (in this case using ``!bi`` or ``?bi``) ``base_url`` : optional - Part of the URL that should be stable accross every request. Can be useful to + Part of the URL that should be stable across every request. Can be useful to use multiple sites using only one engine, or updating the site URL without touching at the code. @@ -164,7 +256,7 @@ Engine settings is described in the file. ``disabled`` : optional - To disable by default the engine, but not deleting it. It will allow the user + To disable by default the engine, but not deleting it. It will allow the user to manually activate it in the settings. ``language`` : optional @@ -175,7 +267,90 @@ Engine settings ``weigth`` : default ``1`` Weighting of the results of this engine. +``display_error_messages`` : default ``True`` + When an engine returns an error, the message is displayed on the user interface. + .. note:: A few more options are possible, but they are pretty specific to some engines, and so won't be described here. + + +.. _settings use_default_settings: + +use_default_settings +==================== + +.. sidebar:: ``use_default_settings: True`` + + - :ref:`settings location` + - :ref:`use_default_settings.yml` + - :origin:`/etc/searx/settings.yml <utils/templates/etc/searx/use_default_settings.yml>` + +The user defined ``settings.yml`` is loaded from the :ref:`settings location` +and can relied on the default configuration :origin:`searx/settings.yml` using: + + ``use_default_settings: True`` + +``server:`` + In the following example, the actual settings are the default settings defined + in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and + the ``bind_address``: + + .. code-block:: yaml + + use_default_settings: True + server: + secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + bind_address: "0.0.0.0" + +``engines:`` + With ``use_default_settings: True``, each settings can be override in a + similar way, the ``engines`` section is merged according to the engine + ``name``. In this example, searx will load all the engine and the arch linux + wiki engine has a :ref:`token<private engines>`: + + .. code-block:: yaml + + use_default_settings: True + server: + secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + +``engines:`` / ``remove:`` + It is possible to remove some engines from the default settings. The following + example is similar to the above one, but searx doesn't load the the google + engine: + + .. code-block:: yaml + + use_default_settings: + engines: + remove: + - google + server: + secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + engines: + - name: arch linux wiki + tokens: ['$ecretValue'] + +``engines:`` / ``keep_only:`` + As an alternative, it is possible to specify the engines to keep. In the + following example, searx has only two engines: + + .. code-block:: yaml + + use_default_settings: + engines: + keep_only: + - google + - duckduckgo + server: + secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA" + engines: + - name: google + tokens: ['$ecretValue'] + - name: duckduckgo + tokens: ['$ecretValue'] diff --git a/docs/admin/update-searx.rst b/docs/admin/update-searx.rst new file mode 100644 index 0000000..71eb4d0 --- /dev/null +++ b/docs/admin/update-searx.rst @@ -0,0 +1,59 @@ +.. _update searx: + +============= +How to update +============= + +How to update depends on the :ref:`installation` method. If you have used the +:ref:`installation scripts`, use ``update`` command from the scripts. + +**Update** :ref:`searx service <searx.sh>` + +.. code:: sh + + sudo -H ./utils/searx.sh update searx + +**Update** :ref:`filtron reverse proxy <filtron.sh>` + +.. code:: sh + + sudo -H ./utils/filtron.sh update filtron + +**Update** :ref:`result proxy <morty.sh>` + +.. code:: bash + + sudo -H ./utils/morty.sh update morty + +.. _inspect searx: + +====================== +How to inspect & debug +====================== + +.. sidebar:: further read + + - :ref:`toolboxing` + - :ref:`Makefile` + +How to debug depends on the :ref:`installation` method. If you have used the +:ref:`installation scripts`, use ``inspect`` command from the scripts. + +**Inspect** :ref:`searx service <searx.sh>` + +.. code:: sh + + sudo -H ./utils/searx.sh inspect service + +**Inspect** :ref:`filtron reverse proxy <filtron.sh>` + +.. code:: sh + + sudo -H ./utils/filtron.sh inspect service + +**Inspect** :ref:`result proxy <morty.sh>` + +.. code:: bash + + sudo -H ./utils/morty.sh inspect service + diff --git a/docs/blog/command-line-engines.rst b/docs/blog/command-line-engines.rst new file mode 100644 index 0000000..746c9e4 --- /dev/null +++ b/docs/blog/command-line-engines.rst @@ -0,0 +1,65 @@ +======================================== +Running shell commands to fetch results +======================================== + +Previously, with searx you could search over the Internet on other people's +computers. Now it is possible to fetch results from your local machine without +connecting to any networks from the same graphical user interface. + + +Command line engines +==================== + +In :pull:`2128` a new type of engine has been introduced called ``command``. +This engine lets administrators add engines which run arbitrary shell commands +and show its output on the web UI of searx. + +When creating and enabling a ``command`` engine on a public searx instance, +you must be careful to avoid leaking private data. The easiest solution +is to add tokens to the engine. Thus, only those who have the appropriate token +can retrieve results from the it. + +The engine base is flexible. Only your imagination can limit the power of this engine. (And +maybe security concerns.) The following options are available: + +* ``command``: A comma separated list of the elements of the command. A special token {{QUERY}} tells searx where to put the search terms of the user. Example: ``['ls', '-l', '-h', '{{QUERY}}']`` +* ``delimiter``: A dict containing a delimiter char and the "titles" of each element in keys. +* ``parse_regex``: A dict containing the regular expressions for each result key. +* ``query_type``: The expected type of user search terms. Possible values: ``path`` and ``enum``. ``path`` checks if the uesr provided path is inside the working directory. If not the query is not executed. ``enum`` is a list of allowed search terms. If the user submits something which is not included in the list, the query returns an error. +* ``query_enum``: A list containing allowed search terms if ``query_type`` is set to ``enum``. +* ``working_dir``: The directory where the command has to be executed. Default: ``.`` +* ``result_separator``: The character that separates results. Default: ``\n`` + + +The example engine below can be used to find files with a specific name in the configured +working directory. + +.. code:: yaml + + - name: find + engine: command + command: ['find', '.', '-name', '{{QUERY}}'] + query_type: path + shortcut: fnd + delimiter: + chars: ' ' + keys: ['line'] + + +Next steps +========== + +In the next milestone, support for local search engines and indexers (e.g. Elasticsearch) +are going to be added. This way, you will be able to query your own databases/indexers. + +Acknowledgement +=============== + +This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . + +.. _Search and Discovery Fund: https://nlnet.nl/discovery +.. _NLnet Foundation: https://nlnet.nl/ + + +| Happy hacking. +| kvch // 2020.09.28 21:26 diff --git a/docs/blog/index.rst b/docs/blog/index.rst index 52fa3f1..857fede 100644 --- a/docs/blog/index.rst +++ b/docs/blog/index.rst @@ -3,8 +3,15 @@ Blog ==== .. toctree:: - :maxdepth: 1 + :titlesonly: + :reversed: python3 admin intro-offline + private-engines + lxcdev-202006 + command-line-engines + search-indexer-engines + sql-engines + search-database-engines diff --git a/docs/blog/lxcdev-202006.rst b/docs/blog/lxcdev-202006.rst new file mode 100644 index 0000000..b53501d --- /dev/null +++ b/docs/blog/lxcdev-202006.rst @@ -0,0 +1,413 @@ +.. _blog-lxcdev-202006: + +======================================= +Developing in Linux containers [202006] +======================================= + +.. _LXC: https://linuxcontainers.org/lxc/introduction/ + +.. sidebar:: Audience + + This blog post is written for experienced admins and developers / readers + should have a serious meaning about: *distributed*, *merge* and *linux + container*. + +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + +In PR :PR:`1803` we added a lot of scripts to Searx's boilerplate. In this blog +post I will show you, how you can make use of them in *distributed and +heterogeneous development cycles* (TL;DR; jump to the :ref:`blog-lxcdev-202006 +abstract`). + +Motivation +========== + +Normally in our development cycle, we edit the sources and run some test and/or +builds by using ``make`` before we commit. This cycle is simple and perfect but +might fail in some aspects we should not overlook. + + The environment in which we run all our development processes matters! + +The :ref:`makefile` and the :ref:`make install` encapsulate a lot for us, but they +do not have access to all prerequisites. For example, there may have +dependencies on packages that are installed on the developer's desktop, but +usually are not preinstalled on a server or client system. Another examples +are; settings have been made to the software on the developer's host that would +never be set on a *production* system. + +*Linux Containers* (LXC_) are isolate environments and not to mix up on +developer's all the prerequisites of all the projects he contribute to, is +always a good choice. + +The scripts from PR :PR:`1803` can divide in those to install and maintain +software: + +- :ref:`searx.sh` +- :ref:`filtron.sh` +- :ref:`morty.sh` + +and the script :ref:`lxc.sh`, with we can scale our installation, maintenance or +even development tasks over a stack of containers, what we call: *Searx's lxc +suite*. + +Gentlemen, start your engines! +============================== + +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _archlinux: https://www.archlinux.org/ + +Before you can start with containers, you need to install and initiate LXD_ +once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ snap install lxd + $ lxd init --auto + +And you need to clone from origin or if you have your own fork, clone from your +fork: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ cd ~/Downloads + $ git clone https://github.com/searx/searx.git + $ cd searx + +The :ref:`lxc-searx.env` consists of several images, see ``export +LXC_SUITE=(...`` near by :origin:`utils/lxc-searx.env#L19`. For this blog post +we exercise on a archlinux_ image. The container of this image is named +``searx-archlinux``. Lets build the container, but be sure that this container +does not already exists, so first lets remove possible old one: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh remove searx-archlinux + $ sudo -H ./utils/lxc.sh build searx-archlinux + +.. sidebar:: The ``searx-archlinux`` container + + is the base of all our exercises here. + +In this container we install all services :ref:`including searx, morty & filtron +<lxc.sh install suite>` in once: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To proxy HTTP from filtron and morty in the container to the outside of the +container, install nginx into the container. Once for the bot blocker filtron: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/filtron.sh nginx install + ... + INFO: got 429 from http://10.174.184.156/searx + +and once for the content sanitizer (content proxy morty): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/morty.sh nginx install + ... + INFO: got 200 from http://10.174.184.156/morty/ + +.. sidebar:: Fully functional searx suite + + From here on you have a fully functional searx suite running with bot blocker + (filtron) and Web content sanitizer (content proxy morty) needed for a + *privacy protecting* search engine. + +On your system, the IP of your ``searx-archlinux`` container differs from +http://10.174.184.156/searx, just open the URL reported in your installation +protocol in your WEB browser from the desktop to test the instance from outside +of the container. + +In such a searx suite admins can maintain and access the debug log of the +different services quite easy. + +.. _working in containers: + +In containers, work as usual +============================ + +Usually you open a root-bash using ``sudo -H bash``. In case of LXC containers +open the root-bash in the container using ``./utils/lxc.sh cmd +searx-archlinux``: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux bash + INFO: [searx-archlinux] bash + [root@searx-archlinux searx]# pwd + /share/searx + +The prompt ``[root@searx-archlinux ...]`` signals, that you are the root user in +the searx-container. To debug the running searx instance use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh inspect service + ... + use [CTRL-C] to stop monitoring the log + ... + +Back in the browser on your desktop open the service http://10.174.184.156/searx +and run your application tests while the debug log is shown in the terminal from +above. You can stop monitoring using ``CTRL-C``, this also disables the *"debug +option"* in searx's settings file and restarts the searx uwsgi application. To +debug services from filtron and morty analogous use: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/filtron.sh inspect service + $ ./utils/morty.sh inspect service + +Another point we have to notice is that each service (:ref:`searx <searx.sh>`, +:ref:`filtron <filtron.sh>` and :ref:`morty <morty.sh>`) runs under dedicated +system user account with the same name (compare :ref:`create searx user`). To +get a shell from theses accounts, simply call one of the scripts: + +.. tabs:: + + .. group-tab:: root@searx-archlinux + + .. code:: sh + + $ ./utils/searx.sh shell + $ ./utils/filtron.sh shell + $ ./utils/morty.sh shell + +To get in touch, open a shell from the service user (searx@searx-archlinux): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh shell + // exit with [CTRL-D] + (searx-pyenv) [searx@searx-archlinux ~]$ ... + +The prompt ``[searx@searx-archlinux]`` signals that you are logged in as system +user ``searx`` in the ``searx-archlinux`` container and the python *virtualenv* +``(searx-pyenv)`` environment is activated. + +.. tabs:: + + .. group-tab:: searx@searx-archlinux + + .. code:: sh + + (searx-pyenv) [searx@searx-archlinux ~]$ pwd + /usr/local/searx + + + +Wrap production into developer suite +==================================== + +In this section we will see how to change the *"Fully functional searx suite"* +from a LXC container (which is quite ready for production) into a developer +suite. For this, we have to keep an eye on the :ref:`installation basic`: + +- searx setup in: ``/etc/searx/settings.yml`` +- searx user's home: ``/usr/local/searx`` +- virtualenv in: ``/usr/local/searx/searx-pyenv`` +- searx software in: ``/usr/local/searx/searx-src`` + +The searx software is a clone of the ``git_url`` (see :ref:`settings global`) and +the working tree is checked out from the ``git_branch``. With the use of the +:ref:`searx.sh` the searx service was installed as :ref:`uWSGI application +<searx uwsgi>`. To maintain this service, we can use ``systemctl`` (compare +:ref:`service architectures on distributions <uwsgi configuration>`). + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl stop uwsgi@searx + +With the command above, we stopped the searx uWSGI-App in the archlinux +container. + +The uWSGI-App for the archlinux dsitros is configured in +:origin:`utils/templates/etc/uwsgi/apps-archlinux/searx.ini`, from where at +least you should attend the settings of ``uid``, ``chdir``, ``env`` and +``http``:: + + env = SEARX_SETTINGS_PATH=/etc/searx/settings.yml + http = 127.0.0.1:8888 + + chdir = /usr/local/searx/searx-src/searx + virtualenv = /usr/local/searx/searx-pyenv + pythonpath = /usr/local/searx/searx-src + +If you have read the :ref:`"Good to know section" <lxc.sh>` you remember, that +each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` handles relative path names **transparent**. To wrap the +searx installation into a developer one, we simple have to create a smylink to +the **transparent** reposetory from the desktop. Now lets replace the +repository at ``searx-src`` in the container with the working tree from outside +of the container: + +.. tabs:: + + .. group-tab:: container becomes a developer suite + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ln -s /share/searx/ /usr/local/searx/searx-src + +Now we can develop as usual in the working tree of our desktop system. Every +time the software was changed, you have to restart the searx service (in the +conatiner): + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + systemctl restart uwsgi@searx + + +Remember: :ref:`working in containers` .. here are just some examples from my +daily usage: + +.. tabs:: + + .. group-tab:: desktop + + To *inspect* the searx instance (already described above): + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh inspect service + + Run :ref:`makefile`, e.g. to test inside the container: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make test + + To install all prerequisites needed for a :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + ./utils/searx.sh install buildhost + + To build the docs on a buildhost :ref:`buildhosts`: + + .. code:: sh + + $ sudo -H ./utils/lxc.sh cmd searx-archlinux \ + make docs.html + +.. _blog-lxcdev-202006 abstract: + +Abstract +======== + +We build up a fully functional searx suite in a archlinux container: + +.. code:: sh + + $ sudo -H ./utils/lxc.sh install suite searx-archlinux + +To access HTTP from the desktop we installed nginx for the services inside the +conatiner: + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ ./utils/filtron.sh nginx install + $ ./utils/morty.sh nginx install + +To wrap the suite into a developer one, we created a symbolic link to the +repository which is shared **transparent** from the desktop's file system into +the container : + +.. tabs:: + + .. group-tab:: [root@searx-archlinux] + + .. code:: sh + + $ mv /usr/local/searx/searx-src /usr/local/searx/searx-src.old + $ ln -s /share/searx/ /usr/local/searx/searx-src + $ systemctl restart uwsgi@searx + +To get remarks from the suite of the archlinux container we can use: + +.. tabs:: + + .. group-tab:: desktop + + .. code:: sh + + $ sudo -H ./utils/lxc.sh show suite searx-archlinux + ... + [searx-archlinux] INFO: (eth0) filtron: http://10.174.184.156:4004/ http://10.174.184.156/searx + [searx-archlinux] INFO: (eth0) morty: http://10.174.184.156:3000/ + [searx-archlinux] INFO: (eth0) docs.live: http://10.174.184.156:8080/ + [searx-archlinux] INFO: (eth0) IPv6: http://[fd42:573b:e0b3:e97e:216:3eff:fea5:9b65] + ... + diff --git a/docs/blog/private-engines.rst b/docs/blog/private-engines.rst new file mode 100644 index 0000000..027cc3d --- /dev/null +++ b/docs/blog/private-engines.rst @@ -0,0 +1,65 @@ +================================== +Limit access to your searx engines +================================== + +Administrators might find themselves wanting to limit access to some of the +enabled engines on their instances. It might be because they do not want to +expose some private information through an offline engine. Or they +would rather share engines only with their trusted friends or colleagues. + +.. _private engines: + +Private engines +=============== + +To solve this issue private engines were introduced in :pull:`1823`. +A new option was added to engines named `tokens`. It expects a list +of strings. If the user making a request presents one of the tokens +of an engine, they can access information about the engine +and make search requests. + +Example configuration to restrict access to the Arch Linux Wiki engine: + +.. code:: yaml + + - name : arch linux wiki + engine : archlinux + shortcut : al + tokens : [ 'my-secret-token' ] + + +Unless a user has configured the right token, the engine is going +to be hidden from him/her. It is not going to be included in the +list of engines on the Preferences page and in the output of +`/config` REST API call. + +Tokens can be added to one's configuration on the Preferences page +under "Engine tokens". The input expects a comma separated list of +strings. + +The distribution of the tokens from the administrator to the users +is not carved in stone. As providing access to such engines +implies that the admin knows and trusts the user, we do not see +necessary to come up with a strict process. Instead, +we would like to add guidelines to the documentation of the feature. + +Next steps +========== + +Now that searx has support for both offline engines and private engines, +it is possible to add concrete engines which benefit from these features. +For example engines which search on the local host running the instance. +Be it searching your file system or querying a private database. Be creative +and come up with new solutions which fit your use case. + +Acknowledgement +=============== + +This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . + +.. _Search and Discovery Fund: https://nlnet.nl/discovery +.. _NLnet Foundation: https://nlnet.nl/ + + +| Happy hacking. +| kvch // 2020.02.28 22:26 diff --git a/docs/blog/python3.rst b/docs/blog/python3.rst index 5bb7f1c..1d2cfc8 100644 --- a/docs/blog/python3.rst +++ b/docs/blog/python3.rst @@ -54,7 +54,7 @@ If you found bugs Please open an issue on `GitHub`_. Make sure that you mention your Python version in your issue, so we can investigate it properly. -.. _GitHub: https://github.com/asciimoo/searx/issues +.. _GitHub: https://github.com/searx/searx/issues Acknowledgment ============== diff --git a/docs/blog/search-database-engines.rst b/docs/blog/search-database-engines.rst new file mode 100644 index 0000000..94f418e --- /dev/null +++ b/docs/blog/search-database-engines.rst @@ -0,0 +1,95 @@ +=============================== +Query more of your NoSQL stores +=============================== + +From now on, searx lets you to query your NoSQL data stores: + +* `Redis`_ +* `MongoDB`_ + +The reference configuration of the engines are included ``settings.yml`` just commented out, +as you have to set various options and install dependencies before using them. + +By default, the engines use ``key-value`` template for displaying results. +If you are not satisfied with the original result layout, +you can use your owm template by placing the template under +``searx/templates/{theme_name}/result_templates/{template_name}`` and setting +``result_template`` attribute to ``{template_name}``. + +Futhermore, if you do not want to expose these engines on a public instance, you can +still add them and limit the access by setting ``tokens`` as described in the `blog post about +private engines`_. + +Configuring searx to use the stores +=================================== + +NoSQL data stores are used for storing arbitrary data without first defining their +structure. + +Redis +----- + +Reqired package: ``redis`` + +Redis is a key value based data store usually stored in memory. + +Select a database to search in and set its index in the option ``db``. You can +either look for exact matches or use partial keywords to find what you are looking for +by configuring ``exact_match_only``. + +In this example you can search for exact matches in your first database: + +.. code:: yaml + + - name : mystore + engine : redis_server + exact_match_only : True + host : 127.0.0.1 + port : 6379 + password : secret-password + db : 0 + shortcut : rds + enable_http : True + + +MongoDB +------- + +Required package: ``pymongo`` + +MongoDB is a document based database program that handles JSON like data. + +In order to query MongoDB, you have to select a ``database`` and a ``collection``. Furthermore, +you have to select a ``key`` that is going to be searched. MongoDB also supports the option ``exact_match_only``, so configure it +as you wish. + +Above is an example configuration for using a MongoDB collection: + +.. code:: yaml + + - name : mymongo + engine : mongodb + shortcut : md + host : '127.0.0.1' + port : 27017 + database : personal + collection : income + key : month + enable_http: True + + +Acknowledgement +=============== + +This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . + +.. _Redis: https://redis.io/ +.. _MongoDB: https://mongodb.com/ +.. _blog post about private engines: private-engines.html#private-engines +.. _Search and Discovery Fund: https://nlnet.nl/discovery +.. _NLnet Foundation: https://nlnet.nl/ + + +| Happy hacking. +| kvch // 2021.07.13 23:16 + diff --git a/docs/blog/search-indexer-engines.rst b/docs/blog/search-indexer-engines.rst new file mode 100644 index 0000000..ca4dd3c --- /dev/null +++ b/docs/blog/search-indexer-engines.rst @@ -0,0 +1,114 @@ +=============================== +Query your local search engines +=============================== + +From now on, searx lets you to query your locally running search engines. The following +ones are supported now: + +* `Elasticsearch`_ +* `Meilisearch`_ +* `Solr`_ + +All of the engines above are added to ``settings.yml`` just commented out, as you have to +``base_url`` for all them. + +Please note that if you are not using HTTPS to access these engines, you have to enable +HTTP requests by setting ``enable_http`` to ``True``. + +Futhermore, if you do not want to expose these engines on a public instance, you can +still add them and limit the access by setting ``tokens`` as described in the `blog post about +private engines`_. + +Configuring searx for search engines +==================================== + +Each search engine is powerful, capable of full-text search. + +Elasticsearch +------------- + +Elasticsearch supports numerous ways to query the data it is storing. At the moment +the engine supports the most popular search methods: ``match``, ``simple_query_string``, ``term`` and ``terms``. + +If none of the methods fit your use case, you can select ``custom`` query type and provide the JSON payload +searx has to submit to Elasticsearch in ``custom_query_json``. + +The following is an example configuration for an Elasticsearch instance with authentication +configured to read from ``my-index`` index. + +.. code:: yaml + + - name : elasticsearch + shortcut : es + engine : elasticsearch + base_url : http://localhost:9200 + username : elastic + password : changeme + index : my-index + query_type : match + enable_http : True + + +Meilisearch +----------- + +This search engine is aimed at individuals and small companies. It is designed for +small-scale (less than 10 million documents) data collections. E.g. it is great for storing +web pages you have visited and searching in the contents later. + +The engine supports faceted search, so you can search in a subset of documents of the collection. +Futhermore, you can search in Meilisearch instances that require authentication by setting ``auth_token``. + +Here is a simple example to query a Meilisearch instance: + +.. code:: yaml + + - name : meilisearch + engine : meilisearch + shortcut: mes + base_url : http://localhost:7700 + index : my-index + enable_http: True + + +Solr +---- + +Solr is a popular search engine based on Lucene, just like Elasticsearch. +But instead of searching in indices, you can search in collections. + +This is an example configuration for searching in the collection ``my-collection`` and get +the results in ascending order. + +.. code:: yaml + + - name : solr + engine : solr + shortcut : slr + base_url : http://localhost:8983 + collection : my-collection + sort : asc + enable_http : True + + +Next steps +========== + +The next step is to add support for various SQL databases. + +Acknowledgement +=============== + +This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . + +.. _blog post about private engines: private-engines.html#private-engines +.. _Elasticsearch: https://www.elastic.co/elasticsearch/ +.. _Meilisearch: https://www.meilisearch.com/ +.. _Solr: https://solr.apache.org/ +.. _Search and Discovery Fund: https://nlnet.nl/discovery +.. _NLnet Foundation: https://nlnet.nl/ + + +| Happy hacking. +| kvch // 2021.04.07 23:16 + diff --git a/docs/blog/sql-engines.rst b/docs/blog/sql-engines.rst new file mode 100644 index 0000000..c40e3e7 --- /dev/null +++ b/docs/blog/sql-engines.rst @@ -0,0 +1,117 @@ +================= +Query SQL servers +================= + +Now you can query SQL servers using searx. The following ones are supported: + +* `PostgreSQL`_ +* `MySQL`_ +* `SQLite`_ + +All of the engines above are added to ``settings.yml`` just commented out, as you have to +set the required attributes for the engines, e.g. ``database``. By default, the engines use +``key-value`` template for displaying results. If you are not satisfied with the original result layout, +you can use your owm template by placing the template under +``searx/templates/{theme_name}/result_templates/{template_name}`` and setting +``result_template`` attribute to ``{template_name}``. + +As mentioned in previous blog posts, if you do not wish to expose these engines on a +public instance, you can still add them and limit the access by setting ``tokens`` +as described in the `blog post about private engines`_. + +Configure the engines +===================== + +The configuration of the new database engines are similar. You must put a valid +SELECT SQL query in ``query_str``. At the moment you can only bind at most +one parameter in your query. By setting the attribute ``limit`` you can +define how many results you want from the SQL server. Basically, it +is the same as the LIMIT keyword in SQL. + +Please, do not include LIMIT or OFFSET in your SQL query as the engines +rely on these keywords during paging. If you want to configure the number of returned results +use the option ``limit``. + +PostgreSQL +---------- + +PostgreSQL is a powerful and robust open source database. + +Before configuring the PostgreSQL engine, you must install the dependency ``psychopg2``. + +You can find an example configuration below: + +.. code:: yaml + + - name : postgresql + engine : postgresql + database : my_database + username : searx + password : password + query_str : 'SELECT * from my_table WHERE my_column = %(query)s' + shortcut : psql + + +MySQL +----- + +MySQL is said to be the most popular open source database. + +Before enabling MySQL engine, you must install the package ``mysql-connector-python``. + +The authentication plugin is configurable by setting ``auth_plugin`` in the attributes. +By default it is set to ``caching_sha2_password``. + +This is an example configuration for quering a MySQL server: + +.. code:: yaml + + - name : mysql + engine : mysql_server + database : my_database + username : searx + password : password + limit : 5 + query_str : 'SELECT * from my_table WHERE my_column=%(query)s' + shortcut : mysql + + +SQLite +------ + +SQLite is a small, fast and reliable SQL database engine. It does not require +any extra dependency. + +You can read from your database ``my_database`` using this example configuration: + +.. code:: yaml + + - name : sqlite + engine : sqlite + shortcut: sq + database : my_database + query_str : 'SELECT * FROM my_table WHERE my_column=:query' + + +Next steps +========== + +The next step is to add support for more data stores, e.g. Redis and MongoDB. + +Acknowledgement +=============== + +This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ . + +.. _PostgreSQL: https://www.postgresql.org/ +.. _MySQL: https://www.mysql.com/ +.. _SQLite: https://www.sqlite.org/index.html +.. _blog post about private engines: private-engines.html#private-engines +.. _Search and Discovery Fund: https://nlnet.nl/discovery +.. _NLnet Foundation: https://nlnet.nl/ + + +| Happy hacking. +| kvch // 2021.05.23 23:16 + + diff --git a/docs/build-templates/filtron.rst b/docs/build-templates/filtron.rst new file mode 100644 index 0000000..79b2543 --- /dev/null +++ b/docs/build-templates/filtron.rst @@ -0,0 +1,52 @@ +.. START create user + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: sh + + $ sudo -H useradd --shell /bin/bash --system \\ + --home-dir \"$SERVICE_HOME\" \\ + --comment \"Privacy-respecting metasearch engine\" $SERVICE_USER + + $ sudo -H mkdir \"$SERVICE_HOME\" + $ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\" + +.. END create user + +.. START install go + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: bash + + $ cat > \"$GO_ENV\" <<EOF + export GOPATH=${SERVICE_HOME}/go-apps + export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin + EOF + $ sudo -i -u \"${SERVICE_USER}\" + (${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile + (${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local + (${SERVICE_USER}) $ wget --progress=bar -O \"${GO_TAR}\" \\ + \"${GO_PKG_URL}\" + (${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_TAR}\" + (${SERVICE_USER}) $ which go + ${SERVICE_HOME}/local/go/bin/go + +.. END install go + +.. START install filtron + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: bash + + $ sudo -i -u \"${SERVICE_USER}\" + (${SERVICE_USER}) $ go get -v -u github.com/asciimoo/filtron + +.. END install filtron diff --git a/docs/build-templates/morty.rst b/docs/build-templates/morty.rst new file mode 100644 index 0000000..2be24da --- /dev/null +++ b/docs/build-templates/morty.rst @@ -0,0 +1,52 @@ +.. START create user + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: sh + + $ sudo -H useradd --shell /bin/bash --system \\ + --home-dir \"$SERVICE_HOME\" \\ + --comment \"Privacy-respecting metasearch engine\" $SERVICE_USER + + $ sudo -H mkdir \"$SERVICE_HOME\" + $ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\" + +.. END create user + +.. START install go + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: bash + + $ cat > \"$GO_ENV\" <<EOF + export GOPATH=${SERVICE_HOME}/go-apps + export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin + EOF + $ sudo -i -u \"${SERVICE_USER}\" + (${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile + (${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local + (${SERVICE_USER}) $ wget --progress=bar -O \"${GO_TAR}\" \\ + \"${GO_PKG_URL}\" + (${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_TAR}\" + (${SERVICE_USER}) $ which go + ${SERVICE_HOME}/local/go/bin/go + +.. END install go + +.. START install morty + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: bash + + $ sudo -i -u \"${SERVICE_USER}\" + (${SERVICE_USER}) $ go get -v -u github.com/asciimoo/morty + +.. END install morty diff --git a/docs/build-templates/searx.rst b/docs/build-templates/searx.rst new file mode 100644 index 0000000..e06bc2c --- /dev/null +++ b/docs/build-templates/searx.rst @@ -0,0 +1,210 @@ +.. template evaluated by: ./utils/searx.sh docs +.. hint: all dollar-names are variables, dollar sign itself is quoted by: \\$ + +.. START distro-packages + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo -H apt-get install -y \\ +${debian} + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo -H pacman -S --noconfirm \\ +${arch} + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo -H dnf install -y \\ +${fedora} + +.. END distro-packages + +.. START build-packages + +.. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo -H apt-get install -y \\ +${debian_build} + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo -H pacman -S --noconfirm \\ +${arch_build} + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo -H dnf install -y \\ +${fedora_build} + +.. END build-packages + +.. START create user + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: sh + + $ sudo -H useradd --shell /bin/bash --system \\ + --home-dir \"$SERVICE_HOME\" \\ + --comment 'Privacy-respecting metasearch engine' $SERVICE_USER + + $ sudo -H mkdir \"$SERVICE_HOME\" + $ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\" + +.. END create user + +.. START clone searx + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: sh + + $ sudo -H -u ${SERVICE_USER} -i + (${SERVICE_USER})$ git clone \"https://github.com/searx/searx.git\" \"$SEARX_SRC\" + +.. END clone searx + +.. START create virtualenv + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: sh + + (${SERVICE_USER})$ python3 -m venv \"${SEARX_PYENV}\" + (${SERVICE_USER})$ echo \". ${SEARX_PYENV}/bin/activate\" >> \"$SERVICE_HOME/.profile\" + +.. END create virtualenv + +.. START manage.sh update_packages + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: sh + + $ sudo -H -u ${SERVICE_USER} -i + + (${SERVICE_USER})$ command -v python && python --version + $SEARX_PYENV/bin/python + Python 3.8.1 + + # update pip's boilerplate .. + pip install -U pip + pip install -U setuptools + pip install -U wheel + pip install -U pyyaml + + # jump to searx's working tree and install searx into virtualenv + (${SERVICE_USER})$ cd \"$SEARX_SRC\" + (${SERVICE_USER})$ pip install -e . + + +.. END manage.sh update_packages + +.. START searx config + +.. tabs:: + + .. group-tab:: Use default settings + + .. code-block:: sh + + $ sudo -H mkdir -p \"$(dirname ${SEARX_SETTINGS_PATH})\" + $ sudo -H cp \"$SEARX_SRC/utils/templates/etc/searx/use_default_settings.yml\" \\ + \"${SEARX_SETTINGS_PATH}\" + + .. group-tab:: searx/settings.yml + + .. code-block:: sh + + $ sudo -H mkdir -p \"$(dirname ${SEARX_SETTINGS_PATH})\" + $ sudo -H cp \"$SEARX_SRC/searx/settings.yml\" \\ + \"${SEARX_SETTINGS_PATH}\" + +.. tabs:: + + .. group-tab:: minimal setup + + .. code-block:: sh + + $ sudo -H sed -i -e \"s/ultrasecretkey/\$(openssl rand -hex 16)/g\" \"$SEARX_SETTINGS_PATH\" + $ sudo -H sed -i -e \"s/{instance_name}/searx@\$(uname -n)/g\" \"$SEARX_SETTINGS_PATH\" + +.. END searx config + +.. START check searx installation + +.. tabs:: + + .. group-tab:: bash + + .. code-block:: sh + + # enable debug .. + $ sudo -H sed -i -e \"s/debug : False/debug : True/g\" \"$SEARX_SETTINGS_PATH\" + + # start webapp + $ sudo -H -u ${SERVICE_USER} -i + (${SERVICE_USER})$ cd ${SEARX_SRC} + (${SERVICE_USER})$ export SEARX_SETTINGS_PATH=\"${SEARX_SETTINGS_PATH}\" + (${SERVICE_USER})$ python searx/webapp.py + + # disable debug + $ sudo -H sed -i -e \"s/debug : True/debug : False/g\" \"$SEARX_SETTINGS_PATH\" + +Open WEB browser and visit http://$SEARX_INTERNAL_URL . If you are inside a +container or in a script, test with curl: + +.. tabs:: + + .. group-tab:: WEB browser + + .. code-block:: sh + + $ xdg-open http://$SEARX_INTERNAL_URL + + .. group-tab:: curl + + .. code-block:: none + + $ curl --location --verbose --head --insecure $SEARX_INTERNAL_URL + + * Trying 127.0.0.1:8888... + * TCP_NODELAY set + * Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0) + > HEAD / HTTP/1.1 + > Host: 127.0.0.1:8888 + > User-Agent: curl/7.68.0 + > Accept: */* + > + * Mark bundle as not supporting multiuse + * HTTP 1.0, assume close after body + < HTTP/1.0 200 OK + HTTP/1.0 200 OK + ... + +.. END check searx installation diff --git a/docs/conf.py b/docs/conf.py index af255e2..6efd762 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -1,17 +1,16 @@ # -*- coding: utf-8 -*- +# SPDX-License-Identifier: AGPL-3.0-or-later import sys, os -from searx.version import VERSION_STRING from pallets_sphinx_themes import ProjectLink -GIT_URL = os.environ.get("GIT_URL", "https://github.com/asciimoo/searx") -SEARX_URL = os.environ.get("SEARX_URL", "https://searx.me") -DOCS_URL = os.environ.get("DOCS_URL", "https://asciimoo.github.io/searx/") +from searx import brand +from searx.version import VERSION_STRING # Project -------------------------------------------------------------- project = u'searx' -copyright = u'2015-2019, Adam Tauber, Noémi Ványi' +copyright = u'2015-2021, Adam Tauber, Noémi Ványi' author = u'Adam Tauber' release, version = VERSION_STRING, VERSION_STRING highlight_language = 'none' @@ -22,35 +21,43 @@ master_doc = "index" source_suffix = '.rst' numfig = True -from searx import webapp +exclude_patterns = ['build-templates/*.rst'] + +import searx.search +import searx.engines +import searx.plugins +searx.search.initialize() jinja_contexts = { - 'webapp': dict(**webapp.__dict__) + 'searx': { + 'engines': searx.engines.engines, + 'plugins': searx.plugins.plugins + }, } # usage:: lorem :patch:`f373169` ipsum extlinks = {} # upstream links -extlinks['wiki'] = ('https://github.com/asciimoo/searx/wiki/%s', ' ') -extlinks['pull'] = ('https://github.com/asciimoo/searx/pull/%s', 'PR ') +extlinks['wiki'] = ('https://github.com/searx/searx/wiki/%s', '%s') +extlinks['pull'] = ('https://github.com/searx/searx/pull/%s', 'PR %s') # links to custom brand -extlinks['origin'] = (GIT_URL + '/blob/master/%s', 'git://') -extlinks['patch'] = (GIT_URL + '/commit/%s', '#') -extlinks['search'] = (SEARX_URL + '/%s', '#') -extlinks['docs'] = (DOCS_URL + '/%s', 'docs: ') -extlinks['pypi'] = ('https://pypi.org/project/%s', 'PyPi: ') -extlinks['man'] = ('https://manpages.debian.org/jump?q=%s', '') +extlinks['origin'] = (brand.GIT_URL + '/blob/' + brand.GIT_BRANCH + '/%s', 'Origin: %s') +extlinks['patch'] = (brand.GIT_URL + '/commit/%s', 'path %s') +extlinks['search'] = (brand.SEARX_URL + '/%s', 'URL: %s') +extlinks['docs'] = (brand.DOCS_URL + '/%s', 'docs: %s') +extlinks['pypi'] = ('https://pypi.org/project/%s', 'PyPi: %s') +extlinks['man'] = ('https://manpages.debian.org/jump?q=%s', 'man: %s') #extlinks['role'] = ( # 'https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-%s', '') extlinks['duref'] = ( - 'http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#%s', '') + 'https://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#%s', '%s') extlinks['durole'] = ( - 'http://docutils.sourceforge.net/docs/ref/rst/roles.html#%s', '') + 'https://docutils.sourceforge.net/docs/ref/rst/roles.html#%s', '%s') extlinks['dudir'] = ( - 'http://docutils.sourceforge.net/docs/ref/rst/directives.html#%s', '') + 'https://docutils.sourceforge.net/docs/ref/rst/directives.html#%s', '%s') extlinks['ctan'] = ( - 'https://ctan.org/pkg/%s', 'CTAN: ') + 'https://ctan.org/pkg/%s', 'CTAN: %s') extensions = [ 'sphinx.ext.imgmath', @@ -60,7 +67,9 @@ extensions = [ "sphinx.ext.intersphinx", "pallets_sphinx_themes", "sphinx_issues", # https://github.com/sloria/sphinx-issues/blob/master/README.rst - "sphinxcontrib.jinja", # https://github.com/tardyp/sphinx-jinja + "sphinx_jinja", # https://github.com/tardyp/sphinx-jinja + "sphinxcontrib.programoutput", # https://github.com/NextThought/sphinxcontrib-programoutput + 'linuxdoc.kernel_include', # Implementation of the 'kernel-include' reST-directive. 'linuxdoc.rstFlatTable', # Implementation of the 'flat-table' reST-directive. 'linuxdoc.kfigure', # Sphinx extension which implements scalable image handling. "sphinx_tabs.tabs", # https://github.com/djungelorm/sphinx-tabs @@ -75,11 +84,12 @@ intersphinx_mapping = { "sphinx" : ("https://www.sphinx-doc.org/en/master/", None), } -issues_github_path = "asciimoo/searx" +issues_github_path = "searx/searx" # HTML ----------------------------------------------------------------- sys.path.append(os.path.abspath('_themes')) +sys.path.insert(0, os.path.abspath("../utils/")) html_theme_path = ['_themes'] html_theme = "searx" @@ -90,14 +100,21 @@ imgmath_font_size = 14 # sphinx.ext.imgmath setup END html_theme_options = {"index_sidebar_logo": True} -html_context = { - "project_links": [ - ProjectLink("Source", GIT_URL), - ProjectLink("Wiki", "https://github.com/asciimoo/searx/wiki"), - ProjectLink("Public instances", "https://asciimoo.github.io/searx/user/public_instances.html"), - ProjectLink("Twitter", "https://twitter.com/Searx_engine"), - ] -} +html_context = {"project_links": [] } +html_context["project_links"].append(ProjectLink("Blog", "blog/index.html")) +if brand.GIT_URL: + html_context["project_links"].append(ProjectLink("Source", brand.GIT_URL)) +if brand.WIKI_URL: + html_context["project_links"].append(ProjectLink("Wiki", brand.WIKI_URL)) +if brand.PUBLIC_INSTANCES: + html_context["project_links"].append(ProjectLink("Public instances", brand.PUBLIC_INSTANCES)) +if brand.TWITTER_URL: + html_context["project_links"].append(ProjectLink("Twitter", brand.TWITTER_URL)) +if brand.ISSUE_URL: + html_context["project_links"].append(ProjectLink("Issue Tracker", brand.ISSUE_URL)) +if brand.CONTACT_URL: + html_context["project_links"].append(ProjectLink("Contact", brand.CONTACT_URL)) + html_sidebars = { "**": ["project.html", "relations.html", "searchbox.html"], } diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst index 459dfb4..ed1c223 100644 --- a/docs/dev/contribution_guide.rst +++ b/docs/dev/contribution_guide.rst @@ -4,6 +4,11 @@ How to contribute ================= +.. contents:: Contents + :depth: 2 + :local: + :backlinks: entry + Prime directives: Privacy, Hackability ====================================== @@ -82,7 +87,7 @@ In order to submit a patch, please follow the steps below: - Add yourself to the :origin:`AUTHORS.rst` file. -- Choose meaning full commit messages, read `Conventional Commits`_ +- Choose meaningful commit messages, read `Conventional Commits`_ .. code:: @@ -112,8 +117,8 @@ Translation currently takes place on :ref:`transifex <translation>`. Documentation ============= -.. _Sphinx: http://www.sphinx-doc.org -.. _reST: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html +.. _Sphinx: https://www.sphinx-doc.org +.. _reST: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html .. sidebar:: The reST sources @@ -127,54 +132,58 @@ Here is an example which makes a complete rebuild: .. code:: sh - $ make docs-clean docs + $ make docs.clean docs.html ... The HTML pages are in dist/docs. -.. _make docs-live: +.. _make docs.live: live build ---------- -.. sidebar:: docs-clean +.. _sphinx-autobuild: + https://github.com/executablebooks/sphinx-autobuild/blob/master/README.md + +.. sidebar:: docs.clean It is recommended to assert a complete rebuild before deploying (use - ``docs-clean``). + ``docs.clean``). Live build is like WYSIWYG. If you want to edit the documentation, its -recommended to use. The Makefile target ``docs-live`` builds the docs, opens +recommended to use. The Makefile target ``docs.live`` builds the docs, opens URL in your favorite browser and rebuilds every time a reST file has been changed. .. code:: sh - $ make docs-live + $ make docs.live ... The HTML pages are in dist/docs. - ... Serving on http://0.0.0.0:8080 + ... Serving on http://0.0.0.0:8000 ... Start watching changes +Live builds are implemented by sphinx-autobuild_. Use environment +``$(SPHINXOPTS)`` to pass arguments to the sphinx-autobuild_ command. Except +option ``--host`` (which is always set to ``0.0.0.0``) you can pass any +argument. E.g to find and use a free port, use: + +.. code:: sh + + $ SPHINXOPTS="--port 0" make docs.live + ... + ... Serving on http://0.0.0.0:50593 + ... + .. _deploy on github.io: deploy on github.io ------------------- -To deploy documentation at :docs:`github.io <.>` use Makefile target -:ref:`make gh-pages`, which will builds the documentation, clones searx into a sub -folder ``gh-pages``, cleans it, copies the doc build into and runs all the -needed git add, commit and push: +To deploy documentation at :docs:`github.io <.>` use Makefile target :ref:`make +docs.gh-pages`, which builds the documentation and runs all the needed git add, +commit and push: .. code:: sh - $ make docs-clean gh-pages - ... - SPHINX docs --> file://<...>/dist/docs - The HTML pages are in dist/docs. - ... - Cloning into 'gh-pages' ... - ... - cd gh-pages; git checkout gh-pages >/dev/null - Switched to a new branch 'gh-pages' - ... - doc available at --> https://asciimoo.github.io/searx + $ make docs.clean docs.gh-pages diff --git a/docs/dev/engine_overview.rst b/docs/dev/engine_overview.rst index 449c837..bbd1413 100644 --- a/docs/dev/engine_overview.rst +++ b/docs/dev/engine_overview.rst @@ -18,6 +18,9 @@ engines. Adapters are stored under the folder :origin:`searx/engines`. :depth: 3 :backlinks: entry + +.. _general engine configuration: + general engine configuration ============================ @@ -34,30 +37,36 @@ settings. However, the standard way is the following: engine file ----------- -======================= =========== =========================================== +======================= =========== ======================================================== argument type information -======================= =========== =========================================== +======================= =========== ======================================================== categories list pages, in which the engine is working paging boolean support multible pages -language_support boolean support language choosing time_range_support boolean support search time range -offline boolean engine runs offline -======================= =========== =========================================== +engine_type str ``online`` by default, other possibles values are + ``offline``, ``online_dictionary``, ``online_currency`` +======================= =========== ======================================================== .. _engine settings: settings.yml ------------ -======================= =========== =========================================== +======================= =========== ============================================= argument type information -======================= =========== =========================================== +======================= =========== ============================================= name string name of search-engine engine string name of searx-engine (filename without ``.py``) +enable_http bool enable HTTP + (by default only HTTPS is enabled). shortcut string shortcut of search-engine timeout string specific timeout for search-engine -======================= =========== =========================================== +display_error_messages boolean display error messages on the web UI +proxies dict set proxies for a specific engine + (e.g. ``proxies : {http: socks5://proxy:port, + https: socks5://proxy:port}``) +======================= =========== ============================================= overrides @@ -89,7 +98,6 @@ example code # engine dependent config categories = ['general'] paging = True - language_support = True making a request @@ -104,21 +112,48 @@ passed arguments These arguments can be used to construct the search query. Furthermore, parameters with default value can be redefined for special purposes. +If the ``engine_type`` is ``online```: + +====================== ============== ======================================================================== +argument type default-value, information +====================== ============== ======================================================================== +url str ``''`` +method str ``'GET'`` +headers set ``{}`` +data set ``{}`` +cookies set ``{}`` +verify bool ``True`` +headers.User-Agent str a random User-Agent +category str current category, like ``'general'`` +safesearch int ``0``, between ``0`` and ``2`` (normal, moderate, strict) +time_range Optional[str] ``None``, can be ``day``, ``week``, ``month``, ``year`` +pageno int current pagenumber +language str specific language code like ``'en_US'``, or ``'all'`` if unspecified +====================== ============== ======================================================================== + + +If the ``engine_type`` is ``online_dictionary```, in addition to the ``online`` arguments: + ====================== ============ ======================================================================== argument type default-value, information ====================== ============ ======================================================================== -url string ``''`` -method string ``'GET'`` -headers set ``{}`` -data set ``{}`` -cookies set ``{}`` -verify boolean ``True`` -headers.User-Agent string a random User-Agent -category string current category, like ``'general'`` -started datetime current date-time -pageno int current pagenumber -language string specific language code like ``'en_US'``, or ``'all'`` if unspecified +from_lang str specific language code like ``'en_US'`` +to_lang str specific language code like ``'en_US'`` +query str the text query without the languages +====================== ============ ======================================================================== + +If the ``engine_type`` is ``online_currency```, in addition to the ``online`` arguments: + ====================== ============ ======================================================================== +argument type default-value, information +====================== ============ ======================================================================== +amount float the amount to convert +from str ISO 4217 code +to str ISO 4217 code +from_name str currency name +to_name str currency name +====================== ============ ======================================================================== + parsed arguments ---------------- @@ -127,16 +162,20 @@ The function ``def request(query, params):`` always returns the ``params`` variable. Inside searx, the following paramters can be used to specify a search request: -============ =========== ========================================================= -argument type information -============ =========== ========================================================= -url string requested url -method string HTTP request method -headers set HTTP header information -data set HTTP data information (parsed if ``method != 'GET'``) -cookies set HTTP cookies -verify boolean Performing SSL-Validity check -============ =========== ========================================================= +=================== =========== ========================================================================== +argument type information +=================== =========== ========================================================================== +url str requested url +method str HTTP request method +headers set HTTP header information +data set HTTP data information +cookies set HTTP cookies +verify bool Performing SSL-Validity check +follow_redirects bool Follow redirects +max_redirects int maximum redirects, hard limit +soft_max_redirects int maximum redirects, soft limit. Record an error but don't stop the engine +raise_for_httperror bool True by default: raise an exception if the HTTP code of response is >= 300 +=================== =========== ========================================================================== example code @@ -255,7 +294,7 @@ latitude latitude of result (in decimal format) longitude longitude of result (in decimal format) boundingbox boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) -geojson geojson of result (http://geojson.org) +geojson geojson of result (https://geojson.org/) osm.type type of osm-object (if OSM-Result) osm.id id of osm-object (if OSM-Result) address.name name of object diff --git a/docs/dev/index.rst b/docs/dev/index.rst index cb913a8..ba0a25a 100644 --- a/docs/dev/index.rst +++ b/docs/dev/index.rst @@ -3,7 +3,8 @@ Developer documentation ======================= .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Contents quickstart contribution_guide diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst index f595700..870b5d4 100644 --- a/docs/dev/makefile.rst +++ b/docs/dev/makefile.rst @@ -1,65 +1,33 @@ .. _makefile: -================ -Makefile Targets -================ +======== +Makefile +======== .. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction .. sidebar:: build environment - Before looking deeper at the targets, first read about :ref:`makefile setup` - and :ref:`make pyenv`. + Before looking deeper at the targets, first read about :ref:`make + install`. -With the aim to simplify development cycles, started with :pull:`1756` a -``Makefile`` based boilerplate was added. If you are not familiar with -Makefiles, we recommend to read gnu-make_ introduction. + To install system requirements follow :ref:`buildhosts`. + +All relevant build tasks are implemented in :origin:`manage.sh` and for CI or +IDE integration a small ``Makefile`` wrapper is available. If you are not +familiar with Makefiles, we recommend to read gnu-make_ introduction. The usage is simple, just type ``make {target-name}`` to *build* a target. -Calling the ``help`` target gives a first overview:: - - $ make help - test - run developer tests - docs - build documentation - docs-live - autobuild HTML documentation while editing - run - run developer instance - install - developer install (./local) - uninstall - uninstall (./local) - gh-pages - build docs & deploy on gh-pages branch - clean - drop builds and environments - ... +Calling the ``help`` target gives a first overview (``make help``): + +.. program-output:: bash -c "cd ..; make --no-print-directory help" .. contents:: Contents :depth: 2 :local: :backlinks: entry - -.. _makefile setup: - -Setup -===== - -.. _git stash: https://git-scm.com/docs/git-stash - -The main setup is done in the :origin:`Makefile`:: - - export GIT_URL=https://github.com/asciimoo/searx - export SEARX_URL=https://searx.me - export DOCS_URL=https://asciimoo.github.io/searx - -.. sidebar:: fork & upstream - - Commit changes in your (local) branch, fork or whatever, but do not push them - upstream / `git stash`_ is your friend. - -:GIT_URL: Changes this, to point to your searx fork. - -:SEARX_URL: Changes this, to point to your searx instance. - -:DOCS_URL: If you host your own (branded) documentation, change this URL. - -.. _make pyenv: +.. _make install: Python environment ================== @@ -68,31 +36,42 @@ Python environment ``source ./local/py3/bin/activate`` -With Makefile we do no longer need to build up the virualenv manually (as -described in the :ref:`devquickstart` guide). Jump into your git working tree -and release a ``make pyenv``: - -.. code:: sh +We do no longer need to build up the virtualenv manually. Jump into your git +working tree and release a ``make install`` to get a virtualenv with a +*developer install* of searx (:origin:`setup.py`). :: $ cd ~/searx-clone - $ make pyenv - PYENV usage: source ./local/py3/bin/activate + $ make install + PYENV [virtualenv] installing ./requirements*.txt into local/py3 ... + PYENV OK + PYENV [install] pip install -e 'searx[test]' + ... + Successfully installed argparse-1.4.0 searx + BUILDENV INFO:searx:load the default settings from ./searx/settings.yml + BUILDENV INFO:searx:Initialisation done + BUILDENV build utils/brand.env -With target ``pyenv`` a development environment (aka virtualenv) was build up in -``./local/py3/``. To make a *developer install* of searx (:origin:`setup.py`) -into this environment, use make target ``install``: - -.. code:: sh +If you release ``make install`` multiple times the installation will only +rebuild if the sha256 sum of the *requirement files* fails. With other words: +the check fails if you edit the requirements listed in +:origin:`requirements-dev.txt` and :origin:`requirements.txt`). :: $ make install - PYENV usage: source ./local/py3/bin/activate - PYENV using virtualenv from ./local/py3 - PYENV install . - -You have never to think about intermediate targets like ``pyenv`` or -``install``, the ``Makefile`` chains them as requisites. Just run your main -target. + PYENV OK + PYENV [virtualenv] requirements.sha256 failed + [virtualenv] - 6cea6eb6def9e14a18bf32f8a3e... ./requirements-dev.txt + [virtualenv] - 471efef6c73558e391c3adb35f4... ./requirements.txt + ... + PYENV [virtualenv] installing ./requirements*.txt into local/py3 + ... + PYENV OK + PYENV [install] pip install -e 'searx[test]' + ... + Successfully installed argparse-1.4.0 searx + BUILDENV INFO:searx:load the default settings from ./searx/settings.yml + BUILDENV INFO:searx:Initialisation done + BUILDENV build utils/brand.env .. sidebar:: drop environment @@ -100,10 +79,7 @@ target. <make clean>` first. If you think, something goes wrong with your ./local environment or you change -the :origin:`setup.py` file (or the requirements listed in -:origin:`requirements-dev.txt` and :origin:`requirements.txt`), you have to call -:ref:`make clean`. - +the :origin:`setup.py` file, you have to call :ref:`make clean`. .. _make run: @@ -113,88 +89,113 @@ the :origin:`setup.py` file (or the requirements listed in To get up a running a developer instance simply call ``make run``. This enables *debug* option in :origin:`searx/settings.yml`, starts a ``./searx/webapp.py`` instance, disables *debug* option again and opens the URL in your favorite WEB -browser (:man:`xdg-open`): +browser (:man:`xdg-open`):: -.. code:: sh - - $ make run - PYENV usage: source ./local/py3/bin/activate - PYENV install . - ./local/py3/bin/python ./searx/webapp.py - ... - INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit) - ... + $ make run + PYENV OK + SEARX_DEBUG=1 ./manage.sh pyenv.cmd python ./searx/webapp.py + ... + INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit) .. _make clean: ``make clean`` ============== -Drop all intermediate files, all builds, but keep sources untouched. Includes -target ``pyclean`` which drops ./local environment. Before calling ``make -clean`` stop all processes using :ref:`make pyenv`. - -.. code:: sh +Drop all intermediate files, all builds, but keep sources untouched. Before +calling ``make clean`` stop all processes using :ref:`make install`. :: $ make clean - CLEAN pyclean - CLEAN clean + CLEAN pyenv + PYENV [virtualenv] drop ./local/py3 + CLEAN docs -- ./build/docs ./dist/docs + CLEAN locally installed npm dependencies + CLEAN test stuff + CLEAN common files .. _make docs: -``make docs docs-live docs-clean`` -================================== +``make docs docs.autobuild docs.clean`` +======================================= -We describe the usage of the ``doc*`` targets in the :ref:`How to contribute / +We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute / Documentation <contrib docs>` section. If you want to edit the documentation -read our :ref:`make docs-live` section. If you are working in your own brand, -adjust your :ref:`Makefile setup <makefile setup>`. - +read our :ref:`make docs.live` section. If you are working in your own brand, +adjust your :ref:`settings global`. -.. _make gh-pages: +.. _make docs.gh-pages: -``make gh-pages`` -================= +``make docs.gh-pages`` +====================== -To deploy on github.io first adjust your :ref:`Makefile setup <makefile -setup>`. For any further read :ref:`deploy on github.io`. +To deploy on github.io first adjust your :ref:`settings global`. For any +further read :ref:`deploy on github.io`. .. _make test: ``make test`` ============= -Runs a series of tests: ``test.pep8``, ``test.unit``, ``test.robot`` and does -additional :ref:`pylint checks <make pylint>`. You can run tests selective, -e.g.: +Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit`` +and ``test.robot``. You can run tests selective, e.g.:: -.. code:: sh + $ make test.pep8 test.unit test.sh + TEST test.pep8 OK + ... + TEST test.unit OK + ... + TEST test.sh OK + +.. _make test.sh: - $ make test.pep8 test.unit - . ./local/py3/bin/activate; ./manage.sh pep8_check - [!] Running pep8 check - . ./local/py3/bin/activate; ./manage.sh unit_tests - [!] Running unit tests +``make test.sh`` +================ + +:ref:`sh lint` / if you have changed some bash scripting run this test before +commit. -.. _make pylint: +.. _make test.pylint: -``make pylint`` -=============== +``make test.pylint`` +==================== .. _Pylint: https://www.pylint.org/ -Before commiting its recommend to do some (more) linting. Pylint_ is known as -one of the best source-code, bug and quality checker for the Python programming -language. Pylint_ is not yet a quality gate within our searx project (like -:ref:`test.pep8 <make test>` it is), but Pylint_ can help to improve code -quality anyway. The pylint profile we use at searx project is found in -project's root folder :origin:`.pylintrc`. +Pylint_ is known as one of the best source-code, bug and quality checker for the +Python programming language. The pylint profile we use at searx project is +found in project's root folder :origin:`.pylintrc`. + +.. _make search.checker: -Code quality is a ongoing process. Don't try to fix all messages from Pylint, -run Pylint and check if your changed lines are bringing up new messages. If so, -fix it. By this, code quality gets incremental better and if there comes the -day, the linting is balanced out, we might decide to add Pylint as a quality -gate. +``search.checker.{engine name}`` +================================ + +To check all engines:: + + make search.checker + +To check a engine with whitespace in the name like *google news* replace space +by underline:: + + make search.checker.google_news + +To see HTTP requests and more use SEARX_DEBUG:: + + make SEARX_DEBUG=1 search.checker.google_news + +.. _3xx: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#3xx_redirection + +To filter out HTTP redirects (3xx_):: + + make SEARX_DEBUG=1 search.checker.google_news | grep -A1 "HTTP/1.1\" 3[0-9][0-9]" + ... + Engine google news Checking + https://news.google.com:443 "GET /search?q=life&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=life&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- + https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0 + https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None + -- ``make pybuild`` @@ -203,9 +204,7 @@ gate. .. _PyPi: https://pypi.org/ .. _twine: https://twine.readthedocs.io/en/latest/ -Build Python packages in ``./dist/py``. - -.. code:: sh +Build Python packages in ``./dist/py``:: $ make pybuild ... @@ -213,9 +212,11 @@ Build Python packages in ``./dist/py``. running sdist running egg_info ... - $ ls ./dist/py/ - searx-0.15.0-py3-none-any.whl searx-0.15.0.tar.gz + running bdist_wheel + + $ ls ./dist + searx-0.18.0-py3-none-any.whl searx-0.18.0.tar.gz -To upload packages to PyPi_, there is also a ``upload-pypi`` target. It needs -twine_ to be installed. Since you are not the owner of :pypi:`searx` you will -never need the latter. +To upload packages to PyPi_, there is also a ``pypi.upload`` target (to test use +``pypi.upload.test``). Since you are not the owner of :pypi:`searx` you will +never need to upload. diff --git a/docs/dev/plugins.rst b/docs/dev/plugins.rst index 2bf44f1..16262ea 100644 --- a/docs/dev/plugins.rst +++ b/docs/dev/plugins.rst @@ -30,6 +30,21 @@ Example plugin ctx['search'].suggestions.add('example') return True +External plugins +================ + +External plugins are standard python modules implementing all the requirements of the standard plugins. +Plugins can be enabled by adding them to :ref:`settings.yml`'s ``plugins`` section. +Example external plugin can be found `here <https://github.com/asciimoo/searx_external_plugin_example>`_. + +Register your plugin +==================== + +To enable your plugin register your plugin in +searx > plugin > __init__.py. +And at the bottom of the file add your plugin like. +``plugins.register(name_of_python_file)`` + Plugin entry points =================== diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst index e40772b..bb9f4d6 100644 --- a/docs/dev/quickstart.rst +++ b/docs/dev/quickstart.rst @@ -4,129 +4,52 @@ Development Quickstart ====================== -.. sidebar:: :ref:`makefile` +.. _npm: https://www.npmjs.com/ - For additional developer purpose there are :ref:`makefile`. - -This quickstart guide gets your environment set up with searx. Furthermore, it -gives a short introduction to the ``manage.sh`` script. - -How to setup your development environment -========================================= - -.. sidebar:: :ref:`make pyenv <make pyenv>` - - Alternatively use the :ref:`make pyenv`. - -First, clone the source code of searx to the desired folder. In this case the -source is cloned to ``~/myprojects/searx``. Then create and activate the -searx-ve virtualenv and install the required packages using ``manage.sh``. - -.. code:: sh - - cd ~/myprojects - git clone https://github.com/asciimoo/searx.git - cd searx - virtualenv searx-ve - . ./searx-ve/bin/activate - ./manage.sh update_dev_packages - - -How to run tests -================ - -.. sidebar:: :ref:`make test.unit <make test>` - - Alternatively use the ``test.pep8``, ``test.unit``, ``test.robot`` targets. - -Tests can be run using the ``manage.sh`` script. Following tests and checks are -available: - -- Unit tests -- Selenium tests -- PEP8 validation -- Unit test coverage check - -For example unit tests are run with the command below: - -.. code:: sh - - ./manage.sh unit_tests - -For further test options, please consult the help of the ``manage.sh`` script or -read :ref:`make test`. - - -How to compile styles and javascript -==================================== - -.. _less: http://lesscss.org/ -.. _NodeJS: https://nodejs.org - -How to build styles -------------------- - -Less_ is required to build the styles of searx. Less_ can be installed using -either NodeJS_ or Apt. +Searx loves developers, just clone and start hacking. All the rest is done for +you simply by using :ref:`make <makefile>`. .. code:: sh - sudo -H apt-get install nodejs - sudo -H npm install -g less + git clone https://github.com/searx/searx.git -OR +Here is how a minimal workflow looks like: -.. code:: sh +1. *start* hacking +2. *run* your code: :ref:`make run` +3. *test* your code: :ref:`make test` - sudo -H apt-get install node-less +If you think at some point something fails, go back to *start*. Otherwise, +choose a meaningful commit message and we are happy to receive your pull +request. To not end in *wild west* we have some directives, please pay attention +to our ":ref:`how to contribute`" guideline. -After satisfying the requirements styles can be build using ``manage.sh`` +If you implement themes, you will need to compile styles and JavaScript before +*run*. .. code:: sh - ./manage.sh styles - - -How to build the source of the oscar theme -========================================== - -.. _grunt: https://gruntjs.com/ + make themes -Grunt_ must be installed in order to build the javascript sources. It depends on -NodeJS, so first Node has to be installed. +Don't forget to install npm_ first. -.. code:: sh - - sudo -H apt-get install nodejs - sudo -H npm install -g grunt-cli +.. tabs:: -After installing grunt, the files can be built using the following command: - -.. code:: sh + .. group-tab:: Ubuntu / debian - ./manage.sh grunt_build + .. code:: sh + sudo -H apt-get install npm -Tips for debugging/development -============================== + .. group-tab:: Arch Linux -.. sidebar:: :ref:`make run` + .. code-block:: sh - Makefile target ``run`` already enables debug option for your developer - session / see :ref:`make run`. + sudo -H pacman -S npm -Turn on debug logging - Whether you are working on a new engine or trying to eliminate a bug, it is - always a good idea to turn on debug logging. When debug logging is enabled a - stack trace appears, instead of the cryptic ``Internal Server Error`` - message. It can be turned on by setting ``debug: False`` to ``debug: True`` in - :origin:`settings.yml <searx/settings.yml>`. + .. group-tab:: Fedora / RHEL -.. sidebar:: :ref:`make test` + .. code-block:: sh - Alternatively use the :ref:`make test` targets. + sudo -H dnf install npm -Run ``./manage.sh tests`` before creating a PR. - Failing build on Travis is common because of PEP8 checks. So a new commit - must be created containing these format fixes. This phase can be skipped if - ``./manage.sh tests`` is run locally before creating a PR. diff --git a/docs/dev/reST.rst b/docs/dev/reST.rst index 4dc1279..1817504 100644 --- a/docs/dev/reST.rst +++ b/docs/dev/reST.rst @@ -15,8 +15,8 @@ generated and deployed at :docs:`github.io <.>`. For build prerequisites read :ref:`docs build`. The source files of Searx's documentation are located at :origin:`docs`. Sphinx -assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs-live -<make docs-live>` to build HTML while editing. +assumes source files to be encoded in UTF-8 by defaul. Run :ref:`make docs.live +<make docs.live>` to build HTML while editing. .. sidebar:: Further reading @@ -285,7 +285,7 @@ content becomes smart. :rst:role:`pep` :pep:`8` ``:pep:`8``` sphinx.ext.extlinks_ -------------------------------------------------------------------------------------------------- - project's wiki article :wiki:`Searx-instances` ``:wiki:`Searx-instances``` + project's wiki article :wiki:`Offline-engines` ``:wiki:`Offline-engines``` to docs public URL :docs:`dev/reST.html` ``:docs:`dev/reST.html``` files & folders origin :origin:`docs/dev/reST.rst` ``:origin:`docs/dev/reST.rst``` pull request :pull:`1756` ``:pull:`1756``` @@ -319,14 +319,18 @@ To list all anchors of the inventory (e.g. ``python``) use: .. code:: sh $ python -m sphinx.ext.intersphinx https://docs.python.org/3/objects.inv + ... + $ python -m sphinx.ext.intersphinx https://searx.github.io/searx/objects.inv + ... Literal blocks ============== The simplest form of :duref:`literal-blocks` is a indented block introduced by two colons (``::``). For highlighting use :dudir:`highlight` or :ref:`reST -code` directive. To include literals from external files use directive -:dudir:`literalinclude`. +code` directive. To include literals from external files use +:rst:dir:`literalinclude` or :ref:`kernel-include <kernel-include-directive>` +directive (latter one expands environment variables in the path name). .. _reST literal: @@ -1272,28 +1276,33 @@ Templating .. sidebar:: Build environment - All *generic-doc* tasks are running in the :ref:`build environment <make - pyenv>`. + All *generic-doc* tasks are running in the :ref:`make install`. Templating is suitable for documentation which is created generic at the build -time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`build -environment <make pyenv>` (with searx modules installed). We use this e.g. to -build chapter: :ref:`engines generic`. Below the jinja directive from the +time. The sphinx-jinja_ extension evaluates jinja_ templates in the :ref:`make +install` (with searx modules installed). We use this e.g. to build chapter: +:ref:`engines generic`. Below the jinja directive from the :origin:`docs/admin/engines.rst` is shown: .. literalinclude:: ../admin/engines.rst :language: reST :start-after: .. _configured engines: -The context for the template is selected in the line ``.. jinja:: webapp``. In -sphinx's build configuration (:origin:`docs/conf.py`) the ``webapp`` context -points to the name space of the python module: ``webapp``. +The context for the template is selected in the line ``.. jinja:: searx``. In +sphinx's build configuration (:origin:`docs/conf.py`) the ``searx`` context +contains the ``engines`` and ``plugins``. .. code:: py - from searx import webapp + import searx.search + import searx.engines + import searx.plugins + searx.search.initialize() jinja_contexts = { - 'webapp': dict(**webapp.__dict__) + 'searx': { + 'engines': searx.engines.engines, + 'plugins': searx.plugins.plugins + }, } @@ -1312,9 +1321,8 @@ others are basic-tabs_ and code-tabs_. Below a *group-tab* example from .. literalinclude:: ../admin/buildhosts.rst :language: reST - :start-after: .. _system requirements: - :end-before: .. _system requirements END: - + :start-after: .. SNIP sh lint requirements + :end-before: .. SNAP sh lint requirements .. _math: @@ -1391,27 +1399,27 @@ The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac`` .. _readability: https://docs.python-guide.org/writing/style/ .. _Sphinx-Primer: - http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html + https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html .. _reST: https://docutils.sourceforge.io/rst.html .. _Sphinx Roles: https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html -.. _Sphinx: http://www.sphinx-doc.org -.. _`sphinx-doc FAQ`: http://www.sphinx-doc.org/en/stable/faq.html +.. _Sphinx: https://www.sphinx-doc.org +.. _`sphinx-doc FAQ`: https://www.sphinx-doc.org/en/stable/faq.html .. _Sphinx markup constructs: - http://www.sphinx-doc.org/en/stable/markup/index.html + https://www.sphinx-doc.org/en/stable/markup/index.html .. _`sphinx cross references`: - http://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations + https://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations .. _sphinx.ext.extlinks: https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html -.. _intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html -.. _sphinx config: http://www.sphinx-doc.org/en/stable/config.html -.. _Sphinx's autodoc: http://www.sphinx-doc.org/en/stable/ext/autodoc.html +.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html +.. _sphinx config: https://www.sphinx-doc.org/en/stable/config.html +.. _Sphinx's autodoc: https://www.sphinx-doc.org/en/stable/ext/autodoc.html .. _Sphinx's Python domain: - http://www.sphinx-doc.org/en/stable/domains.html#the-python-domain + https://www.sphinx-doc.org/en/stable/domains.html#the-python-domain .. _Sphinx's C domain: - http://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs + https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs .. _doctree: - http://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases + https://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases .. _docutils: http://docutils.sourceforge.net/docs/index.html .. _docutils FAQ: http://docutils.sourceforge.net/FAQ.html .. _linuxdoc: https://return42.github.io/linuxdoc @@ -1424,5 +1432,5 @@ The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac`` .. _ImageMagick: https://www.imagemagick.org .. _`Emacs Table Mode`: https://www.emacswiki.org/emacs/TableMode -.. _`Online Tables Generator`: http://www.tablesgenerator.com/text_tables +.. _`Online Tables Generator`: https://www.tablesgenerator.com/text_tables .. _`OASIS XML Exchange Table Model`: https://www.oasis-open.org/specs/tm9901.html diff --git a/docs/dev/search_api.rst b/docs/dev/search_api.rst index 922548f..68fee94 100644 --- a/docs/dev/search_api.rst +++ b/docs/dev/search_api.rst @@ -6,7 +6,7 @@ Search API The search supports both ``GET`` and ``POST``. -Furthermore, two enpoints ``/`` and ``/search`` are available for querying. +Furthermore, two endpoints ``/`` and ``/search`` are available for querying. ``GET /`` @@ -76,12 +76,12 @@ Parameters supports safe search in the preferences page of an instance. ``theme`` : default ``oscar`` - [ ``oscar``, ``simple``, ``legacy``, ``pix-art``, ``courgette`` ] + [ ``oscar``, ``simple`` ] Theme of instance. Please note, available themes depend on an instance. It is possible that an - instance administrator deleted, created or renamed themes on his/her instance. + instance administrator deleted, created or renamed themes on their instance. See the available options in the preferences page of the instance. ``oscar-style`` : default ``logicodev`` @@ -91,7 +91,7 @@ Parameters ``oscar``. Please note, available styles depend on an instance. It is possible that an - instance administrator deleted, created or renamed styles on his/her + instance administrator deleted, created or renamed styles on their instance. See the available options in the preferences page of the instance. ``enabled_plugins`` : optional diff --git a/docs/index.rst b/docs/index.rst index d9503fe..a406da1 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -2,7 +2,14 @@ Welcome to searx ================ -Search without being tracked. + *Search without being tracked.* + +Searx is a free internet metasearch engine which aggregates results from more +than 70 search services. Users are neither tracked nor profiled. Additionally, +searx can be used over Tor for online anonymity. + +Get started with searx by using one of the Searx-instances_. If you don't trust +anyone, you can set up your own, see :ref:`installation`. .. sidebar:: Features @@ -16,17 +23,15 @@ Search without being tracked. - Hosted by organizations, such as *La Quadrature du Net*, which promote digital rights -Searx is a free internet metasearch engine which aggregates results from more -than 70 search services. Users are neither tracked nor profiled. Additionally, -searx can be used over Tor for online anonymity. - -Get started with searx by using one of the :wiki:`Searx-instances`. If you -don't trust anyone, you can set up your own, see :ref:`installation`. - .. toctree:: :maxdepth: 2 + :caption: Contents user/index admin/index dev/index + searx_extra/index + utils/index blog/index + +.. _Searx-instances: https://searx.space diff --git a/docs/searx_extra/index.rst b/docs/searx_extra/index.rst new file mode 100644 index 0000000..64d0b90 --- /dev/null +++ b/docs/searx_extra/index.rst @@ -0,0 +1,14 @@ +.. _searx_extra: + +====================================================== +Tooling box ``searx_extra`` for developers and users +====================================================== + +In the folder :origin:`searx_extra/` we maintain some tools useful for +developers and users. + +.. toctree:: + :maxdepth: 2 + :caption: Contents + + standalone_searx.py diff --git a/docs/searx_extra/standalone_searx.py.rst b/docs/searx_extra/standalone_searx.py.rst new file mode 100644 index 0000000..ff4b533 --- /dev/null +++ b/docs/searx_extra/standalone_searx.py.rst @@ -0,0 +1,9 @@ + +.. _standalone_searx.py: + +=================================== +``searx_extra/standalone_searx.py`` +=================================== + +.. automodule:: searx_extra.standalone_searx + :members: diff --git a/docs/user/index.rst b/docs/user/index.rst index 329be3b..96d11bf 100644 --- a/docs/user/index.rst +++ b/docs/user/index.rst @@ -3,8 +3,8 @@ User documentation ================== .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + :caption: Contents - public_instances search_syntax own-instance diff --git a/docs/user/own-instance.rst b/docs/user/own-instance.rst index a2f7365..bb4c5d5 100644 --- a/docs/user/own-instance.rst +++ b/docs/user/own-instance.rst @@ -2,8 +2,10 @@ Why use a private instance? =========================== -"Is it worth to run my own instance?" is a common question among searx users. -Before answering this question, see what options a searx user has. + *"Is it worth to run my own instance?"* + +\.\. is a common question among searx users. Before answering this question, +see what options a searx user has. Public instances are open to everyone who has access to its URL. Usually, these are operated by unknown parties (from the users' point of view). Private @@ -42,9 +44,9 @@ hidden from visited result pages. What are the consequences of using public instances? ---------------------------------------------------- -If someone uses a public instance, he/she has to trust the administrator of that +If someone uses a public instance, they have to trust the administrator of that instance. This means that the user of the public instance does not know whether -his/her requests are logged, aggregated and sent or sold to a third party. +their requests are logged, aggregated and sent or sold to a third party. Also, public instances without proper protection are more vulnerable to abusing the search service, In this case the external service in exchange returns @@ -54,9 +56,9 @@ results. I see. What about private instances? ------------------------------------ -If users run their own instances, everything is in their control: the source -code, logging settings and private data. Unknown instance administrators do not -have to be trusted. +If users run their :ref:`own instances <installation>`, everything is in their +control: the source code, logging settings and private data. Unknown instance +administrators do not have to be trusted. Furthermore, as the default settings of their instance is editable, there is no need to use cookies to tailor searx to their needs. So preferences will not be diff --git a/docs/user/public_instances.rst b/docs/user/public_instances.rst deleted file mode 100644 index 9665c58..0000000 --- a/docs/user/public_instances.rst +++ /dev/null @@ -1,318 +0,0 @@ -.. _public instances: - -.. - links has been ported from markdown to reST by:: - - regexpr: \[([^\]]*)\]\(([^)]*)\) - substitution: `\1 <\2>`__ - - -====================== -Public Searx instances -====================== - -.. _mailing list: mailto:searx-instances@autistici.org -.. _subscription page: https://www.autistici.org/mailman/listinfo/searx-instances - - -Useful information -================== - -* Up-to-date health report available on https://stats.searx.xyz [1]_, for onion - (tor) services: https://stats.searx.xyz/tor.html - -* Searx instances `mailing list`_ & `subscription page`_. - -* Some of the Searx instances have a CAcert SSL certificate. You can install the - missing root cert `from here <http://www.cacert.org/index.php?id=3>`__. - -* To add your own Searx instance to this page send us your PR. A GitHub account - is required to send PR or add an issue. - -.. [1] Note that most of the instances with a A+ grade in CSP column in this - site are not fully functional - for example auto-completion may not work. - - -List of public Searx instances -============================== - -Meta-searx instances -==================== - -These are websites that source from other searx instances. These are useful if -you can't decide which Searx instance to use: - - -.. flat-table:: Meta-searx instances - :header-rows: 1 - :stub-columns: 0 - :widths: 2 1 2 4 4 - - * - clearnet host - - onion host - - issuer - - source selection method - - extra privacy features - - * - `Neocities <https://searx.neocities.org/>`__ - - n/a - - Comodo (`Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.neocities.org>`__) - - Redirects users directly to a random selection of any known running - server after entering query. Requires - Javascript. `Changelog <https://searx.neocities.org/changelog.html>`__. - - Excludes servers with user tracking and analytics or are proxied through - Cloudflare. - - * - `Searxes <https://searxes.danwin1210.me/>`__ @Danwin - - onion v3 `hidden service - <http://searxes.nmqnkngye4ct7bgss4bmv5ca3wpa55yugvxen5kz2bbq67lwy6ps54yd.onion/>`__ - - Let's Encrypt (`Verification - <https://www.ssllabs.com/ssltest/analyze.html?d=searxes.danwin1210.me>`__) - - sources data from a randomly selected running server that satisfies - admin's quality standards which is used for post-processing - - filters out privacy-hostile websites (like CloudFlare) and either marks - them as such or folds them below the high ranking results. - - -Alive and running -================= - -**BEFORE EDITING**: Please add your Searx instance by respecting the alphabetic order. - -.. note:: - - Public instances listed here may yield less accurate results as they have - much higher traffic and consequently have a higher chance of being blocked by - search providers such as Google, Qwant, Bing, Startpage, etc. Hosting your - own instance or using an instance that isn't listed here may give you a more - consistent search experience. - -* `ai.deafpray.wtf/searx <https://ai.deafpray.wtf/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=ai.deafpray.wtf/searx>`__ -* `bamboozle.it <https://bamboozle.it/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=bamboozle.it>`__ -* `bee.jaekr.dev <https://bee.jaekr.dev>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=bee.jaekr.dev>`__ -* `beezboo.com <https://beezboo.com/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=beezboo.com>`__ -* `burtrum.org/searx <https://burtrum.org/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=burtrum.org/searx>`__ -* `darmarit.cloud/searx <https://darmarit.cloud/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=darmarit.cloud/searx>`__ -* `dc.ax <https://dc.ax>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=dc.ax>`__ -* `dynabyte.ca <https://dynabyte.ca>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=dynabyte.ca>`__ -* `goso.ga <https://goso.ga/search>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=goso.ga>`__ -* `gruble.de <https://www.gruble.de/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=www.gruble.de>`__ -* `haku.ahmia.fi <https://haku.ahmia.fi/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=haku.ahmia.fi&latest>`__ -* `haku.lelux.fi <https://haku.lelux.fi/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=haku.lelux.fi>`__ -* `huyo.me <https://huyo.me/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=huyo.me>`__ -* `jsearch.pw <https://jsearch.pw>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=jsearch.pw>`__ -* `le-dahut.com/searx <https://le-dahut.com/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=le-dahut.com/searx>`__ -* `mijisou.com <https://mijisou.com/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=mijisou.com>`__ -* `null.media <https://null.media>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=null.media>`__ -* `openworlds.info <https://openworlds.info/>`__ - Issuer: Let's Encrypt -* `perfectpixel.de/searx/ <https://www.perfectpixel.de/searx/>`__ - Issuer: LetsEncrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=www.perfectpixel.de>`__ -* `ransack.i2p <http://ransack.i2p/>`__ - I2P eepsite, only accessible with `I2P <https://geti2p.net/>`__ (`base32 address <http://mqamk4cfykdvhw5kjez2gnvse56gmnqxn7vkvvbuor4k4j2lbbnq.b32.i2p>`__) -* `rapu.nz <https://rapu.nz/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=rapu.nz>`__ -* `roflcopter.fr <https://wtf.roflcopter.fr/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=wtf.roflcopter.fr>`__ -* `roteserver.de/searx <https://roteserver.de/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=roteserver.de>`__ -* `s.cmd.gg <https://s.cmd.gg>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=s.cmd.gg>`__ -* `search.activemail.de <https://search.activemail.de/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.activemail.de&latest>`__ -* `search.anonymize.com <https://search.anonymize.com/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.anonymize.com>`__ -* `search.azkware.net <https://search.azkware.net/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.azkware.net>`__ -* `search.biboumail.fr <https://search.biboumail.fr/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.biboumail.fr>`__ -* `search.blankenberg.eu <https://search.blankenberg.eu>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.blankenberg.eu>`__ -* `search.d4networks.com <https://search.d4networks.com/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.d4networks.com>`__ -* `search.datensturm.net <https://search.datensturm.net>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.datensturm.net>`__ -* `search.disroot.org <https://search.disroot.org/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.disroot.org>`__ -* `search.ethibox.fr <https://search.ethibox.fr>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.ethibox.fr>`__ -* `search.fossdaily.xyz <https://search.fossdaily.xyz>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.fossdaily.xyz>`__ -* `search.galaxy.cat <https://search.galaxy.cat>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.galaxy.cat>`__ -* `search.gibberfish.org <https://search.gibberfish.org/>`__ (as `Hidden Service <http://o2jdk5mdsijm2b7l.onion/>`__ or `Proxied through Tor <https://search.gibberfish.org/tor/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.gibberfish.org>`__ -* `search.koehn.com <https://search.koehn.com>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.koehn.com>`__ -* `search.lgbtq.cool <https://search.lgbtq.cool/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.lgbtq.cool>`__ -* `search.mdosch.de <https://search.mdosch.de/>`__ (as `Hidden Service <http://search.4bkxscubgtxwvhpe.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.mdosch.de>`__ -* `search.modalogi.com <https://search.modalogi.com/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.modalogi.com&latest>`__ -* `search.moravit.com <https://search.moravit.com>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.moravit.com>`__ -* `search.nebulacentre.net <https://search.nebulacentre.net>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.nebulacentre.net>`__ -* `search.paulla.asso.fr <https://search.paulla.asso.fr/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.paulla.asso.fr>`__ -* `search.pifferi.info <https://search.pifferi.info/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.pifferi.info&latest>`__ -* `search.poal.co <https://search.poal.co/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.poal.co>`__ -* `search.privacytools.io <https://search.privacytools.io/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.privacytools.io>`__ - Uses Matomo for user tracking and analytics -* `search.seds.nl <https://search.seds.nl/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.seds.nl&latest>`__ -* `search.snopyta.org <https://search.snopyta.org/>`__ (as `Hidden Service <http://juy4e6eicawzdrz7.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.snopyta.org>`__ -* `search.spaeth.me <https://search.spaeth.me/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.spaeth.me&latest>`__ -* `search.st8.at <https://search.st8.at/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.st8.at>`__ -* `search.stinpriza.org <https://search.stinpriza.org>`__ (as `Hidden Service <http://z5vawdol25vrmorm4yydmohsd4u6rdoj2sylvoi3e3nqvxkvpqul7bqd.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.stinpriza.org&hideResults=on>`__ -* `search.sudo-i.net <https://search.sudo-i.net/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.sudo-i.net>`__ -* `search.tolstoevsky.ml <https://search.tolstoevsky.ml>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.tolstoevsky.ml>`__ -* `searchsin.com/searx <https://searchsin.com/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searchsin.com/searx>`__ -* `searx.anongoth.pl <https://searx.anongoth.pl>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.anongoth.pl&latest>`__ -* `searx.be <https://searx.be>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.be>`__ -* `searx.ca <https://searx.ca/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.ca>`__ -* `searx.canox.net <https://searx.canox.net/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.canox.net>`__ -* `searx.cybt.de <https://searx.cybt.de/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.cybt.de>`__ -* `searx.de <https://www.searx.de/>`__ - Issuer: COMODO `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.de>`__ -* `searx.decatec.de <https://searx.decatec.de>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.decatec.de>`__ -* `searx.devol.it <https://searx.devol.it/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=sears.devol.it>`__ -* `searx.dnswarden.com <https://searx.dnswarden.com>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.dnswarden.com>`__ -* `searx.drakonix.net <https://searx.drakonix.net/>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.drakonix.net>`__ -* `searx.dresden.network <https://searx.dresden.network/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.dresden.network>`__ -* `searx.elukerio.org <https://searx.elukerio.org/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.elukerio.org/>`__ -* `searx.everdot.org <https://searx.everdot.org/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.everdot.org/>`__ - Crawls using YaCy -* `searx.foo.li <https://searx.foo.li>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.foo.li&hideResults=on>`__ -* `searx.fossencdi.org <https://searx.fossencdi.org>`__ (as `Hidden Service <http://searx.cwuzdtzlubq5uual.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.fossencdi.org>`__ -* `searx.fr32k.de <https://searx.fr32k.de/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.fr32k.de>`__ -* `searx.good.one.pl <https://searx.good.one.pl>`__ (as `Hidden Service <http://searxl7u2y6gvonm.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.good.one.pl>`__ -* `searx.gotrust.de <https://searx.gotrust.de/>`__ (as `Hidden Service <http://nxhhwbbxc4khvvlw.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.gotrust.de>`__ -* `searx.hardwired.link <https://searx.hardwired.link/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.hardwired.link>`__ -* `searx.hlfh.space <https://searx.hlfh.space>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.hlfh.space>`__ -* `searx.info <https://searx.info>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.info>`__ -* `searx.itunix.eu <https://searx.itunix.eu/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.itunix.eu>`__ -* `searx.john-at-me.net <https://searx.john-at-me.net/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.john-at-me.net>`__ -* `searx.kvch.me <https://searx.kvch.me>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.kvch.me>`__ -* `searx.laquadrature.net <https://searx.laquadrature.net>`__ (as `Hidden Service <http://searchb5a7tmimez.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.laquadrature.net>`__ -* `searx.lelux.fi <https://searx.lelux.fi/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=haku.lelux.fi>`__ -* `searx.lhorn.de <https://searx.lhorn.de/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.lhorn.de&latest>`__ -* `searx.li <https://searx.li/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.li>`__ -* `searx.libmail.eu <https://searx.libmail.eu/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.libmail.eu/>`__ -* `searx.linux.pizza <https://searx.linux.pizza>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.linux.pizza>`__ -* `searx.lynnesbian.space <https://searx.lynnesbian.space/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.lynnesbian.space>`__ -* `searx.mastodontech.de <https://searx.mastodontech.de/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.mastodontech.de>`__ -* `searx.me <https://searx.me>`__ (as `Hidden Service <http://ulrn6sryqaifefld.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.me>`__ -* `searx.mxchange.org <https://searx.mxchange.org/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.mxchange.org>`__ -* `searx.nakhan.net <https://searx.nakhan.net>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.nakhan.net>`__ -* `searx.nixnet.xyz <https://searx.nixnet.xyz>`__ (as `Hidden Service <http://searx.l4qlywnpwqsluw65ts7md3khrivpirse744un3x7mlskqauz5pyuzgqd.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.nixnet.xyz>`__ -* `searx.nnto.net <https://searx.nnto.net/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.nnto.net>`__ -* `searx.openhoofd.nl <https://searx.openhoofd.nl/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=openhoofd.nl>`__ -* `searx.openpandora.org <https://searx.openpandora.org/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.openpandora.org&latest>`__ -* `searx.operationtulip.com <https://searx.operationtulip.com/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.operationtulip.com>`__ -* `searx.orcadian.net <https://searx.orcadian.net/>`__ - Issuer: Comodo CA Limited `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.orcadian.net>`__ -* `searx.ouahpit.info <https://searx.ouahpiti.info/>`__ - Issuer: Let's Encrypt -* `searx.pofilo.fr <https://searx.pofilo.fr/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.pofilo.fr>`__ -* `searx.prvcy.eu <https://searx.prvcy.eu/>`__ (as `Hidden Service <http://hmfztxt3pfhevucl.onion/>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.prvcy.eu>`__ -* `searx.pwoss.org <https://searx.pwoss.org>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.pwoss.org>`__ -* `searx.ro <https://searx.ro/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.ro>`__ -* `searx.ru <https://searx.ru/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.ru>`__ -* `searx.solusar.de <https://searx.solusar.de/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.solusar.de>`__ -* `searx.targaryen.house <https://searx.targaryen.house/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.targaryen.house>`__ -* `searx.tuxcloud.net <https://searx.tuxcloud.net>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.tuxcloud.net>`__ -* `searx.tyil.nl <https://searx.tyil.nl>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.tyil.nl>`__ -* `searx.wegeeks.win <https://searx.wegeeks.win>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.wegeeks.win>`__ -* `searx.win <https://searx.win/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.win&latest>`__ -* `searx.xyz <https://searx.xyz/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.xyz&latest>`__ -* `searx.zareldyn.net <https://searx.zareldyn.net/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.zareldyn.net>`__ -* `searx.zdechov.net <https://searx.zdechov.net>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.zdechov.net>`__ -* `searxs.eu <https://www.searxs.eu>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=www.searxs.eu&hideResults=on>`__ -* `seeks.hsbp.org <https://seeks.hsbp.org/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=seeks.hsbp.org>`__ - `PGP signed fingerprints of cert <https://seeks.hsbp.org/cert>`__ -* `skyn3t.in/srx <https://skyn3t.in/srx/>`__ - Issuer: Let's Encrypt | onion `hidden service <http://skyn3tb3bas655mw.onion/srx/>`__ -* `spot.ecloud.global <https://spot.ecloud.global/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=spot.ecloud.global>`__ -* `srx.sx <https://srx.sx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=srx.sx>`__ -* `stemy.me/searx <https://stemy.me/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=stemy.me>`__ -* `suche.dasnetzundich.de <https://suche.dasnetzundich.de>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=suche.dasnetzundich.de>`__ -* `suche.elaon.de <https://suche.elaon.de>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=suche.elaon.de>`__ -* `suche.xyzco456vwisukfg.onion <http://suche.xyzco456vwisukfg.onion/>`__ -* `suche.uferwerk.org <https://suche.uferwerk.org>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=suche.uferwerk.org>`__ -* `timdor.noip.me/searx <https://timdor.noip.me/searx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=timdor.noip.me/searx>`__ -* `trovu.komun.org <https://trovu.komun.org>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=trovu.komun.org>`__ -* `unmonito.red <https://unmonito.red/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=unmonito.red>`__ -* `www.finden.tk <https://www.finden.tk/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=www.finden.tk>`__ -* `zoek.anchel.nl <https://zoek.anchel.nl/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=zoek.anchel.nl>`__ - - - -Running in exclusive private walled-gardens -=========================================== - -These instances run in walled-gardens that exclude some segment of the general -public (e.g. Tor users and users sharing IPs with many other users). Caution: -privacy is also compromised on these sites due to exposure of cleartext traffic -to a third party other than the website operator. - -* `intelme.com <https://intelme.com>`__ - Issuer: Cloudflare `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=intelme.com>`__ -* `search404.io <https://www.search404.io/>`__ - Issuer: Cloudflare `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search404.io>`__ -* `searx.com.au <https://searx.com.au/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.com.au>`__ -* `searx.lavatech.top <https://searx.lavatech.top/>`__ - Issuer: Cloudflare `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.lavatech.top>`__ -* `searchx.mobi <https://searchx.mobi/>`__ - Issuer: Cloudflare `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searchx.mobi>`__ -* `searx.org <https://searx.org/>`__ - Issuer: Cloudflare `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.org>`__ -* `searx.run <https://searx.run/>`__ - Issuer: Cloudflare `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.run>`__ -* `searx.world <https://searx.world>`__ - Issuer: Cloudflare `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.world>`__ - Adds Amazon affiliate links - - -Running with an incorrect SSL certificate -========================================= - -* `listi.me <https://listi.me/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=listi.me&latest>`__ -* `s.matejc.com <https://s.matejc.com/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=s.matejc.com>`__ -* `search.jollausers.de <https://search.jollausers.de>`__ - Incorrectly configured `SSL certificate <https://www.ssllabs.com/ssltest/analyze.html?d=search.jollausers.de>`__ -* `search.paviro.de <https://search.paviro.de>`__ - Issuer: LetsEncrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.paviro.de>`__ -* `searx.abenthung.it <https://searx.abenthung.it/>`__ - Issuer: Comodo CA Limited `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.abenthung.it>`__ -* `searx.coding4schoki.org <https://searx.coding4schoki.org/>`__ - Incorrectly configured `SSL Certificate <https://www.ssllabs.com/ssltest/analyze.html?d=searx.coding4schoki.org>`__ -* `searx.haxors.club <https://searx.haxors.club/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.haxors.club>`__ -* `searx.nulltime.net <https://searx.nulltime.net/>`__ (as `Hidden Service <http://searx7gwtu5rh6wr.onion>`__) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.nulltime.net>`__ -* `searx.ch <https://searx.ch/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.ch>`__ (cert clock problems) - - -Offline -======= - -* `a.searx.space <https://a.searx.space>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=a.searx.space>`__ (unstable, under construction). -* `anyonething.de <https://anyonething.de>`__ - (was found to have become a pastebin on or before 2019-03-01) Issuer: Comodo CA Limited (Warning: uses Cloudflare) `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=anyonething.de>`__ -* `h7jwxg5rakyfvikpi.onion <http://7jwxg5rakyfvikpi.onion/>`__ - available only as Tor Hidden Service (down on 2019-06-26) -* `hacktivis.me/searx <https://hacktivis.me/searx>`__ - (down) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=hacktivis.me/searx>`__ -* `icebal.com <https://icebal.com>`__ - (down) Issuer: Let's Encrypt -* `netrangler.host <https://netrangler.host>`__ - (down) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=netrangler.host>`__ -* `opengo.nl <https://www.opengo.nl>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=www.opengo.nl>`__ -* `p9e.de <https://p9e.de/>`__ - (down - timeout) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=p9e.de>`__ -* `rubri.co <https://rubri.co>`__ - (down) Issuer: Let's Encrypt -* `s.bacafe.xyz <https://s.bacafe.xyz/>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=s.bacafe.xyz&latest>`__ -* `search.alecpap.com <https://search.alecpap.com/>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.alecpap.com>`__ -* `search.blackit.de <https://search.blackit.de/>`__ - (down) Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.blackit.de>`__ -* `search.deblan.org <https://search.deblan.org/>`__ (down) - Issuer: COMODO via GANDI `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.deblan.org>`__ -* `search.homecomputing.fr <https://search.homecomputing.fr/>`__ - (down) Issuer: CAcert `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.homecomputing.fr>`__ -* `search.jpope.org <https://search.jpope.org>`__ - (down - timeout) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.jpope.org>`__ -* `search.kakise.xyz <https://search.kakise.xyz/>`__ - down -* `search.kosebamse.com <https://search.kosebamse.com>`__ - Issuer: LetsEncrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.kosebamse.com>`__ -* `search.kujiu.org <https://search.kujiu.org>`__ - (down) Issuer: Let's Encrypt -* `search.mailaender.coffee <https://search.mailaender.coffee/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.mailaender.coffee>`__ -* `search.matrix.ac <https://search.matrix.ac>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=matrix.ac>`__ -* `search.mypsc.ca <https://search.mypsc.ca/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.mypsc.ca>`__ -* `search.namedkitten.pw <https://search.namedkitten.pw>`__ - (SSL error) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.namedkitten.pw>`__ -* `search.opentunisia.org <https://search.opentunisia.org>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.opentunisia.org>`__ -* `search.r3d007.com <https://search.r3d007.com/>`__ - (down) Issuer: Let's Encrypt -* `search.static.lu <https://search.static.lu/>`__ - (down) Issuer: StartCom `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.static.lu>`__ -* `search.teej.xyz <https://search.teej.xyz>`__ - (down) Issuer: LetsEncrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.teej.xyz>`__ -* `search.wxzm.sx <https://search.wxzm.sx>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=search.wxzm.sx>`__ -* `searx.4ray.co <https://searx.4ray.co/>`__ - (no longer an instance, redirects to main page) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.4ray.co>`__ -* `searx.32bitflo.at <https://searx.32bitflo.at/>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.32bitflo.at>`__ -* `searx.ahh.si <https://searx.ahh.si/>`__ - (down) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.ahh.si>`__ -* `searx.angristan.xyz <https://searx.angristan.xyz/>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.angristan.xyz>`__ -* `searx.antirep.net <https://searx.antirep.net/>`__ - (return a 502 HTTP error) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.antirep.net>`__ -* `searx.aquilenet.fr <https://searx.aquilenet.fr/>`__ - (down - 429 HTTP error) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.aquilenet.fr>`__ -* `searx.at <https://searx.at/>`__ - (return "request exception" at every search) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.at>`__ -* `searx.cc <https://searx.cc/>`__ - (down on 2019-06-26) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.cc>`__ -* `searx.dk <https://searx.dk/>`__ - (down - 429 HTTP error) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.dk>`__ -* `searx.ehrmanns.ch <https://searx.ehrmanns.ch>`__ - (down) Issuer: Let's Encrypt -* `searx.glibre.net <https://searx.glibre.net>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.glibre.net>`__ -* `searx.infini.fr <https://searx.infini.fr>`__ - (return a page stating that the website is not installed) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.infini.fr>`__ -* `searx.jeanphilippemorvan.info <https://searx.jeanphilippemorvan.info/>`__ - (down) Issuer: StartCom `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.jeanphilippemorvan.info>`__ -* `searx.lhorn.de <https://searx.lhorn.de/>`__ - (redirect the Searx's github repository page) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.lhorn.de&latest>`__ (only reachable from european countries) -* `searx.lvweb.host <https://searx.lvweb.host>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.lvweb.host>`__ -* `searx.mrtino.eu <https://searx.mrtino.eu>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.mrtino.eu>`__ -* `searx.netzspielplatz.de <https://searx.netzspielplatz.de/>`__ - (error page about GDPR even when browsing it from USA and Asia) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.netzspielplatz.de>`__ -* `searx.new-admin.net <https://searx.new-admin.net>`__ - (down) Issuer: Let's Encrypt -* `searx.nogafa.org <https://searx.nogafa.org/>`__ - (broken CSS) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.nogafa.org>`__ -* `searx.potato.hu <https://searx.potato.hu>`__ - (not a searx instance) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.potato.hu>`__ -* `searx.rubbeldiekatz.info <https://searx.rubbeldiekatz.info/>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.rubbeldiekatz.info/>`__ -* `searx.s42.space <https://searx.s42.space>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.s42.space>`__ -* `searx.salcay.hu <https://searx.salcay.hu/>`__ - (down - blank page) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.salcay.hu>`__ -* `searx.selea.se <https://searx.selea.se>`__ - (Leads to default Apache page) Issuer: RapidSSL (HSTS preloaded, DNSSEC) `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.selea.se>`__ | `HSTS Preload <https://hstspreload.org/?domain=searx.selea.se>`__ -* `searx.steinscraft.net <https://searx.steinscraft.net/>`__ - (down) Issuer: Cloudflare -* `searx.techregion.de <https://searx.techregion.de/>`__ - (domain expired) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.techregion.de>`__ -* `searx.tognella.com <https://searx.tognella.com/>`__ - (down) Issuer: Cloudflare -* `searx.xi.ht <https://searx.xi.ht/>`__ - (return a 502 HTTP error) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searx.xi.ht>`__ -* `searxist.com <https://searxist.com/>`__ - (down) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=searxist.com>`__ -* `so.sb <https://so.sb/>`__ - (down) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=so.sb>`__ -* `srx.stdout.net <https://srx.stdout.net/>`__ - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=srx.stdout.net>`__ -* `w6f7cgdm54cyvohcuhraaafhajctyj3ihenrovuxogoagrr5g43qmoid.onion <http://w6f7cgdm54cyvohcuhraaafhajctyj3ihenrovuxogoagrr5g43qmoid.onion/>`__ - Hidden Service -* `win8linux.nohost.me <https://win8linux.nohost.me/searx/>`__ - (down) Issuer: Let's Encrypt -* `wiznet.tech <https://wiznet.tech>`__ - (down) - Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=wiznet.tech>`__ -* `www.mercurius.space <https://www.mercurius.space/>`__ - (down) Issuer: Let's Encrypt -* `www.ready.pm <https://www.ready.pm>`__ - Issuer: WoSign `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=www.ready.pm>`__ -* `z.awsmppl.com <https://z.awsmppl.com>`__ - (down) Issuer: Let's Encrypt `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=z.awsmppl.com>`__ -* `zlsdzh.tk <https://zlsdzh.tk>`__ - (down - 404 HTTP error) Issuer: TrustAsia Technologies, Inc. `Verification <https://www.ssllabs.com/ssltest/analyze.html?d=zlsdzh.tk>`__ * - diff --git a/docs/user/search_syntax.rst b/docs/user/search_syntax.rst index b738c72..57cb519 100644 --- a/docs/user/search_syntax.rst +++ b/docs/user/search_syntax.rst @@ -40,3 +40,27 @@ Image search: Custom language in wikipedia: - :search:`:hu !wp hackerspace <?q=%3Ahu%20%21wp%20hackerspace>` + +Multilingual Search +=================== + +Searx does not support true multilingual search. +You have to use the language prefix in your search query when searching in a different language. + +But there is a workaround: +By adding a new search engine with a different language, Searx will search in your default and other language. + +Example configuration in settings.yml for a German and English speaker: + .. code-block:: yaml + + search: + language : "de" + ... + + engines: + - name : google english + engine : google + language : english + ... + +When searching, the default google engine will return German results and "google english" will return English results. diff --git a/docs/utils/filtron.sh.rst b/docs/utils/filtron.sh.rst new file mode 100644 index 0000000..86e3fa8 --- /dev/null +++ b/docs/utils/filtron.sh.rst @@ -0,0 +1,80 @@ + +.. _filtron.sh: + +==================== +``utils/filtron.sh`` +==================== + +.. sidebar:: further reading + + - :ref:`searx filtron` + - :ref:`architecture` + - :ref:`installation` (:ref:`nginx <installation nginx>` & :ref:`apache + <installation apache>`) + +.. _Go: https://golang.org/ +.. _filtron: https://github.com/asciimoo/filtron +.. _filtron README: https://github.com/asciimoo/filtron/blob/master/README.md + +To simplify installation and maintenance of a filtron instance you can use the +script :origin:`utils/filtron.sh`. In most cases you will install filtron_ +simply by running the command: + +.. code:: bash + + sudo -H ./utils/filtron.sh install all + +The script adds a ``${SERVICE_USER}`` (default:``filtron``) and installs filtron_ +into this user account: + +#. Create a separated user account (``filtron``). +#. Download and install Go_ binary in user's $HOME (``~filtron``). +#. Install filtron with the package management from Go_ (``go get -v -u + github.com/asciimoo/filtron``) +#. Setup a proper rule configuration :origin:`[ref] + <utils/templates/etc/filtron/rules.json>` (``/etc/filtron/rules.json``). +#. Setup a systemd service unit :origin:`[ref] + <utils/templates/lib/systemd/system/filtron.service>` + (``/lib/systemd/system/filtron.service``). + + +Create user +=========== + +.. kernel-include:: $DOCS_BUILD/includes/filtron.rst + :start-after: START create user + :end-before: END create user + + +Install go +========== + +.. kernel-include:: $DOCS_BUILD/includes/filtron.rst + :start-after: START install go + :end-before: END install go + + +Install filtron +=============== + +Install :origin:`rules.json <utils/templates/etc/filtron/rules.json>` at +``/etc/filtron/rules.json`` (see :ref:`Sample configuration of filtron`) and +install filtron software and systemd unit: + +.. kernel-include:: $DOCS_BUILD/includes/filtron.rst + :start-after: START install filtron + :end-before: END install filtron + +.. kernel-include:: $DOCS_BUILD/includes/filtron.rst + :start-after: START install systemd unit + :end-before: END install systemd unit + +.. _filtron.sh overview: + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory +(:ref:`toolboxing common`): + +.. program-output:: ../utils/filtron.sh --help diff --git a/docs/utils/index.rst b/docs/utils/index.rst new file mode 100644 index 0000000..699641d --- /dev/null +++ b/docs/utils/index.rst @@ -0,0 +1,52 @@ +.. _searx_utils: +.. _toolboxing: + +=================== +Admin's tooling box +=================== + +In the folder :origin:`utils/` we maintain some tools useful for administrators. + +.. toctree:: + :maxdepth: 2 + :caption: Contents + + searx.sh + filtron.sh + morty.sh + lxc.sh + +.. _toolboxing common: + +Common commands & environment +============================= + +Scripts to maintain services often dispose of common commands and environments. + +``shell`` : command + Opens a shell from the service user ``${SERVICE_USSR}``, very helpful for + troubleshooting. + +``inspect service`` : command + Shows status and log of the service, most often you have a option to enable + more verbose debug logs. Very helpful for debugging, but be careful not to + enable debugging in a production environment! + +``FORCE_TIMEOUT`` : environment + Sets timeout for interactive prompts. If you want to run a script in batch + job, with defaults choices, set ``FORCE_TIMEOUT=0``. By example; to install a + reverse proxy for filtron on all containers of the :ref:`searx suite + <lxc-searx.env>` use :: + + sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/filtron.sh apache install + +.. _toolboxing setup: + +Tooling box setup +================= + +The main setup is done in the :origin:`.config.sh` (read also :ref:`settings +global`). + +.. literalinclude:: ../../.config.sh + :language: bash diff --git a/docs/utils/lxc.sh.rst b/docs/utils/lxc.sh.rst new file mode 100644 index 0000000..56bac26 --- /dev/null +++ b/docs/utils/lxc.sh.rst @@ -0,0 +1,149 @@ + +.. _snap: https://snapcraft.io +.. _snapcraft LXD: https://snapcraft.io/lxd +.. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/ +.. _LXC: https://linuxcontainers.org/lxc/introduction/ +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _`LXD@github`: https://github.com/lxc/lxd + +.. _archlinux: https://www.archlinux.org/ + +.. _lxc.sh: + +================ +``utils/lxc.sh`` +================ + +.. sidebar:: further reading + + - snap_, `snapcraft LXD`_ + - LXC_, LXD_ + - `LXC/LXD Image Server`_ + - `LXD@github`_ + +With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of +containers, what we call the: *lxc suite*. The *searx suite* +(:origin:`lxc-searx.env <utils/lxc-searx.env>`) is loaded by default, every time +you start the ``lxc.sh`` script (*you do not need to care about*). + +Before you can start with containers, you need to install and initiate LXD_ +once:: + + $ snap install lxd + $ lxd init --auto + +To make use of the containers from the *searx suite*, you have to build the +:ref:`LXC suite containers <lxc.sh help>` initial. But be warned, **this might +take some time**:: + + $ sudo -H ./utils/lxc.sh build + +A cup of coffee later, your LXC suite is build up and you can run whatever task +you want / in a selected or even in all :ref:`LXC suite containers <lxc.sh +help>`. If you do not want to build all containers, **you can build just +one**:: + + $ sudo -H ./utils/lxc.sh build searx-ubu1804 + +*Good to know ...* + +Each container shares the root folder of the repository and the command +``utils/lxc.sh cmd`` **handles relative path names transparent**, compare output +of:: + + $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile + ... + +In the containers, you can run what ever you want, e.g. to start a bash use:: + + $ sudo -H ./utils/lxc.sh cmd searx-ubu1804 bash + INFO: [searx-ubu1804] bash + root@searx-ubu1804:/share/searx# + +If there comes the time you want to **get rid off all** the containers and +**clean up local images** just type:: + + $ sudo -H ./utils/lxc.sh remove + $ sudo -H ./utils/lxc.sh remove images + +.. _lxc.sh install suite: + +Install suite +============= + +To install the complete :ref:`searx suite (includes searx, morty & filtron) +<lxc-searx.env>` into all LXC_ use:: + + $ sudo -H ./utils/lxc.sh install suite + +The command above installs a searx suite (see :ref:`installation scripts`). To +get the IP (URL) of the filtron service in the containers use ``show suite`` +command. To test instances from containers just open the URLs in your +WEB-Browser:: + + $ sudo ./utils/lxc.sh show suite | grep filtron + [searx-ubu1604] INFO: (eth0) filtron: http://n.n.n.246:4004/ http://n.n.n.246/searx + [searx-ubu1804] INFO: (eth0) filtron: http://n.n.n.147:4004/ http://n.n.n.147/searx + [searx-ubu1910] INFO: (eth0) filtron: http://n.n.n.140:4004/ http://n.n.n.140/searx + [searx-ubu2004] INFO: (eth0) filtron: http://n.n.n.18:4004/ http://n.n.n.18/searx + [searx-fedora31] INFO: (eth0) filtron: http://n.n.n.46:4004/ http://n.n.n.46/searx + [searx-archlinux] INFO: (eth0) filtron: http://n.n.n.32:4004/ http://n.n.n.32/searx + +To :ref:`install a nginx <installation nginx>` reverse proxy for filtron and +morty use (or alternatively use :ref:`apache <installation apache>`):: + + sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/filtron.sh nginx install + sudo -H ./utils/lxc.sh cmd -- FORCE_TIMEOUT=0 ./utils/morty.sh nginx install + + +Running commands +================ + +**Inside containers, you can use make or run scripts** from the +:ref:`toolboxing`. By example: to setup a :ref:`buildhosts` and run the +Makefile target ``test`` in the archlinux_ container:: + + sudo -H ./utils/lxc.sh cmd searx-archlinux ./utils/searx.sh install buildhost + sudo -H ./utils/lxc.sh cmd searx-archlinux make test + + +Setup searx buildhost +===================== + +You can **install the searx buildhost environment** into one or all containers. +The installation procedure to set up a :ref:`build host<buildhosts>` takes its +time. Installation in all containers will take more time (time for another cup +of coffee).:: + + sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh install buildhost + +To build (live) documentation inside a archlinux_ container:: + + sudo -H ./utils/lxc.sh cmd searx-archlinux make docs.clean docs.live + ... + [I 200331 15:00:42 server:296] Serving on http://0.0.0.0:8080 + +To get IP of the container and the port number *live docs* is listening:: + + $ sudo ./utils/lxc.sh show suite | grep docs.live + ... + [searx-archlinux] INFO: (eth0) docs.live: http://n.n.n.12:8080/ + + +.. _lxc.sh help: + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory: + +.. program-output:: ../utils/lxc.sh --help + + +.. _lxc-searx.env: + +searx suite +=========== + +.. literalinclude:: ../../utils/lxc-searx.env + :language: bash diff --git a/docs/utils/morty.sh.rst b/docs/utils/morty.sh.rst new file mode 100644 index 0000000..3bdf9bd --- /dev/null +++ b/docs/utils/morty.sh.rst @@ -0,0 +1,80 @@ + +.. _morty: https://github.com/asciimoo/morty +.. _morty's README: https://github.com/asciimoo/morty +.. _Go: https://golang.org/ + +.. _morty.sh: + +================== +``utils/morty.sh`` +================== + +.. sidebar:: further reading + + - :ref:`architecture` + - :ref:`installation` (:ref:`nginx <installation nginx>` & :ref:`apache + <installation apache>`) + - :ref:`searx morty` + +To simplify installation and maintenance of a morty_ instance you can use the +script :origin:`utils/morty.sh`. In most cases you will install morty_ simply by +running the command: + +.. code:: bash + + sudo -H ./utils/morty.sh install all + +The script adds a ``${SERVICE_USER}`` (default:``morty``) and installs morty_ +into this user account: + +#. Create a separated user account (``morty``). +#. Download and install Go_ binary in user's $HOME (``~morty``). +#. Install morty_ with the package management from Go_ (``go get -v -u + github.com/asciimoo/morty``) +#. Setup a systemd service unit :origin:`[ref] + <utils/templates/lib/systemd/system/morty.service>` + (``/lib/systemd/system/morty.service``). + +.. hint:: + + To add morty to your searx instance read chapter :ref:`searx morty`. + +Create user +=========== + +.. kernel-include:: $DOCS_BUILD/includes/morty.rst + :start-after: START create user + :end-before: END create user + + +Install go +========== + +.. kernel-include:: $DOCS_BUILD/includes/morty.rst + :start-after: START install go + :end-before: END install go + + +Install morty +============= + +Install morty software and systemd unit: + +.. kernel-include:: $DOCS_BUILD/includes/morty.rst + :start-after: START install morty + :end-before: END install morty + +.. kernel-include:: $DOCS_BUILD/includes/morty.rst + :start-after: START install systemd unit + :end-before: END install systemd unit + +.. _morty.sh overview: + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory +(:ref:`toolboxing common`): + +.. program-output:: ../utils/morty.sh --help + diff --git a/docs/utils/searx.sh.rst b/docs/utils/searx.sh.rst new file mode 100644 index 0000000..dd4442f --- /dev/null +++ b/docs/utils/searx.sh.rst @@ -0,0 +1,39 @@ + +.. _searx.sh: + +================== +``utils/searx.sh`` +================== + +.. sidebar:: further reading + + - :ref:`architecture` + - :ref:`installation` + - :ref:`installation nginx` + - :ref:`installation apache` + +To simplify installation and maintenance of a searx instance you can use the +script :origin:`utils/searx.sh`. + +Install +======= + +In most cases you will install searx simply by running the command: + +.. code:: bash + + sudo -H ./utils/searx.sh install all + +The script adds a ``${SERVICE_USER}`` (default:``searx``) and installs searx +into this user account. The installation is described in chapter +:ref:`installation basic`. + +.. _intranet reverse proxy: + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory +(:ref:`toolboxing common`): + +.. program-output:: ../utils/searx.sh --help |