README.md

   1 <div align="center"><img height="180" src="https://gitlab.iscpif.fr/gargantext/main/raw/master/images/logo.png"></div>
   2
   3 &nbsp;
   4 # Gargantext with Haskell (Backend instance)
   5
   6 ![Haskell](https://img.shields.io/badge/Code-Haskell-informational?style=flat&logo=haskell&color=6144b3)&nbsp;&nbsp;![Stack](https://img.shields.io/badge/Tools-Stack-informational?style=flat&logo=&color=6144b3)&nbsp;&nbsp;![GHC](https://img.shields.io/badge/Tools-GHC-informational?style=flat&logo=&color=2E677B)&nbsp;&nbsp;![Nix](https://img.shields.io/badge/Package%20manager-Nix-informational?style=flat&logo=debian&color=6586c8)&nbsp;&nbsp;![Docker](https://img.shields.io/badge/Tools-Docker-informational?style=flat&logo=docker&color=003f8c)
   7
   8 #### Table of Contents
   9 1. [About the project](#about)
  10 2. [Installation](#install)
  11 3. [Initialization](#init)
  12 4. [Launch & develop GarganText](#launch)
  13 5. [Uses cases](#use-cases)
  14 6. [GraphQL](#graphql)
  15 7. [PostgreSQL](#postgresql)
  16
  17 ## About the project <a name="about"></a>
  18
  19 GarganText is a collaborative web-decentralized-based macro-service
  20 platform for the exploration of unstructured texts. It combines tools
  21 from natural language processing, text-data-mining bricks, complex
  22 networks analysis algorithms and interactive data visualization tools
  23 to pave the way toward new kinds of interactions with your textual and
  24 digital corpora.
  25
  26 This software is free (as "Libre" in French) software, developed by the
  27 CNRS Complex Systems Institute of Paris Île-de-France (ISC-PIF) and its
  28 partners.
  29
  30 GarganText Project: this repo builds the backend for the frontend server built by
  31 [backend](https://gitlab.iscpif.fr/gargantext/haskell-gargantext).
  32
  33
  34 ## Installation <a name="install"></a>
  35
  36 Disclaimer: since this project is still in development, this document
  37 remains in progress. Please report and improve this documentation if you
  38 encounter any issues.
  39
  40 #### Prerequisite
  41
  42 Clone the project.
  43 ```shell
  44 git clone https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git
  45 cd haskell-gargantext
  46 ```
  47 #### 1. Installation
  48
  49 This project can be built with either Stack or Cabal. For historical
  50 reasons, we generate a `cabal.project` from the `package.yaml`, and
  51 we do not commit the former to the repo, to have a single "source of truth".
  52 However, it's always possible to generate a `cabal.project` thanks
  53 to [stack2cabal](https://hackage.haskell.org/package/stack2cabal).
  54
  55 #### Install Nix
  56
  57 Gargantext requires [nix](https://github.com/NixOS/nix) to provide
  58 system dependencies (for example, C libraries), but its use is limited
  59 to that. In order to install [Nix](https://nixos.org/download.html):
  60
  61 ```shell
  62 sh <(curl -L https://nixos.org/nix/install) --daemon
  63 ```
  64
  65 Verify the installation is complete with
  66 ```shell
  67 nix-env --version
  68 nix-env (Nix) 2.11.0
  69 ```
  70
  71 **Important:** Before building the project with either `stack` or
  72 `cabal` you need to be in the correct Nix shell, which will fetch
  73 all the required system dependencies. To do so, just type:
  74
  75 ```shell
  76 nix-shell
  77 ```
  78
  79 This will take a bit of time the first time.
  80
  81 #### Build with Stack
  82
  83 Install [Stack (or Haskell Tool Stack)](https://docs.haskellstack.org/en/stable/):
  84
  85 ```shell
  86 curl -sSL https://get.haskellstack.org/ | sh
  87 ```
  88
  89 Verify the installation is complete with
  90 ```shell
  91 stack --version
  92 Version 2.9.1
  93 ```
  94
  95 NOTE: Default build (with optimizations) requires large amounts of RAM
  96 (16GB at least). To avoid heavy compilation times and swapping out your
  97 machine, it is recommended to `stack build` with the `--fast` flag,
  98 i.e.:
  99
 100 ```shell
 101 stack build --fast
 102 ```
 103
 104 #### Build with Cabal
 105
 106 Once you have a valid version of `cabal`, building requires generating
 107 a valid `cabal.project`. This can be done by installing `stack2cabal`:
 108
 109 ```shell
 110 cabal v2-install stack2cabal-1.0.14
 111 ```
 112
 113 And finally:
 114
 115 ```shell
 116 stack2cabal --no-run-hpack -p '2023-06-25'
 117 cabal v2-build
 118 ```
 119
 120 #### Keeping the cabal.project updated with stack.yaml
 121
 122 Simply run:
 123
 124 ```shell
 125 ./bin/update-cabal-project
 126 ```
 127
 128 ## Initialization <a name="init"></a>
 129
 130 #### 1. Docker-compose will configure your database and some NLP bricks (such as CoreNLP):
 131
 132 ``` sh
 133 # If docker is not installed:
 134 # curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/docker/docker-install | sh
 135 cd devops/docker
 136 docker compose up
 137 ```
 138 Initialization schema should be loaded automatically (from `devops/postgres/schema.sql`).
 139
 140 #### 2. Then install:
 141 ``` sh
 142 stack install
 143 ```
 144
 145 #### 3. Copy the configuration file:
 146 ``` sh
 147 cp gargantext.ini_toModify gargantext.ini
 148 ```
 149 > Do not worry, `.gitignore` avoids adding this file to the repository by mistake, then you can change the passwords in gargantext.ini safely.
 150
 151 #### 4. A user have to be created first as instance:
 152 ``` sh
 153 ~/.local/bin/gargantext-init "gargantext.ini"
 154 ```
 155 Now, `user1` is created with password `1resu`
 156
 157 #### 5. Clone FRONTEND repository:
 158
 159 From the Backend root folder (haskell-gargantext):
 160
 161 ```shell
 162 git clone ssh://git@gitlab.iscpif.fr:20022/gargantext/purescript-gargantext.git
 163 ```
 164 &nbsp;
 165
 166 ## Launch & develop GarganText <a name="launch"></a>
 167
 168 From the Backend root folder (haskell-gargantext):
 169
 170 ``` shell
 171 ./start
 172 ```
 173
 174 > The start script runs following commands:
 175 > `docker compose up` to run the Docker for postgresql from devops/docker folder
 176 > `stack exec gargantext-server -- --ini gargantext.ini --run Prod` to run other services
 177
 178 For frontend development and compilation, see the [Frontend Readme.md](https://gitlab.iscpif.fr/gargantext/purescript-gargantext#dev)
 179
 180
 181 ## Use Cases <a name="use-cases"></a>
 182
 183 ### Multi-User with Graphical User Interface (Server Mode)
 184
 185 ``` sh
 186 ~/.local/bin/stack --docker exec gargantext-server -- --ini "gargantext.ini" --run Prod
 187 ```
 188
 189 Then you can log in with `user1` / `1resu`
 190
 191
 192 ### Command Line Mode tools
 193
 194 #### Simple cooccurrences computation and indexation from a list of Ngrams
 195
 196 ``` sh
 197 stack --docker exec gargantext-cli -- CorpusFromGarg.csv ListFromGarg.csv Ouput.json
 198 ```
 199
 200 ### Analyzing the ngrams table repo
 201
 202 We store the repository in directory `repos` in the [CBOR](https://cbor.io/)
 203 file format. To decode it to JSON and analyze, say, using
 204 [jq](https://shapeshed.com/jq-json/), use the following command:
 205
 206 ``` sh
 207 cat repos/repo.cbor.v5 | stack exec gargantext-cbor2json | jq .
 208 ```
 209 ### Documentation
 210
 211 To build documentation, run:
 212
 213 ```sh
 214 stack build --haddock --no-haddock-deps --fast
 215 ```
 216
 217 (in `.stack-work/dist/x86_64-linux-nix/Cabal-3.2.1.0/doc/html/gargantext`).
 218
 219 ## GraphQL <a name="graphql"></a>
 220
 221 Some introspection information.
 222
 223 Playground is located at http://localhost:8008/gql
 224
 225 ### List all GraphQL types in the Playground
 226
 227 ```
 228 {
 229   __schema {
 230     types {
 231       name
 232     }
 233   }
 234 }
 235 ```
 236
 237 ### List details about a type in GraphQL
 238
 239 ```
 240 {
 241   __type(name:"User") {
 242         fields {
 243         name
 244       description
 245       type {
 246         name
 247       }
 248         }
 249         }
 250 }
 251 ```
 252 ## PostgreSQL <a name="pgsql"></a>
 253
 254 ### Upgrading using Docker
 255
 256 https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/
 257
 258 To upgrade PostgreSQL in Docker containers, for example from 11.x to 14.x, simply run:
 259 ```sh
 260 docker exec -it <container-id> pg_dumpall -U gargantua > 11-db.dump
 261 ```
 262
 263 Then, shut down the container, replace `image` section in
 264 `devops/docker/docker-compose.yaml` with `postgres:14`. Also, it is a good practice to create a new volume, say `garg-pgdata14` and bind the new container to it. If you want to keep the same volume, remember about removing it like so:
 265 ```sh
 266 docker-compose rm postgres
 267 docker volume rm docker_garg-pgdata
 268 ```
 269
 270 Now, start the container and execute:
 271 ```sh
 272 # need to drop the empty DB first, since schema will be created when restoring the dump
 273 docker exec -i <new-container-id> dropdb -U gargantua gargandbV5
 274 # recreate the db, but empty with no schema
 275 docker exec -i <new-container-id> createdb -U gargantua gargandbV5
 276 # now we can restore the dump
 277 docker exec -i <new-container-id> psql -U gargantua -d gargandbV5 < 11-db.dump
 278 ```
 279
 280 ### Upgrading using
 281
 282 There is a solution using pgupgrade_cluster but you need to manage the
 283 clusters version 14 and 13. Hence here is a simple solution to upgrade.
 284
 285 First save your data:
 286 ```
 287 sudo su postgres
 288 pg_dumpall > gargandb.dump
 289 ```
 290
 291 Upgrade postgresql:
 292 ```
 293 sudo apt install postgresql-server-14 postgresql-client-14
 294 sudo apt remove --purge postgresql-13
 295 ```
 296 Restore your data:
 297 ```
 298 sudo su postgres
 299 psql < gargandb.dump
 300 ```
 301
 302 Maybe you need to restore the gargantua password
 303 ```
 304 ALTER ROLE gargantua PASSWORD 'yourPasswordIn_gargantext.ini'
 305 ```
 306 Maybe you need to change the port to 5433 for database connection in
 307 your gargantext.ini file.
 308
 309
 310
 311