README.md

   1 <div align="center"><img height="180" src="https://gitlab.iscpif.fr/gargantext/main/raw/master/images/logo.png"></div>
   2
   3 &nbsp;
   4 # Gargantext with Haskell (Backend instance)
   5
   6 ![Haskell](https://img.shields.io/badge/Code-Haskell-informational?style=flat&logo=haskell&color=6144b3)&nbsp;&nbsp;![Stack](https://img.shields.io/badge/Tools-Stack-informational?style=flat&logo=&color=6144b3)&nbsp;&nbsp;![GHC](https://img.shields.io/badge/Tools-GHC-informational?style=flat&logo=&color=2E677B)&nbsp;&nbsp;![Nix](https://img.shields.io/badge/Package%20manager-Nix-informational?style=flat&logo=debian&color=6586c8)&nbsp;&nbsp;![Docker](https://img.shields.io/badge/Tools-Docker-informational?style=flat&logo=docker&color=003f8c)
   7
   8 #### Table of Contents
   9 1. [About the project](#about)
  10 2. [Installation](#install)
  11 3. [Initialization](#init)
  12 4. [Launch & develop GarganText](#launch)
  13 5. [Uses cases](#use-cases)
  14 6. [GraphQL](#graphql)
  15 7. [PostgreSQL](#postgresql)
  16
  17 ## About the project <a name="about"></a>
  18
  19 GarganText is a collaborative web-decentralized-based macro-service
  20 platform for the exploration of unstructured texts. It combines tools
  21 from natural language processing, text-data-mining bricks, complex
  22 networks analysis algorithms and interactive data visualization tools
  23 to pave the way toward new kinds of interactions with your textual and
  24 digital corpora.
  25
  26 This software is free (as "Libre" in French) software, developed by the
  27 CNRS Complex Systems Institute of Paris Île-de-France (ISC-PIF) and its
  28 partners.
  29
  30 GarganText Project: this repo builds the backend for the frontend server built by
  31 [backend](https://gitlab.iscpif.fr/gargantext/haskell-gargantext).
  32
  33
  34 ## Installation <a name="install"></a>
  35
  36 Disclaimer: since this project is still in development, this document
  37 remains in progress. Please report and improve this documentation if you
  38 encounter any issues.
  39
  40 #### Prerequisite
  41
  42 Clone the project.
  43 ```shell
  44 git clone https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git
  45 cd haskell-gargantext
  46 ```
  47 #### 1. Install Stack
  48
  49 Install [Stack (or Haskell Tool Stack)](https://docs.haskellstack.org/en/stable/):
  50
  51 ```shell
  52 curl -sSL https://get.haskellstack.org/ | sh
  53 ```
  54
  55 Verify the installation is complete with
  56 ```shell
  57 stack --version
  58 Version 2.9.1
  59 ```
  60
  61 #### 2. Install Nix
  62
  63 Install [Nix](https://nixos.org/download.html):
  64
  65 ```shell
  66 $ sh <(curl -L https://nixos.org/nix/install) --daemon
  67 ```
  68
  69 Verify the installation is complete with
  70 ```shell
  71 $ nix-env --version
  72 nix-env (Nix) 2.11.0
  73 ```
  74
  75 > **NOTE INFO (upgrade/downgrade if needed)**
  76 > Gargantext works with Nix 2.11.0 (older version than current 2.13.2). To downgrade your Nix version:
  77 > `nix-channel --update; nix-env -iA nixpkgs.nixVersions.nix_2_11 nixpkgs.cacert; systemctl daemon-reload; systemctl restart nix-daemon`
  78 > Upgrading Nix: https://nixos.org/manual/nix/unstable/installation/upgrading.html
  79
  80
  81 #### 3. Build Core Code
  82
  83 NOTE: Default build (with optimizations) requires large amounts of RAM
  84 (16GB at least). To avoid heavy compilation times and swapping out your
  85 machine, it is recommended to `stack build` with the `--fast` flag,
  86 i.e.:
  87
  88 ``` sh
  89 stack --nix build --fast
  90 ```
  91
  92 If the build is finishing without error, you are ready to launch
  93 GarganText! See next step.
  94
  95 &nbsp;
  96
  97
  98 ## Initialization <a name="init"></a>
  99
 100 #### 1. Docker-compose will configure your database and some NLP bricks (such as CoreNLP):
 101
 102 ``` sh
 103 # If docker is not installed:
 104 # curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/docker/docker-install | sh
 105 cd devops/docker
 106 docker compose up
 107 ```
 108 Initialization schema should be loaded automatically (from `devops/postgres/schema.sql`).
 109
 110 #### 2. Then install:
 111 ``` sh
 112 stack --nix install
 113 ```
 114
 115 #### 3. Copy the configuration file:
 116 ``` sh
 117 cp gargantext.ini_toModify gargantext.ini
 118 ```
 119 > Do not worry, `.gitignore` avoids adding this file to the repository by mistake, then you can change the passwords in gargantext.ini safely.
 120
 121 #### 4. A user have to be created first as instance:
 122 ``` sh
 123 ~/.local/bin/gargantext-init "gargantext.ini"
 124 ```
 125 Now, `user1` is created with password `1resu`
 126
 127 #### 5. Clone FRONTEND repository:
 128
 129 From the Backend root folder (haskell-gargantext):
 130
 131 ```shell
 132 git clone ssh://git@gitlab.iscpif.fr:20022/gargantext/purescript-gargantext.git
 133 ```
 134 &nbsp;
 135
 136 ## Launch & develop GarganText <a name="launch"></a>
 137
 138 From the Backend root folder (haskell-gargantext):
 139
 140 ``` shell
 141 ./start
 142 ```
 143
 144 > The start script runs following commands:
 145 > `docker compose up` to run the Docker for postgresql from devops/docker folder
 146 > `stack --nix exec gargantext-server -- --ini gargantext.ini --run Prod` to run other services
 147
 148 For frontend development and compilation, see the [Frontend Readme.md](https://gitlab.iscpif.fr/gargantext/purescript-gargantext#dev)
 149
 150
 151 ## Use Cases <a name="use-cases"></a>
 152
 153 ### Multi-User with Graphical User Interface (Server Mode)
 154
 155 ``` sh
 156 ~/.local/bin/stack --docker exec gargantext-server -- --ini "gargantext.ini" --run Prod
 157 ```
 158
 159 Then you can log in with `user1` / `1resu`
 160
 161
 162 ### Command Line Mode tools
 163
 164 #### Simple cooccurrences computation and indexation from a list of Ngrams
 165
 166 ``` sh
 167 stack --docker exec gargantext-cli -- CorpusFromGarg.csv ListFromGarg.csv Ouput.json
 168 ```
 169
 170 ### Analyzing the ngrams table repo
 171
 172 We store the repository in directory `repos` in the [CBOR](https://cbor.io/)
 173 file format. To decode it to JSON and analyze, say, using
 174 [jq](https://shapeshed.com/jq-json/), use the following command:
 175
 176 ``` sh
 177 cat repos/repo.cbor.v5 | stack --nix exec gargantext-cbor2json | jq .
 178 ```
 179 ### Documentation
 180
 181 To build documentation, run:
 182
 183 ```sh
 184 stack --nix build --haddock --no-haddock-deps --fast
 185 ```
 186
 187 (in `.stack-work/dist/x86_64-linux-nix/Cabal-3.2.1.0/doc/html/gargantext`).
 188
 189 ## GraphQL <a name="graphql"></a>
 190
 191 Some introspection information.
 192
 193 Playground is located at http://localhost:8008/gql
 194
 195 ### List all GraphQL types in the Playground
 196
 197 ```
 198 {
 199   __schema {
 200     types {
 201       name
 202     }
 203   }
 204 }
 205 ```
 206
 207 ### List details about a type in GraphQL
 208
 209 ```
 210 {
 211   __type(name:"User") {
 212         fields {
 213         name
 214       description
 215       type {
 216         name
 217       }
 218         }
 219         }
 220 }
 221 ```
 222 ## PostgreSQL <a name="pgsql"></a>
 223
 224 ### Upgrading using Docker
 225
 226 https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/
 227
 228 To upgrade PostgreSQL in Docker containers, for example from 11.x to 14.x, simply run:
 229 ```sh
 230 docker exec -it <container-id> pg_dumpall -U gargantua > 11-db.dump
 231 ```
 232
 233 Then, shut down the container, replace `image` section in
 234 `devops/docker/docker-compose.yaml` with `postgres:14`. Also, it is a good practice to create a new volume, say `garg-pgdata14` and bind the new container to it. If you want to keep the same volume, remember about removing it like so:
 235 ```sh
 236 docker-compose rm postgres
 237 docker volume rm docker_garg-pgdata
 238 ```
 239
 240 Now, start the container and execute:
 241 ```sh
 242 # need to drop the empty DB first, since schema will be created when restoring the dump
 243 docker exec -i <new-container-id> dropdb -U gargantua gargandbV5
 244 # recreate the db, but empty with no schema
 245 docker exec -i <new-container-id> createdb -U gargantua gargandbV5
 246 # now we can restore the dump
 247 docker exec -i <new-container-id> psql -U gargantua -d gargandbV5 < 11-db.dump
 248 ```
 249
 250 ### Upgrading using
 251
 252 There is a solution using pgupgrade_cluster but you need to manage the
 253 clusters version 14 and 13. Hence here is a simple solution to upgrade.
 254
 255 First save your data:
 256 ```
 257 sudo su postgres
 258 pg_dumpall > gargandb.dump
 259 ```
 260
 261 Upgrade postgresql:
 262 ```
 263 sudo apt install postgresql-server-14 postgresql-client-14
 264 sudo apt remove --purge postgresql-13
 265 ```
 266 Restore your data:
 267 ```
 268 sudo su postgres
 269 psql < gargandb.dump
 270 ```
 271
 272 Maybe you need to restore the gargantua password
 273 ```
 274 ALTER ROLE gargantua PASSWORD 'yourPasswordIn_gargantext.ini'
 275 ```
 276 Maybe you need to change the port to 5433 for database connection in
 277 your gargantext.ini file.
 278
 279
 280
 281