README.md

   1 <div align="center"><img height="180" src="https://gitlab.iscpif.fr/gargantext/main/raw/master/images/logo.png"></div>
   2
   3 &nbsp;
   4 # Gargantext with Haskell (Backend instance)
   5
   6 ![Haskell](https://img.shields.io/badge/Code-Haskell-informational?style=flat&logo=haskell&color=6144b3)&nbsp;&nbsp;![Stack](https://img.shields.io/badge/Tools-Stack-informational?style=flat&logo=&color=6144b3)&nbsp;&nbsp;![GHC](https://img.shields.io/badge/Tools-GHC-informational?style=flat&logo=&color=2E677B)&nbsp;&nbsp;![Nix](https://img.shields.io/badge/Package%20manager-Nix-informational?style=flat&logo=debian&color=6586c8)&nbsp;&nbsp;![Docker](https://img.shields.io/badge/Tools-Docker-informational?style=flat&logo=docker&color=003f8c)
   7
   8 #### Table of Contents
   9 1. [About the project](#about)
  10 2. [Installation](#install)
  11 3. [Initialization](#init)
  12 4. [Launch & develop GarganText](#launch)
  13 5. [Uses cases](#use-cases)
  14 6. [GraphQL](#graphql)
  15 7. [PostgreSQL](#postgresql)
  16
  17 ## About the project <a name="about"></a>
  18
  19 GarganText is a collaborative web-decentralized-based macro-service
  20 platform for the exploration of unstructured texts. It combines tools
  21 from natural language processing, text-data-mining bricks, complex
  22 networks analysis algorithms and interactive data visualization tools
  23 to pave the way toward new kinds of interactions with your textual and
  24 digital corpora.
  25
  26 This software is free (as "Libre" in French) software, developed by the
  27 CNRS Complex Systems Institute of Paris Île-de-France (ISC-PIF) and its
  28 partners.
  29
  30 GarganText Project: this repo builds the backend for the frontend server built by
  31 [backend](https://gitlab.iscpif.fr/gargantext/haskell-gargantext).
  32
  33
  34 ## Installation <a name="install"></a>
  35
  36 Disclaimer: since this project is still in development, this document
  37 remains in progress. Please report and improve this documentation if you
  38 encounter any issues.
  39
  40 #### Prerequisite
  41
  42 Clone the project.
  43 ```shell
  44 git clone https://gitlab.iscpif.fr/gargantext/haskell-gargantext.git
  45 cd haskell-gargantext
  46 ```
  47 #### 1. Install Stack
  48
  49 Install [Stack (or Haskell Tool Stack)](https://docs.haskellstack.org/en/stable/):
  50
  51 ```shell
  52 curl -sSL https://get.haskellstack.org/ | sh
  53 ```
  54
  55 Verify the installation is complete with
  56 ```shell
  57 stack --version
  58 Version 2.9.1
  59 ```
  60
  61 #### 2. Install Nix
  62
  63 Install [Nix](https://nixos.org/download.html):
  64
  65 ```shell
  66 $ sh <(curl -L https://nixos.org/nix/install) --daemon
  67 ```
  68
  69 Verify the installation is complete with
  70 ```shell
  71 $ nix-env --version
  72 nix-env (Nix) 2.11.0
  73 ```
  74
  75 > **NOTE INFO (upgrade/downgrade if needed)**
  76 > Gargantext works with Nix 2.11.0 (older version than current 2.13.2). To downgrade your Nix version:
  77 >
  78 > `nix-channel --update; nix-env -iA nixpkgs.nixVersions.nix_2_11 nixpkgs.cacert; systemctl daemon-reload; systemctl restart nix-daemon`
  79 >
  80 > Upgrading Nix: https://nixos.org/manual/nix/unstable/installation/upgrading.html
  81 >
  82 > **Then, don't forget to exit Terminal and reload to take into account the version change**
  83
  84
  85 #### 3. Build Core Code
  86
  87 NOTE: Default build (with optimizations) requires large amounts of RAM
  88 (16GB at least). To avoid heavy compilation times and swapping out your
  89 machine, it is recommended to `stack build` with the `--fast` flag,
  90 i.e.:
  91
  92 ``` sh
  93 stack --nix build --fast
  94 ```
  95
  96 If the build is finishing without error, you are ready to launch
  97 GarganText! See next step.
  98
  99 &nbsp;
 100
 101
 102 ## Initialization <a name="init"></a>
 103
 104 #### 1. Docker-compose will configure your database and some NLP bricks (such as CoreNLP):
 105
 106 ``` sh
 107 # If docker is not installed:
 108 # curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/docker/docker-install | sh
 109 cd devops/docker
 110 docker compose up
 111 ```
 112 Initialization schema should be loaded automatically (from `devops/postgres/schema.sql`).
 113
 114 #### 2. Then install:
 115 ``` sh
 116 stack --nix install
 117 ```
 118
 119 #### 3. Copy the configuration file:
 120 ``` sh
 121 cp gargantext.ini_toModify gargantext.ini
 122 ```
 123 > Do not worry, `.gitignore` avoids adding this file to the repository by mistake, then you can change the passwords in gargantext.ini safely.
 124
 125 #### 4. A user have to be created first as instance:
 126 ``` sh
 127 ~/.local/bin/gargantext-init "gargantext.ini"
 128 ```
 129 Now, `user1` is created with password `1resu`
 130
 131 #### 5. Clone FRONTEND repository:
 132
 133 From the Backend root folder (haskell-gargantext):
 134
 135 ```shell
 136 git clone ssh://git@gitlab.iscpif.fr:20022/gargantext/purescript-gargantext.git
 137 ```
 138 &nbsp;
 139
 140 ## Launch & develop GarganText <a name="launch"></a>
 141
 142 From the Backend root folder (haskell-gargantext):
 143
 144 ``` shell
 145 ./start
 146 ```
 147
 148 > The start script runs following commands:
 149 > `docker compose up` to run the Docker for postgresql from devops/docker folder
 150 > `stack --nix exec gargantext-server -- --ini gargantext.ini --run Prod` to run other services
 151
 152 For frontend development and compilation, see the [Frontend Readme.md](https://gitlab.iscpif.fr/gargantext/purescript-gargantext#dev)
 153
 154
 155 ## Use Cases <a name="use-cases"></a>
 156
 157 ### Multi-User with Graphical User Interface (Server Mode)
 158
 159 ``` sh
 160 ~/.local/bin/stack --docker exec gargantext-server -- --ini "gargantext.ini" --run Prod
 161 ```
 162
 163 Then you can log in with `user1` / `1resu`
 164
 165
 166 ### Command Line Mode tools
 167
 168 #### Simple cooccurrences computation and indexation from a list of Ngrams
 169
 170 ``` sh
 171 stack --docker exec gargantext-cli -- CorpusFromGarg.csv ListFromGarg.csv Ouput.json
 172 ```
 173
 174 ### Analyzing the ngrams table repo
 175
 176 We store the repository in directory `repos` in the [CBOR](https://cbor.io/)
 177 file format. To decode it to JSON and analyze, say, using
 178 [jq](https://shapeshed.com/jq-json/), use the following command:
 179
 180 ``` sh
 181 cat repos/repo.cbor.v5 | stack --nix exec gargantext-cbor2json | jq .
 182 ```
 183 ### Documentation
 184
 185 To build documentation, run:
 186
 187 ```sh
 188 stack --nix build --haddock --no-haddock-deps --fast
 189 ```
 190
 191 (in `.stack-work/dist/x86_64-linux-nix/Cabal-3.2.1.0/doc/html/gargantext`).
 192
 193 ## GraphQL <a name="graphql"></a>
 194
 195 Some introspection information.
 196
 197 Playground is located at http://localhost:8008/gql
 198
 199 ### List all GraphQL types in the Playground
 200
 201 ```
 202 {
 203   __schema {
 204     types {
 205       name
 206     }
 207   }
 208 }
 209 ```
 210
 211 ### List details about a type in GraphQL
 212
 213 ```
 214 {
 215   __type(name:"User") {
 216         fields {
 217         name
 218       description
 219       type {
 220         name
 221       }
 222         }
 223         }
 224 }
 225 ```
 226 ## PostgreSQL <a name="pgsql"></a>
 227
 228 ### Upgrading using Docker
 229
 230 https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/
 231
 232 To upgrade PostgreSQL in Docker containers, for example from 11.x to 14.x, simply run:
 233 ```sh
 234 docker exec -it <container-id> pg_dumpall -U gargantua > 11-db.dump
 235 ```
 236
 237 Then, shut down the container, replace `image` section in
 238 `devops/docker/docker-compose.yaml` with `postgres:14`. Also, it is a good practice to create a new volume, say `garg-pgdata14` and bind the new container to it. If you want to keep the same volume, remember about removing it like so:
 239 ```sh
 240 docker-compose rm postgres
 241 docker volume rm docker_garg-pgdata
 242 ```
 243
 244 Now, start the container and execute:
 245 ```sh
 246 # need to drop the empty DB first, since schema will be created when restoring the dump
 247 docker exec -i <new-container-id> dropdb -U gargantua gargandbV5
 248 # recreate the db, but empty with no schema
 249 docker exec -i <new-container-id> createdb -U gargantua gargandbV5
 250 # now we can restore the dump
 251 docker exec -i <new-container-id> psql -U gargantua -d gargandbV5 < 11-db.dump
 252 ```
 253
 254 ### Upgrading using
 255
 256 There is a solution using pgupgrade_cluster but you need to manage the
 257 clusters version 14 and 13. Hence here is a simple solution to upgrade.
 258
 259 First save your data:
 260 ```
 261 sudo su postgres
 262 pg_dumpall > gargandb.dump
 263 ```
 264
 265 Upgrade postgresql:
 266 ```
 267 sudo apt install postgresql-server-14 postgresql-client-14
 268 sudo apt remove --purge postgresql-13
 269 ```
 270 Restore your data:
 271 ```
 272 sudo su postgres
 273 psql < gargandb.dump
 274 ```
 275
 276 Maybe you need to restore the gargantua password
 277 ```
 278 ALTER ROLE gargantua PASSWORD 'yourPasswordIn_gargantext.ini'
 279 ```
 280 Maybe you need to change the port to 5433 for database connection in
 281 your gargantext.ini file.
 282
 283
 284
 285