README.md

   1 <div align="center"><img height="180" src="https://gitlab.iscpif.fr/gargantext/main/images/logo.png"></div>
   2
   3 &nbsp;
   4 # Gargantext with Haskell (Backend instance)
   5
   6 ![Haskell](https://img.shields.io/badge/Code-Haskell-informational?style=flat&logo=haskell&color=6144b3)&nbsp;&nbsp;![Stack](https://img.shields.io/badge/Tools-Stack-informational?style=flat&logo=&color=6144b3)&nbsp;&nbsp;![GHC](https://img.shields.io/badge/Tools-GHC-informational?style=flat&logo=&color=2E677B)&nbsp;&nbsp;![Nix](https://img.shields.io/badge/Package%20manager-Nix-informational?style=flat&logo=debian&color=6586c8)&nbsp;&nbsp;![Docker](https://img.shields.io/badge/Tools-Docker-informational?style=flat&logo=docker&color=003f8c)
   7
   8 #### Table of Contents
   9 1. [About the project](#about)
  10 2. [Example2](#example2)
  11 3. [Third Example](#third-example)
  12 4. [Fourth Example](#fourth-examplehttpwwwfourthexamplecom)
  13
  14 ## About the project  <a name="about"></a>
  15
  16 GarganText is a collaborative web-decentralized-based macro-service
  17 platform for the exploration of unstructured texts. It combines tools
  18 from natural language processing, text-data-mining tricks, complex
  19 networks analysis algorithms and interactive data visualization tools to
  20 pave the way toward new kinds of interactions with your digital corpora.
  21
  22 This software is free software, developed and offered by the CNRS
  23 Complex Systems Institute of Paris Île-de-France (ISC-PIF) and its
  24 partners.
  25
  26 GarganText Project: this repo builds the
  27 backend for the frontend server built by
  28 [backend](https://gitlab.iscpif.fr/gargantext/haskell-gargantext).
  29
  30
  31 ## Installation
  32
  33 Disclaimer: this project is still in development, this is work in
  34 progress. Please report and improve this documentation if you encounter issues.
  35
  36 ### Prerequisite
  37
  38 Clone the project.
  39
  40 ### 1. Install Stack
  41
  42 You need to install [Stack (or Haskell Tool Stack)](https://docs.haskellstack.org/en/stable/) first:
  43
  44 ```shell
  45 curl -sSL https://get.haskellstack.org/ | sh
  46 ```
  47
  48 Verify the installation is complete with
  49 ```shell
  50 stack --version
  51 Version 2.9.1
  52 ```
  53
  54 ### 2. Install Nix
  55
  56 First install [Nix](https://nixos.org/download.html):
  57
  58 ```shell
  59 $ sh <(curl -L https://nixos.org/nix/install) --daemon
  60 ```
  61
  62 Verify the installation is complete
  63 ```shell
  64 $ nix-env --version
  65 nix-env (Nix) 2.12.0
  66 ```
  67
  68 > **NOTE INFO (upgrade/downgrade if needed)**
  69 >
  70 > Gargantext works with Nix 2.12.0 (older version than current 2.13.2). To downgrade your Nix version:
  71 >
  72 > `nix-channel --update; nix-env -iA nixpkgs.nixVersions.nix_2_12 nixpkgs.cacert; systemctl daemon-reload; systemctl restart nix-daemon`
  73 >
  74 > Upgrading Nix: https://nixos.org/manual/nix/unstable/installation/upgrading.html
  75
  76
  77 ### 3. Build Core Code
  78
  79 NOTE: Default build (with optimizations) requires large amounts of RAM
  80 (16GB at least). To avoid heavy compilation times and swapping out your
  81 machine, it is recommended to `stack build` with the `--fast-` flag,
  82 i.e.:
  83
  84 ``` sh
  85 stack --nix build --fast
  86 ```
  87 or
  88
  89 ``` sh
  90 stack --docker build --fast
  91 ```
  92
  93
  94
  95 &nbsp;
  96 &nbsp;
  97 &nbsp;
  98 &nbsp;
  99 &nbsp;
 100
 101
 102
 103 #### Docker
 104
 105 ``` sh
 106 curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/docker/docker-install | sh
 107 ```
 108
 109 #### Debian
 110
 111 ``` sh
 112 curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/debian/install | sh
 113 ```
 114
 115 #### Ubuntu
 116
 117 ``` sh
 118 curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/ubuntu/install | sh
 119 ```
 120
 121 ### Add dependencies
 122
 123 1. CoreNLP is needed (EN and FR); This dependency will not be needed soon.
 124
 125 ``` sh
 126 ./devops/install-corenlp
 127 ```
 128 Note: if necessary change permission (```sudo chmod 755 ./devops/install-corenlp```)
 129
 130 ### Launch Gargantext (It should have been initialized first)
 131 Run from haskell-gargantext/devops/docker$
 132
 133 ``` sh
 134 docker compose up
 135 ```
 136
 137 Then
 138 ``` sh
 139 stack --nix install
 140 ~/.local/bin/gargantext-init "gargantext.ini"
 141 ```
 142 Or for Docker env, first create the appropriate image:
 143
 144 ``` sh
 145 cd devops/docker
 146 docker build -t cgenie/stack-build:lts-18.12-garg .
 147 ```
 148
 149 then run:
 150
 151 ``` sh
 152 stack --docker exec gargantext-init -- gargantext.ini
 153 ```
 154
 155 ### Initialization
 156
 157 #### Docker
 158
 159 Run PostgreSQL first:
 160
 161 ``` sh
 162 cd devops/docker
 163 docker-compose up
 164 ```
 165
 166 Initialization schema should be loaded automatically (from `devops/postgres/schema.sql`).
 167
 168 #### Gargantext
 169
 170 ##### Fix the passwords
 171
 172 Change the passwords in gargantext.ini_toModify then move it:
 173
 174 ``` sh
 175 cp gargantext.ini_toModify gargantext.ini
 176 ```
 177 (`.gitignore` avoids adding this file to the repository by mistake)
 178
 179
 180 ##### Run Gargantext
 181
 182 Users have to be created first (`user1` is created as instance):
 183
 184 ``` sh
 185 stack --nix install
 186 ~/.local/bin/gargantext-init "gargantext.ini"
 187 ```
 188
 189 For Docker env, first create the appropriate image:
 190
 191 ``` sh
 192 cd devops/docker
 193 docker build -t cgenie/stack-build:lts-18.12-garg .
 194 ```
 195
 196 then run:
 197
 198 ``` sh
 199 stack --docker exec gargantext-init -- gargantext.ini
 200 ```
 201
 202 ### Importing data
 203
 204 You can import some data with:
 205 ``` sh
 206 docker run --rm -it -p 9000:9000 cgenie/corenlp-garg
 207 stack exec gargantext-import -- "corpusCsvHal" "user1" "IMT3" gargantext.ini 10000 ./1000.csv
 208 ```
 209
 210 ### Nix
 211
 212 It is also possible to build everything with [Nix](https://nixos.org/) instead of Docker:
 213 ``` sh
 214 stack --nix build
 215 stack --nix exec gargantext-import -- "corpusCsvHal" "user1" "IMT3" gargantext.ini 10000 ./1000.csv
 216 stack --nix exec gargantext-server -- --ini gargantext.ini --run Prod
 217 ```
 218
 219 ## Use Cases
 220
 221 ### Multi-User with Graphical User Interface (Server Mode)
 222
 223 ``` sh
 224 ~/.local/bin/stack --docker exec gargantext-server -- --ini "gargantext.ini" --run Prod
 225 ```
 226
 227 Then you can log in with `user1` / `1resu`.
 228
 229
 230 ### Command Line Mode tools
 231
 232 #### Simple cooccurrences computation and indexation from a list of Ngrams
 233
 234 ``` sh
 235 stack --docker exec gargantext-cli -- CorpusFromGarg.csv ListFromGarg.csv Ouput.json
 236 ```
 237
 238 ### Analyzing the ngrams table repo
 239
 240 We store the repository in directory `repos` in the [CBOR](https://cbor.io/)
 241 file format. To decode it to JSON and analyze, say, using
 242 [jq](https://shapeshed.com/jq-json/), use the following command:
 243
 244 ``` sh
 245 cat repos/repo.cbor.v5 | stack --nix exec gargantext-cbor2json | jq .
 246 ```
 247 ### Documentation
 248
 249 To build documentation, run:
 250
 251 ```sh
 252 stack --docker build --haddock --no-haddock-deps --fast
 253 ```
 254
 255 (in `.stack-work/dist/x86_64-linux-nix/Cabal-3.2.1.0/doc/html/gargantext`).
 256
 257 ## GraphQL
 258
 259 Some introspection information.
 260
 261 Playground is located at http://localhost:8008/gql
 262
 263 ### List all GraphQL types in the Playground
 264
 265 ```
 266 {
 267   __schema {
 268     types {
 269       name
 270     }
 271   }
 272 }
 273 ```
 274
 275 ### List details about a type in GraphQL
 276
 277 ```
 278 {
 279   __type(name:"User") {
 280         fields {
 281         name
 282       description
 283       type {
 284         name
 285       }
 286         }
 287         }
 288 }
 289 ```
 290 ## PostgreSQL
 291
 292 ### Upgrading using Docker
 293
 294 https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/
 295
 296 To upgrade PostgreSQL in Docker containers, for example from 11.x to 14.x, simply run:
 297 ```sh
 298 docker exec -it <container-id> pg_dumpall -U gargantua > 11-db.dump
 299 ```
 300
 301 Then, shut down the container, replace `image` section in
 302 `devops/docker/docker-compose.yaml` with `postgres:14`. Also, it is a good practice to create a new volume, say `garg-pgdata14` and bind the new container to it. If you want to keep the same volume, remember about removing it like so:
 303 ```sh
 304 docker-compose rm postgres
 305 docker volume rm docker_garg-pgdata
 306 ```
 307
 308 Now, start the container and execute:
 309 ```sh
 310 # need to drop the empty DB first, since schema will be created when restoring the dump
 311 docker exec -i <new-container-id> dropdb -U gargantua gargandbV5
 312 # recreate the db, but empty with no schema
 313 docker exec -i <new-container-id> createdb -U gargantua gargandbV5
 314 # now we can restore the dump
 315 docker exec -i <new-container-id> psql -U gargantua -d gargandbV5 < 11-db.dump
 316 ```
 317
 318 ### Upgrading using
 319
 320 There is a solution using pgupgrade_cluster but you need to manage the
 321 clusters version 14 and 13. Hence here is a simple solution to upgrade.
 322
 323 First save your data:
 324 ```
 325 sudo su postgres
 326 pg_dumpall > gargandb.dump
 327 ```
 328
 329 Upgrade postgresql:
 330 ```
 331 sudo apt install postgresql-server-14 postgresql-client-14
 332 sudo apt remove --purge postgresql-13
 333 ```
 334 Restore your data:
 335 ```
 336 sudo su postgres
 337 psql < gargandb.dump
 338 ```
 339
 340 Maybe you need to restore the gargantua password
 341 ```
 342 ALTER ROLE gargantua PASSWORD 'yourPasswordIn_gargantext.ini'
 343 ```
 344 Maybe you need to change the port to 5433 for database connection in
 345 your gargantext.ini file.
 346
 347
 348
 349