README.md

   1 # Gargantext with Haskell (Backend instance)
   2
   3 ## About the project
   4
   5 GarganText is a collaborative web-decentralized-based macro-service
   6 platform for the exploration of unstructured texts. It combines tools
   7 from natural language processing, text-data-mining tricks, complex
   8 networks analysis algorithms and interactive data visualization tools to
   9 pave the way toward new kinds of interactions with your digital corpora.
  10
  11 This software is free software, developed and offered by the CNRS
  12 Complex Systems Institute of Paris Île-de-France (ISC-PIF) and its
  13 partners.
  14
  15 GarganText Project: this repo builds the
  16 backend for the frontend server built by
  17 [backend](https://gitlab.iscpif.fr/gargantext/haskell-gargantext).
  18
  19
  20 ## Installation
  21
  22 Disclaimer: this project is still in development, this is work in
  23 progress. Please report and improve this documentation if you encounter issues.
  24
  25 ### Stack setup
  26
  27 You need to install stack first:
  28
  29 ```shell
  30 curl -sSL https://get.haskellstack.org/ | sh
  31 ```
  32
  33 Verify the installation is complete with
  34 ```shell
  35 stack --version
  36 ```
  37
  38 ### With Nix setup
  39
  40 First install [nix](https://nixos.org/guides/install-nix.html):
  41
  42 ```shell
  43 curl -sSL https://nixos.org/nix/install | sh
  44 ```
  45
  46 Verify the installation is complete
  47 ```shell
  48 $ nix-env
  49 nix-env (Nix) 2.3.12
  50 ```
  51 And just build:
  52 ``` sh
  53 stack --nix build --fast
  54 ```
  55
  56 ### Build Core Code
  57
  58 NOTE: Default build (with optimizations) requires large amounts of RAM
  59 (16GB at least). To avoid heavy compilation times and swapping out your
  60 machine, it is recommended to `stack build` with the `--fast-` flag,
  61 i.e.:
  62
  63 ``` sh
  64 stack --nix build --fast
  65 ```
  66 or
  67
  68 ``` sh
  69 stack --docker build --fast
  70 ```
  71
  72 #### Docker
  73
  74 ``` sh
  75 curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/docker/docker-install | sh
  76 ```
  77
  78 #### Debian
  79
  80 ``` sh
  81 curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/debian/install | sh
  82 ```
  83
  84 #### Ubuntu
  85
  86 ``` sh
  87 curl -sSL https://gitlab.iscpif.fr/gargantext/haskell-gargantext/raw/dev/devops/ubuntu/install | sh
  88 ```
  89
  90 ### Add dependencies
  91
  92 1. CoreNLP is needed (EN and FR); This dependency will not be needed soon.
  93
  94 ``` sh
  95 ./devops/install-corenlp
  96 ```
  97 ### Launch Gargantext (It should have been initialized first)
  98 Run
  99 haskell-gargantext/devops/docker$ docker-compose up
 100
 101 Then
 102 ``` sh
 103 stack --nix install
 104 ~/.local/bin/gargantext-init "gargantext.ini"
 105 ```
 106 Or for Docker env, first create the appropriate image:
 107
 108 ``` sh
 109 cd devops/docker
 110 docker build -t cgenie/stack-build:lts-18.12-garg .
 111 ```
 112
 113 then run:
 114
 115 ``` sh
 116 stack --docker exec gargantext-init -- gargantext.ini
 117
 118 ### Initialization
 119
 120 #### Docker
 121
 122 Run PostgreSQL first:
 123
 124 ``` sh
 125 cd devops/docker
 126 docker-compose up
 127 ```
 128
 129 Initialization schema should be loaded automatically (from `devops/postgres/schema.sql`).
 130
 131 #### Gargantext
 132
 133 ##### Fix the passwords
 134
 135 Change the passwords in gargantext.ini_toModify then move it:
 136
 137 ``` sh
 138 cp gargantext.ini_toModify gargantext.ini
 139 ```
 140 (`.gitignore` avoids adding this file to the repository by mistake)
 141
 142
 143 ##### Run Gargantext
 144
 145 Users have to be created first (`user1` is created as instance):
 146
 147 ``` sh
 148 stack --nix install
 149 ~/.local/bin/gargantext-init "gargantext.ini"
 150 ```
 151
 152 For Docker env, first create the appropriate image:
 153
 154 ``` sh
 155 cd devops/docker
 156 docker build -t cgenie/stack-build:lts-18.12-garg .
 157 ```
 158
 159 then run:
 160
 161 ``` sh
 162 stack --docker exec gargantext-init -- gargantext.ini
 163 ```
 164
 165 ### Importing data
 166
 167 You can import some data with:
 168 ``` sh
 169 docker run --rm -it -p 9000:9000 cgenie/corenlp-garg
 170 stack exec gargantext-import -- "corpusCsvHal" "user1" "IMT3" gargantext.ini 10000 ./1000.csv
 171 ```
 172
 173 ### Nix
 174
 175 It is also possible to build everything with [Nix](https://nixos.org/) instead of Docker:
 176 ``` sh
 177 stack --nix build
 178 stack --nix exec gargantext-import -- "corpusCsvHal" "user1" "IMT3" gargantext.ini 10000 ./1000.csv
 179 stack --nix exec gargantext-server -- --ini gargantext.ini --run Prod
 180 ```
 181
 182 ## Use Cases
 183
 184 ### Multi-User with Graphical User Interface (Server Mode)
 185
 186 ``` sh
 187 ~/.local/bin/stack --docker exec gargantext-server -- --ini "gargantext.ini" --run Prod
 188 ```
 189
 190 Then you can log in with `user1` / `1resu`.
 191
 192
 193 ### Command Line Mode tools
 194
 195 #### Simple cooccurrences computation and indexation from a list of Ngrams
 196
 197 ``` sh
 198 stack --docker exec gargantext-cli -- CorpusFromGarg.csv ListFromGarg.csv Ouput.json
 199 ```
 200
 201 ### Analyzing the ngrams table repo
 202
 203 We store the repository in directory `repos` in the [CBOR](https://cbor.io/)
 204 file format. To decode it to JSON and analyze, say, using
 205 [jq](https://shapeshed.com/jq-json/), use the following command:
 206
 207 ``` sh
 208 cat repos/repo.cbor.v5 | stack --nix exec gargantext-cbor2json | jq .
 209 ```
 210 ### Documentation
 211
 212 To build documentation, run:
 213
 214 ```sh
 215 stack --docker build --haddock --no-haddock-deps --fast
 216 ```
 217
 218 (in `.stack-work/dist/x86_64-linux-nix/Cabal-3.2.1.0/doc/html/gargantext`).
 219
 220 ## GraphQL
 221
 222 Some introspection information.
 223
 224 Playground is located at http://localhost:8008/gql
 225
 226 ### List all GraphQL types in the Playground
 227
 228 ```
 229 {
 230   __schema {
 231     types {
 232       name
 233     }
 234   }
 235 }
 236 ```
 237
 238 ### List details about a type in GraphQL
 239
 240 ```
 241 {
 242   __type(name:"User") {
 243         fields {
 244         name
 245       description
 246       type {
 247         name
 248       }
 249         }
 250         }
 251 }
 252 ```
 253 ## PostgreSQL
 254
 255 ### Upgrading using Docker
 256
 257 https://www.cloudytuts.com/tutorials/docker/how-to-upgrade-postgresql-in-docker-and-kubernetes/
 258
 259 To upgrade PostgreSQL in Docker containers, for example from 11.x to 14.x, simply run:
 260 ```sh
 261 docker exec -it <container-id> pg_dumpall -U gargantua > 11-db.dump
 262 ```
 263
 264 Then, shut down the container, replace `image` section in
 265 `devops/docker/docker-compose.yaml` with `postgres:14`. Also, it is a good practice to create a new volume, say `garg-pgdata14` and bind the new container to it. If you want to keep the same volume, remember about removing it like so:
 266 ```sh
 267 docker-compose rm postgres
 268 docker volume rm docker_garg-pgdata
 269 ```
 270
 271 Now, start the container and execute:
 272 ```sh
 273 # need to drop the empty DB first, since schema will be created when restoring the dump
 274 docker exec -i <new-container-id> dropdb -U gargantua gargandbV5
 275 # recreate the db, but empty with no schema
 276 docker exec -i <new-container-id> createdb -U gargantua gargandbV5
 277 # now we can restore the dump
 278 docker exec -i <new-container-id> psql -U gargantua -d gargandbV5 < 11-db.dump
 279 ```
 280
 281 ### Upgrading using
 282
 283 There is a solution using pgupgrade_cluster but you need to manage the
 284 clusters version 14 and 13. Hence here is a simple solution to upgrade.
 285
 286 First save your data:
 287 ```
 288 sudo su postgres
 289 pg_dumpall > gargandb.dump
 290 ```
 291
 292 Upgrade postgresql:
 293 ```
 294 sudo apt install postgresql-server-14 postgresql-client-14
 295 sudo apt remove --purge postgresql-13
 296 ```
 297 Restore your data:
 298 ```
 299 sudo su postgres
 300 psql < gargandb.dump
 301 ```
 302
 303 Maybe you need to restore the gargantua password
 304 ```
 305 ALTER ROLE gargantua PASSWORD 'yourPasswordIn_gargantext.ini'
 306 ```
 307 Maybe you need to change the port to 5433 for database connection in
 308 your gargantext.ini file.
 309
 310
 311
 312