diff --git a/README.md b/README.md index 33e11f9..7174530 100644 --- a/README.md +++ b/README.md @@ -56,9 +56,388 @@ Features: esctl doc add -i foo '{"title":"curry in a hurry", "message":"australian thai"}' ``` +## Installation + +The tool does not have any dependencies. Just download the binary for +your platform from the releases page and you're good to go. + +### Installation using a pre-compiled binary + +You can use [stew](https://github.com/marwanhawari/stew) to install esctl: +```default +stew install https://codeberg.org/scip/esctl +``` + +Or go to the [latest release page](https://codeberg.org/scip/esctl/releases/) +and look for your OS and platform. There are two options to install the binary: + +Directly download the binary for your platform, +e.g. `esctl-linux-amd64-0.0.2`, rename it to `esctl` (or whatever +you like more!) and put it into your bin dir (e.g. `$HOME/bin` or as +root to `/usr/local/bin`). + +Be sure to verify the signature of the binary file. For this also +download the matching `esctl-linux-amd64-0.0.2.sha256` file and: + +```shell +cat esctl-linux-amd64-0.0.2.sha25 && sha256sum esctl-linux-amd64-0.0.2 +``` +You should see the same SHA256 hash. + +You may also download a binary tarball for your platform, e.g. +`esctl-linux-amd64-0.0.2.tar.gz`, unpack and install it. GNU Make is +required for this: + +```shell +tar xvfz esctl-linux-amd64-0.0.2.tar.gz +cd esctl-linux-amd64-0.0.2 +sudo make install +``` + +### Installation from source + +Check out the repository and execute `go build`, then copy the +compiled binary to your `$PATH`. + +Or, if you have GNU Make installed, just execute: + +```default +make +sudo make install +``` + +## Configure + +Configure `esctl` with environment variables: + +- `ES_URI`: elasticsearch uri +- `ES_USER`: username +- `ES_PASS`: password + +Or create a config file such as this: + +```yaml +clusters: + foobar: + uri: https://es.foo.bar:9200/ + user: elastic + pass: 123456 + other: + uri: https://myes.foo:9200/ + user: elastic + pass: asdasdasd +``` + +and specify it with `-c configfile`. You may also put clusters into a +default config file in `~/.config/esctl/config.yaml`. In this case you +can omit `-c ...`. + +If you want to work on a specific cluster, you need to make it the +current default one. You can either manually configure it in the +config: + +```yaml +clusters: + foobar: + uri: https://es.foo.bar:9200/ + user: elastic + pass: 123456 + default: true +``` + +or - if the cluster already exists in the config - issue this command: + +```console +esctl cluster switch foobar +``` + +You may also temporary set a cluster as the current default with the +global `-C` option. + +The following rules apply for default cluster selection: + +1. If there's just one cluster configured (either via environment + variables or config), this one will be selected. +2. If a cluster have been specified with `-C `, use this. +3. If there is one cluster configured with the `default` flag `true`, + use this one. + +Use `esctl cluster ls` to check status and see which one would be +used. + +Please be aware, that prior to use a cluster and send API requests to +it, `esctl` checks the connectivity to it first. + ## Usage -Command tree: +Once you have configured one or more (and marked one as the default, +see above) elasticsearch clusters, you can start using `esctl`. Since +there are just too many things you can do with it, we'll just outline +a couple of examples here. + +To see the status of a cluster: + +```console +$ esctl cluster status +DEFAULT STATUS +Cluster Name eck-cluster-dev-s1 +ES Status green +ES Version 9.4.2 +Is Leader true +Active Shards 447 +Active Primary Shards 221 +Unassigned Shards 0 +Unassigned Primary Shards 0 +Pending Tasks 0 +Nodes 3 +Red Indices 0 +Long Running Tasks 2 +``` + +you may also add the `-v` flag to see more: + +```console +esctl cluster status -v +DEFAULT STATUS +Cluster Name eck-cluster-dev-s1 +ES Status green +ES Version 9.4.2 +Is Leader true +Active Shards 447 +Active Primary Shards 221 +Unassigned Shards 0 +Unassigned Primary Shards 0 +Pending Tasks 0 +Nodes 3 +Red Indices 0 +Long Running Tasks 2 +Indicies 221 +Docs 9854777 +Total Size 3.4 GB +Total Queries 4210411 +Shards Primaries 824642680872 +Shards Total 824642680808 +Storage 6.9 GB/6.9 GB +JVM Heap Memory 16 GB/39 GB +JVM Threads 458 +JVM Version OpenJDK 64-Bit Server VM 26.0.1+8-34 +CPUs 18 +CPU Usage 0% +Open FDs 1423 +``` + +View cluster settings: + +```console +$ esctl cluster settings list +SETTING VALUE +cluster.remote.svct13.mode proxy +cluster.remote.svct13.proxy_address 100.127.161.24:9300 +cluster.remote.svct13.proxy_socket_connections 18 +cluster.remote.svct13.skip_unavailable false +cluster.routing.allocation.enable all +indices.recovery.max_bytes_per_sec 320mb +``` + +Settings can be changed, use json path to specify a setting: + +```console +$ esctl cluster settings set indices.recovery.max_bytes_per_sec:480mb +``` + +Searching is pretty easy. First check which indices exist: + +```console +$ esctl index ls +NAME SIZE DOCSCOUNT +[..] +foo 249b 0 +foo2 31.4kb 14 +mock 9.6kb 20 +mr-strange 249b 0 +silo18 5.2kb 1 +``` + +Like a dataview in kibana you can specify an index pattern for search: + +```console +$ esctl search -i foo* +{"id":"1779873299", "score":1.0000, "index":"foo2", "source":{"message":"foo bar"}}{"id":"1779870997", "score":1.0000, "index":"foo2", "source":{"message":"start","time":"27.05.2026T10:55:33+0200","title":"zeitbuchung"}}{"id":"1779870750", "score":1.0000, "index":"foo2", "source":{"message":"pause","time":"27.05.2026T10:33:45+0200","title":"zeitbuchung"}}{"id":"1779870103", "score":1.0000, "index":"foo2", "source":{"message":"punktlich gebucht","time":"27.05.2026T10:20:01+0200","title":"zeitbuchuung"}}{"id":"1779432806", "score":1.0000, "index":"foo2", "source":{"message":"strange compound 2"}}{"id":"1779432750", "score":1.0000, "index":"foo2", "source":{"message":"strange compound 1"}}{"id":"1778600947", "score":1.0000, "index":"foo2", "source":{"id":"d8d8d","user":"scip"}}{"id":"1778600945", "score":1.0000, "index":"foo2", "source":{"id":"d8d8d","user":"scip"}}{"id":"1778600857", "score":1.0000, "index":"foo2", "source":{"id":"d8d8d","user":"scip"}}{"id":"1778601788", "score":1.0000, "index":"foo2", "source":{"id":"new1","message":"live long and prosper"}}{"id":"1778601422", "score":1.0000, "index":"foo2", "source":{"id":"new1","message":"love long and prosper"}}{"id":"1779447242", "score":1.0000, "index":"foo2", "source":{"message":"australian thai","title":"curry in a hurry"}}{"id":"1779447284", "score":1.0000, "index":"foo2", "source":{"message":"german killer food","title":"wuerger burger"}}{"id":"1779364394", "score":1.0000, "index":"foo2", "source":{"id":"1","message":"non-dead human looks curious at TargetOne"}} +``` + +Oops, that looks gibberish. It's because `esctl` just prints the raw +JSON search results. There are multiple ways to post-process them: + +You can use `jq` (here, we're using `-l 1` to limit the output to 1 document): + +```console +$ esctl search -i foo* -l 1 | jq +{ + "id": "1779364394", + "score": 1.0000, + "index": "foo2", + "source": { + "id": "1", + "message": "non-dead human looks curious at TargetOne" + } +} +``` + +However, you might as well use internal filtering using jsonpaths: + +```console +$ esctl search -i foo* -l 1 -p source.message +non-dead human looks curious at TargetOne +``` + +To get more help about internal jsonpath filtering, run `esctl +help-jsonpath`. + +A full text search looks like this: + +```console +$ esctl search -i foo* thai | jq +{ + "id": "1779447242", + "score": 2.3842, + "index": "foo2", + "source": { + "message": "australian thai", + "title": "curry in a hurry" + } +} +``` + +Instead of a costly full text you might as well specify the field to +search within: + +```console +$ esctl search -i foo* message=thai | jq +{ + "id": "1779447242", + "score": 2.3842, + "index": "foo2", + "source": { + "message": "australian thai", + "title": "curry in a hurry" + } +} +``` + +Of course you can search for multiple fields (or multiple terms in +full text search). By default search terms are applied using logical +AND, which you can change with the option `--or`. + +You can also change the sorting (`--sort-by`) and the time range +(`--timerange`). + +To further narrow down search results you can add filters, which must +match literally: + +```console +$ esctl search -i foo* -F title=zeitbuchung message=pause | jq +{ + "id": "1779870750", + "score": 2.8426, + "index": "foo2", + "source": { + "message": "pause", + "time": "27.05.2026T10:33:45+0200", + "title": "zeitbuchung" + } +} +``` + +To check which field mappings are available for an index: +```console +$ esctl index show foo2 +INDEX-PROPERTY VALUE +name foo2 +replicas 1 +shards 1 +created 2026-05-12 17:47:37 +uuid XmRtlh_JSreIzTB7jJNNJA +fields id,message,time,title,user +``` + +Now, let's create a new index: + +```console +$ esctl index create hyperdrive @timestamp:date message:keyword tag:text + +esctl index show hyperdrive +INDEX-PROPERTY VALUE +name hyperdrive +replicas 1 +shards 1 +created 2026-06-24 10:16:23 +uuid gYfUj8TiR6aMB_BHmMYr3Q +fields @timestamp,message,tag +``` + +Once we have an index, we can insert documents into it (you'll need +[gfn](https://codeberg.org/scip/gfn) for this): + +```console +$ for name in $(gfn JapaneseNamesDiverse -n 20); do \ + esctl doc add -i hyperdrive \ + "{\"message\":\"$name\",\"tag\":\"loop\",\"@timestamp\":\"$(date + --iso-8601=second)\"}"; \ + sleep 1; \ + done +3644770736267480972 +1249004675657072369 +5499556813583921546 +947619563748335783 +6804953496467561176 +1716813071323879201 +463762522269463473 +4674152788118344945 +7741776772191747414 +8531321287239151007 +3335734000533108391 +7020487792409206103 +4926995290680601261 +374509677635815298 +49895974689085266 +7603943644788310 +2102931816420108217 +696238814931581311 +2463257155346426021 +596556586001412147 +``` + +Now, let's search them: + +```console +$ esctl search -i hyperdrive -l 2 | jq +{ + "id": "2463257155346426021", + "score": 0.0000, + "index": "hyperdrive", + "source": { + "@timestamp": "2026-06-24T10:25:19+02:00", + "message": "inenen", + "tag": "loop" + } +} +{ + "id": "596556586001412147", + "score": 0.0000, + "index": "hyperdrive", + "source": { + "@timestamp": "2026-06-24T10:25:20+02:00", + "message": "ominuhen", + "tag": "loop" + } +} +``` + +Please note, that `esctl` is still in its early stages and things are +changing heavily every now and then. New commands are being added +constantly as well. + +### Command tree: ```console api - api access and documentation @@ -149,116 +528,6 @@ completion - Output shell completion script for bash, zsh, fish, or P pwsh - Output pwsh completion script ``` -Configure `esctl` with environment variables: - -- `ES_URI`: elasticsearch uri -- `ES_USER`: username -- `ES_PASS`: password - -Or create a config file such as this: - -```yaml -clusters: - foobar: - uri: https://es.foo.bar:9200/ - user: elastic - pass: 123456 - other: - uri: https://myes.foo:9200/ - user: elastic - pass: asdasdasd -``` - -and specify it with `-c configfile`. You may also put clusters into a -default config file in `~/.config/esctl/config.yaml`. In this case you -can omit `-c ...`. - -If you want to work on a specific cluster, you need to make it the -current default one. You can either manually configure it in the -config: - -```yaml -clusters: - foobar: - uri: https://es.foo.bar:9200/ - user: elastic - pass: 123456 - default: true -``` - -or - if the cluster already exists in the config - issue this command: - -```console -esctl cluster switch foobar -``` - -You may also temporary set a cluster as the current default with the -global `-C` option. - -The following rules apply for default cluster selection: - -1. If there's just one cluster configured (either via environment - variables or config), this one will be selected. -2. If a cluster have been specified with `-C `, use this. -3. If there is one cluster configured with the `default` flag `true`, - use this one. - -Use `esctl cluster ls` to check status and see which one would be -used. - -Please be aware, that prior to use a cluster and send API requests to -it, `esctl` checks the connectivity to it first. - -## Installation - -The tool does not have any dependencies. Just download the binary for -your platform from the releases page and you're good to go. - -### Installation using a pre-compiled binary - -You can use [stew](https://github.com/marwanhawari/stew) to install esctl: -```default -stew install https://codeberg.org/scip/esctl -``` - -Or go to the [latest release page](https://codeberg.org/scip/esctl/releases/) -and look for your OS and platform. There are two options to install the binary: - -Directly download the binary for your platform, -e.g. `esctl-linux-amd64-0.0.2`, rename it to `esctl` (or whatever -you like more!) and put it into your bin dir (e.g. `$HOME/bin` or as -root to `/usr/local/bin`). - -Be sure to verify the signature of the binary file. For this also -download the matching `esctl-linux-amd64-0.0.2.sha256` file and: - -```shell -cat esctl-linux-amd64-0.0.2.sha25 && sha256sum esctl-linux-amd64-0.0.2 -``` -You should see the same SHA256 hash. - -You may also download a binary tarball for your platform, e.g. -`esctl-linux-amd64-0.0.2.tar.gz`, unpack and install it. GNU Make is -required for this: - -```shell -tar xvfz esctl-linux-amd64-0.0.2.tar.gz -cd esctl-linux-amd64-0.0.2 -sudo make install -``` - -### Installation from source - -Check out the repository and execute `go build`, then copy the -compiled binary to your `$PATH`. - -Or, if you have GNU Make installed, just execute: - -```default -make -sudo make install -``` - # Development ## To test completion