scip/esctl
1
0
Fork 0
mirror of https://codeberg.org/scip/esctl.git synced 2026-08-24 15:14:18 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
554c1b19b27fb9edc66925d73d9840a60428bde1
esctl/README.md
Thomas von Dein c5a21bd677 add esql sample
2026-07-10 15:14:23 +02:00

643 lines
24 KiB
Markdown
Raw Blame History

[![status-badge](https://ci.codeberg.org/api/badges/16999/status.svg)](https://ci.codeberg.org/repos/16999)
[![humanmande](https://img.shields.io/badge/human-made-green)](CONTRIBUTING.md)
[![License](https://img.shields.io/badge/license-GPL-blue.svg)](https://codeberg.org/scip/esctl/blob/master/LICENSE)
[![Go Report Card](https://goreportcard.com/badge/codeberg.org/scip/esctl)](https://goreportcard.com/report/codeberg.org/scip/esctl)
[![Latest Release](https://flat.badgen.net/codeberg/release/scip/esctl)](https://codeberg.org/scip/esctl/releases)
# esctl
Elasticsearch CLI
## Introduction
This is a handy cli tool which interfaces to an elasticsearch cluster
(or two of them if you're using cross cluster replication). It is a
work-in-progress project yet, things might change occasionally. Expect
a stable release once we reach major version 1.0.0.
Features:
- Configuration of cluster credentials using environment vars or
config file. Multiple clusters can be configured. `esctl cluster ls`
shows which one is reachable.
- Shell completion support (bash, zsh and fish). Put this into your
rc: `source <(esctl completion bash)`.
- Cluster settings can be viewed and modified.
- Comprehensive cluster status.
- Cluster reroute support.
- Search: you can search indices using full text or by fields, select
logical condition (OR, AND), use PIT, limit datetime (ES date math
can be used), etc. It is however not yet possible to create
recursive searches like: `(cond1 AND cond2) OR (cond3 OR cond4)`.
- Search using ES|QL language: `esctl searchql`.
- Cross cluster replication (ccr): view, pause, resume, delete
replication. You can also manage follower configuration.
- Index management: manage aliases, create, modify, delete indices,
display field mappings etc. Automatic rollover of aliases supported.
- Index template management: create, modify, delete etc
- Index alias management: create, modify, delete etc
- Node management: only list nodes yet.
- Shard management: only list shards yet.
- Snapshot management: only list snapshots yet.
- ILM management: list, create, delete etc.
- Task management: list and cancel tasks
- Role management: only list roles yet. There's also a `role diff`
subcommand, which is for internal use. It can be used to verify if
role defs in a CSV match the deployed roles.
- API documentation (`api list` and `api show <path>`) with
interactive markdown pager for endpoint documentation.
- Repl: this is an interactive REPL (read eval print loop) towards the
elasticsearch API. You can run API calls on the current selected
cluster w/o the hassle to specify the whole url, credentials etc. It
has line editing and history support. If `jq` is installed output
JSON will be syntax highlighted. A simple internal pager will be
used if output exceeds the terminal height. You can tweak this using
the `$ES_JSON_PAGER` environment variable (I'd recommend [fx](https://fx.wtf/)).
- Doc support. You can put, delete and show docs for an index. Very
handy if you want to play with it. Just create a new index:
`esctl index create foo` and then insert docs into it for search
experiments:
```console
esctl doc add -i foo '{"title":"curry in a hurry", "message":"australian thai"}'
```
## Installation
The tool does not have any dependencies. Just download the binary for
your platform from the releases page and you're good to go.
### Installation using a pre-compiled binary
You can use [stew](https://github.com/marwanhawari/stew) to install esctl:
```default
stew install https://codeberg.org/scip/esctl
```
Or go to the [latest release page](https://codeberg.org/scip/esctl/releases/)
and look for your OS and platform. There are two options to install the binary:
Directly download the binary for your platform,
e.g. `esctl-linux-amd64-0.0.2`, rename it to `esctl` (or whatever
you like more!) and put it into your bin dir (e.g. `$HOME/bin` or as
root to `/usr/local/bin`).
Be sure to verify the signature of the binary file. For this also
download the matching `esctl-linux-amd64-0.0.2.sha256` file and:
```shell
cat esctl-linux-amd64-0.0.2.sha25 && sha256sum esctl-linux-amd64-0.0.2
```
You should see the same SHA256 hash.
You may also download a binary tarball for your platform, e.g.
`esctl-linux-amd64-0.0.2.tar.gz`, unpack and install it. GNU Make is
required for this:
```shell
tar xvfz esctl-linux-amd64-0.0.2.tar.gz
cd esctl-linux-amd64-0.0.2
sudo make install
```
### Installation from source
Check out the repository and execute `go build`, then copy the
compiled binary to your `$PATH`.
Or, if you have GNU Make installed, just execute:
```default
make
sudo make install
```
## Configure
Configure `esctl` with environment variables:
- `ES_URI`: elasticsearch uri
- `ES_USER`: username
- `ES_PASS`: password
- `ES_TOKEN`: API token, instead of user+password
Or create a config file such as this:
```yaml
clusters:
foobar:
uri: https://es.foo.bar:9200/
user: elastic
pass: ******
other:
uri: https://myes.foo:9200/
token: ******
```
and specify it with `-c configfile`. You may also put clusters into a
default config file in `~/.config/esctl/config.yaml`. In this case you
can omit `-c ...`.
If you want to work on a specific cluster, you need to make it the
current default one. You can either manually configure it in the
config:
```yaml
clusters:
foobar:
uri: https://es.foo.bar:9200/
user: elastic
pass: 123456
default: true
```
or - if the cluster already exists in the config - issue this command:
```console
esctl cluster switch foobar
```
You may also temporary set a cluster as the current default with the
global `-C` option.
The following rules apply for default cluster selection:
1. If there's just one cluster configured (either via environment
variables or config), this one will be selected.
2. If a cluster have been specified with `-C <name>`, use this.
3. If there is one cluster configured with the `default` flag `true`,
use this one.
Use `esctl cluster ls` to check status and see which one would be
used.
Please be aware, that prior to use a cluster and send API requests to
it, `esctl` checks the connectivity to it first.
## Usage
Once you have configured one or more (and marked one as the default,
see above) elasticsearch clusters, you can start using `esctl`. Since
there are just too many things you can do with it, we'll just outline
a couple of examples here.
To see the status of a cluster:
```console
$ esctl cluster status
DEFAULT STATUS
Cluster Name eck-cluster-dev-s1
ES Status green
ES Version 9.4.2
Is Leader true
Active Shards 447
Active Primary Shards 221
Unassigned Shards 0
Unassigned Primary Shards 0
Pending Tasks 0
Nodes 3
Red Indices 0
Long Running Tasks 2
```
you may also add the `-v` flag to see more:
```console
$ esctl cluster status -v
DEFAULT STATUS
Cluster Name eck-cluster-dev-s1
ES Status green
ES Version 9.4.2
Is Leader true
Active Shards 447
Active Primary Shards 221
Unassigned Shards 0
Unassigned Primary Shards 0
Pending Tasks 0
Nodes 3
Red Indices 0
Long Running Tasks 2
Indicies 221
Docs 9854777
Total Size 3.4 GB
Total Queries 4210411
Shards Primaries 824642680872
Shards Total 824642680808
Storage 6.9 GB/6.9 GB
JVM Heap Memory 16 GB/39 GB
JVM Threads 458
JVM Version OpenJDK 64-Bit Server VM 26.0.1+8-34
CPUs 18
CPU Usage 0%
Open FDs 1423
```
View cluster settings:
```console
$ esctl cluster settings list
SETTING VALUE
cluster.remote.svct13.mode proxy
cluster.remote.svct13.proxy_address 100.127.161.24:9300
cluster.remote.svct13.proxy_socket_connections 18
cluster.remote.svct13.skip_unavailable false
cluster.routing.allocation.enable all
indices.recovery.max_bytes_per_sec 320mb
```
Settings can be changed, use json path to specify a setting:
```console
$ esctl cluster settings set indices.recovery.max_bytes_per_sec:480mb
```
Searching is pretty easy. First check which indices exist:
```console
$ esctl index ls
NAME SIZE DOCSCOUNT
[..]
foo 249b 0
foo2 31.4kb 14
mock 9.6kb 20
mr-strange 249b 0
silo18 5.2kb 1
```
Like a dataview in kibana you can specify an index pattern for search:
```console
$ esctl search -i foo*
{"id":"1779873299", "score":1.0000, "index":"foo2", "source":{"message":"foo bar"}}{"id":"1779870997", "score":1.0000, "index":"foo2", "source":{"message":"start","time":"27.05.2026T10:55:33+0200","title":"zeitbuchung"}}{"id":"1779870750", "score":1.0000, "index":"foo2", "source":{"message":"pause","time":"27.05.2026T10:33:45+0200","title":"zeitbuchung"}}{"id":"1779870103", "score":1.0000, "index":"foo2", "source":{"message":"punktlich gebucht","time":"27.05.2026T10:20:01+0200","title":"zeitbuchuung"}}{"id":"1779432806", "score":1.0000, "index":"foo2", "source":{"message":"strange compound 2"}}{"id":"1779432750", "score":1.0000, "index":"foo2", "source":{"message":"strange compound 1"}}{"id":"1778600947", "score":1.0000, "index":"foo2", "source":{"id":"d8d8d","user":"scip"}}{"id":"1778600945", "score":1.0000, "index":"foo2", "source":{"id":"d8d8d","user":"scip"}}{"id":"1778600857", "score":1.0000, "index":"foo2", "source":{"id":"d8d8d","user":"scip"}}{"id":"1778601788", "score":1.0000, "index":"foo2", "source":{"id":"new1","message":"live long and prosper"}}{"id":"1778601422", "score":1.0000, "index":"foo2", "source":{"id":"new1","message":"love long and prosper"}}{"id":"1779447242", "score":1.0000, "index":"foo2", "source":{"message":"australian thai","title":"curry in a hurry"}}{"id":"1779447284", "score":1.0000, "index":"foo2", "source":{"message":"german killer food","title":"wuerger burger"}}{"id":"1779364394", "score":1.0000, "index":"foo2", "source":{"id":"1","message":"non-dead human looks curious at TargetOne"}}
```
Oops, that looks gibberish. It's because `esctl` just prints the raw
JSON search results. There are multiple ways to post-process them:
You can use `jq` (here, we're using `-l 1` to limit the output to 1 document):
```console
$ esctl search -i foo* -l 1 | jq
{
"id": "1779364394",
"score": 1.0000,
"index": "foo2",
"source": {
"id": "1",
"message": "non-dead human looks curious at TargetOne"
}
}
```
However, you might as well use internal filtering using jsonpaths:
```console
$ esctl search -i foo* -l 1 -p source.message
non-dead human looks curious at TargetOne
```
To get more help about internal jsonpath filtering, run `esctl
help-jsonpath`.
A full text search looks like this:
```console
$ esctl search -i foo* thai | jq
{
"id": "1779447242",
"score": 2.3842,
"index": "foo2",
"source": {
"message": "australian thai",
"title": "curry in a hurry"
}
}
```
Instead of a costly full text you might as well specify the field to
search within:
```console
$ esctl search -i foo* message=thai | jq
{
"id": "1779447242",
"score": 2.3842,
"index": "foo2",
"source": {
"message": "australian thai",
"title": "curry in a hurry"
}
}
```
Of course you can search for multiple fields (or multiple terms in
full text search). By default search terms are applied using logical
AND, which you can change with the option `--or`.
You can also change the sorting (`--sort-by`) and the time range
(`--timerange`).
To further narrow down search results you can add filters, which must
match literally:
```console
$ esctl search -i foo* -F title=zeitbuchung message=pause | jq
{
"id": "1779870750",
"score": 2.8426,
"index": "foo2",
"source": {
"message": "pause",
"time": "27.05.2026T10:33:45+0200",
"title": "zeitbuchung"
}
}
```
You can also search using [ES|QL](https://www.elastic.co/docs/reference/query-languages/esql/esql-getting-started):
```console
$ esctl searchql "from hyperdrive | sort @timestamp | limit 5"
@TIMESTAMP MESSAGE TAG
2026-06-24T08:24:56.000Z arosu loop
2026-06-24T08:24:58.000Z hami loop
2026-06-24T08:24:59.000Z ishininu loop
2026-06-24T08:25:00.000Z uyomoruron loop
2026-06-24T08:25:02.000Z ishimime loop
```
There are several output modes (json, yaml, csv), to get esql output as CSV:
```console
$ esctl searchql "from hyperdrive | sort @timestamp | limit 5" -o csv
@timestamp,message,tag
2026-06-24T08:24:56.000Z,arosu,loop
2026-06-24T08:24:58.000Z,hami,loop
2026-06-24T08:24:59.000Z,ishininu,loop
2026-06-24T08:25:00.000Z,uyomoruron,loop
2026-06-24T08:25:02.000Z,ishimime,loop
```
To check which field mappings are available for an index:
```console
$ esctl index show foo2
INDEX-PROPERTY VALUE
name foo2
replicas 1
shards 1
created 2026-05-12 17:47:37
uuid XmRtlh_JSreIzTB7jJNNJA
fields id,message,time,title,user
```
Now, let's create a new index:
```console
$ esctl index create hyperdrive @timestamp:date message:keyword tag:text
esctl index show hyperdrive
INDEX-PROPERTY VALUE
name hyperdrive
replicas 1
shards 1
created 2026-06-24 10:16:23
uuid gYfUj8TiR6aMB_BHmMYr3Q
fields @timestamp,message,tag
```
Once we have an index, we can insert documents into it (you'll need
[gfn](https://codeberg.org/scip/gfn) for this):
```console
$ for name in $(gfn JapaneseNamesDiverse -n 20); do \
esctl doc add -i hyperdrive \
"{\"message\":\"$name\",\"tag\":\"loop\",\"@timestamp\":\"$(date
--iso-8601=second)\"}"; \
sleep 1; \
done
3644770736267480972
1249004675657072369
5499556813583921546
947619563748335783
6804953496467561176
1716813071323879201
463762522269463473
4674152788118344945
7741776772191747414
8531321287239151007
3335734000533108391
7020487792409206103
4926995290680601261
374509677635815298
49895974689085266
7603943644788310
2102931816420108217
696238814931581311
2463257155346426021
596556586001412147
```
Now, let's search them:
```console
$ esctl search -i hyperdrive -l 2 | jq
{
"id": "2463257155346426021",
"score": 0.0000,
"index": "hyperdrive",
"source": {
"@timestamp": "2026-06-24T10:25:19+02:00",
"message": "inenen",
"tag": "loop"
}
}
{
"id": "596556586001412147",
"score": 0.0000,
"index": "hyperdrive",
"source": {
"@timestamp": "2026-06-24T10:25:20+02:00",
"message": "ominuhen",
"tag": "loop"
}
}
```
There are also some uniq features which are not directly available via
the ES API or GUI. Here's one example: say you have a number of
indices with ILM policies each. At some day there were a surge in
incoming data in some of them and you want to know, when the excess
data will be rolled over to cold storage:
```console
$ esctl ilm forecast sh -f warm -w 3d
1.1 TB bytes of data in warm phase will be rolled within 72h0m0s to the next phase
```
You may also look at a detailed ilm forecast list:
```console
$ esctl ilm forecast ls -f warm -w 3d
INDEX CURRENT-SIZE CURRENT-AGE VIRTUAL-AGE MIN-AGE MIN-SIZE CURRENT-PHASE NEXT-PHASE
foobar-n1-p01-elastic-000126 12 GB 2d:8h:12m 3d:5h:16m 7d 25 GB hot warm
foobar-n1-p01-misc-000070 16 GB 5d:16h:31m 4d:10h:31m 7d 25 GB hot warm
foobar-n1-q01-elastic-000122 23 GB 5d:0h:11m 6d:10h:33m 7d 25 GB hot warm
foobar-n1-q01-misc-000072 18 GB 6d:15h:21m 4d:21h:36m 7d 25 GB hot warm
delaware-f1-p01-kafka-000023 1.3 MB 15h:12m 0s 7d 25 GB hot warm
delaware-f1-p01-misc-000024 17 GB 6d:16h:1m 4d:16h:33m 7d 25 GB hot warm
delaware-f1-q01-kafka-000023 1.5 MB 15h:12m 0s 7d 25 GB hot warm
delaware-f1-q01-misc-000024 16 GB 6d:15h:31m 4d:13h:52m 7d 25 GB hot warm
delaware-n1-p01-elastic-000417 16 GB 12h:42m 4d:10h:31m 7d 25 GB hot warm
```
So you can see, which index will be rolled over when. Note the
**VIRTUAL-AGE** field however: it is calculated from the current
storage usage of the index in relation to rollover max shard size. So
you can see, when an index will be rolled over either because it aged
out or because its storage exceeded the limit.
---
Please note, that `esctl` is still in its early stages and things are
changing heavily every now and then. New commands are being added
constantly as well.
### Command tree:
```console
api - api access and documentation
list - list index of API calls
show - show an API doc
repl - interactive API repl
ccr - manage cross cluster replication
status - cross cluster replication status (yaml config with 2 clusters required)
pause - pause shard allocation
resume - resume shard allocation
follower - manage ccr follower indices
show - show ccr follower index details
add - add ccr follower index
delete - delete ccr follower index
unfollow - unfollow ccr follower index
pause - pause ccr index to follow
resume - resume ccr index to follow
renew - renew ccr follower index
info - show ccr remote info
cluster - manage cluster[s]
status - show cluster status
switch - set current elasticsearch cluster
list - list configured clusters
settings - cluster settings management
list - show cluster settings
set - set|update cluster settings
reroute - manually change the allocation of individual shards in the cluster.
move - move shard to another node
allocate-replica - allocate-replica replica to another node
cancel - cancel a reroute operation
allocate-empty-primary - allocate an empty primary shard to a node
allocate-stale-primary - allocate a stale primary shard to a node
datastream - manage data streams
list - list indicies
show - show details about an data stream
create - create a new data stream
delete - delete a data stream
rollover - roll over a data stream
doc - manage documents
add - add JSON document index
show - show a JSON document
delete - delete JSON document[s] from index[es]
ilm - manage index lifecycle
retry - retry applying an ILM profile to an index
status - get the current index lifecycle management status
list - list index lifecycle policies
show - show details about an index lifecycle policy
create - create a new lifecycle policy
update - update an lifecycle policy
forecast - calculate index phase movements
list - list index rollover config
show - show rollover forecast over all indices
explain - explain ilm condition of an index
index - manage indicies
list - list indicies
show - show details about an index
create - create a new index
update - update an index
delete - delete an index
close - close an index
fields - show info about field capabilities
ilm - show ilm status
du - show index disk usage
alias - manage index aliases
create - create an index alias
list - list index aliases
delete - delete an index alias
rollover - roll over an index alias
template - manage index templates
list - list index templates
show - show details about an index template
create - create a new index template
update - update a new index template
delete - delete an index template
license - manage cluster license
show - show details about the cluster license
node - manage nodes
list - list nodes
show - show details about a node
clients - show node http clients
usage - show node usage stats
role - manage roles
list - list roles
show - show details about a role
diff - show differences between roles and CSV baseline
search - search within an index
shard - manage shards
list - list shards
show - show details about a shard
allocation - explain shard allocation
snapshot - manage snapshots
list - list snapshots
show - show details about a snapshot
task - manage tasks
list - list tasks
cancel - cancel running task
version - show esctl version information
debug - developer only
help-jsonpath - show jsonpath help
help-usage - show overview of all available commands
help-esql - show esql help
```
# Development
## To test completion
Add the flag `--generate-shell-completion` to any command, e.g.:
```console
./esctl role show --generate-shell-completion
machine_learning_admin
rollup_admin
editor
reporting_user
snapshot_user
fcn_admin
machine_learning_user
kibana_system
beats_admin
kibana_user
fcns_space
transport_client
transform_user
[..]
```
# Report bugs
[Please open an issue](https://codeberg.org/scip/esctl/issues). Thanks!
# License
This work is licensed under the terms of the General Public Licens
version 3.
# Author
Copyleft (c) 2026 Thomas von Dein
Reference in New Issue View Git Blame Copy Permalink