Release v1.0.17 (#9)

* add shortcut -H to --no-headers, it's too cumbersome to type
* added fuzzy search support
* Added basic lisp plugin facilities
* Lisp plugin Addidions:
- added process hook facilities
- added working example lisp plugin for filter hook
- load-path can now be a file as well
- added a couple of lisp helper functions (atoi, split), more may
  follow, see lisplib.go
* linting fixes

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,31 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[bug-report]"
+labels: bug
+assignees: TLINDEN
+
+---
+
+**Describtion**
+<!-- Please provide a clear and concise description of the issue: -->
+
+
+**Steps To Reproduce**
+<!-- Please detail the steps to reproduce the behavior: -->
+
+
+**Expected behavior**
+<!-- What do you expected to happen instead? -->
+
+
+**Version information**
+<!--
+Please provide as much version information as possible:
+- if you have just installed a binary, provide the output of: tablizer --version
+- if you installed from source, provide the output of: make show-version
+- provide additional details: operating system and version and shell environment
+-->
+
+
+**Additional informations**

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,23 @@
+---
+name: Feature request
+about: Suggest a feature
+title: "[feature-request]"
+labels: feature-request
+assignees: TLINDEN
+
+---
+
+**Describtion**
+<!-- Please provide a clear and concise description of the feature you desire: -->
+
+
+
+**Version information**
+<!--
+Just in case the feature is already present, please provide as
+much version information as possible:
+- if you have just installed a binary, provide the output of: tablizer --version
+- if you installed from source, provide the output of: make show-version
+- provide additional details: operating system and version and shell environment
+-->
+

.gitignore

@@ -0,0 +1,2 @@
+releases
+tablizer

CHANGELOG.md

@@ -4,7 +4,31 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org).
 
-## [v1.0.12](https://github.com/TLINDEN/tablizer/tree/v1.0.13) - 2022-11-03
+## [v1.0.14](https://github.com/TLINDEN/tablizer/tree/v1.0.14) - 2023-01-23
+
+[Full Changelog](https://github.com/TLINDEN/tablizer/compare/v1.0.13...v1.0.14)
+
+### Fixed
+
+- The -D parameter could not be used together with -a.
+
+- Fixed invalid argv handling: when the user wanted to read from stdin
+  but gave an argument which was  meant as a pattern, but also existed
+  as a filename, then tablizer opened the file, ignored stdin.
+  
+- Makefile indentation
+
+  
+### Added
+
+- added licens notes about dependencies
+
+- using hard coded uniseq version, see actions#3396457307
+
+- updated dependencies (go module versions)
+
+
+## [v1.0.13](https://github.com/TLINDEN/tablizer/tree/v1.0.13) - 2022-11-03
 
 [Full Changelog](https://github.com/TLINDEN/tablizer/compare/v1.0.12...v1.0.13)
 
@@ -18,7 +42,9 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 - The `-A` flag wasn't implemented (default output mode).
 
-
+- Fixed building from source on systems w/o perls pod tools,
+  which is not requrired anyway since I always commit the latest
+  manpage.
 
 
 ## [v1.0.12](https://github.com/TLINDEN/tablizer/tree/v1.0.12) - 2022-10-25

Makefile

@@ -27,14 +27,17 @@ BRANCH  = $(shell git branch --show-current)
 COMMIT    = $(shell git rev-parse --short=8 HEAD)
 BUILD     = $(shell date +%Y.%m.%d.%H%M%S) 
 VERSION  := $(if $(filter $(BRANCH), development),$(version)-$(BRANCH)-$(COMMIT)-$(BUILD),$(version))
-
+HAVE_POD := $(shell pod2text -h 2>/dev/null)
 
 all: $(tool).1 cmd/$(tool).go buildlocal
 
 %.1: %.pod
+ifdef HAVE_POD
 	pod2man -c "User Commands" -r 1 -s 1 $*.pod > $*.1
+endif
 
 cmd/%.go: %.pod
+ifdef HAVE_POD
 	echo "package cmd" > cmd/$*.go
 	echo >> cmd/$*.go
 	echo "var manpage = \`" >> cmd/$*.go
@@ -44,6 +47,7 @@ cmd/%.go: %.pod
 	echo "var usage = \`" >> cmd/$*.go
 	awk '/SYNOPS/{f=1;next} /DESCR/{f=0} f' $*.pod  | sed 's/^    //' >> cmd/$*.go
 	echo "\`" >> cmd/$*.go
+endif
 
 buildlocal:
 	go build -ldflags "-X 'github.com/tlinden/tablizer/cfg.VERSION=$(VERSION)'"
@@ -63,6 +67,7 @@ clean:
 
 test:
 	go test -v ./...
+	bash t/test.sh
 
 singletest:
 	@echo "Call like this: ''make singletest TEST=TestPrepareColumns MOD=lib"
@@ -71,3 +76,18 @@ singletest:
 cover-report:
 	go test ./... -cover -coverprofile=coverage.out
 	go tool cover -html=coverage.out
+
+show-versions: buildlocal
+	@echo "### tablizer version:"
+	@./tablizer --version
+
+	@echo
+	@echo "### go module versions:"
+	@go list -m all
+
+	@echo
+	@echo "### go version used for building:"
+	@grep -m 1 go go.mod
+
+goupdate:
+	go get -t -u=patch ./...

TODO.md

@@ -6,4 +6,13 @@
 
 - add --no-headers option
 
+### Lisp Plugin Infrastructure using zygo
 
+Hooks:
+
+| Filter    | Purpose                                                     | Args                | Return |
+|-----------|-------------------------------------------------------------|---------------------|--------|
+| filter    | include or exclude lines                                    | row as hash         | bool   |
+| process   | do calculations with data, store results in global lisp env | whole dataset       | nil    |
+| transpose | modify a cell                                               | headername and cell | cell   |
+| append    | add one or more rows to the dataset (use this to add stats) | nil                 | rows   |

cfg/config.go

@@ -19,19 +19,25 @@ package cfg
 import (
 	"errors"
 	"fmt"
-	"github.com/gookit/color"
 	"os"
 	"regexp"
+
+
+	"github.com/glycerine/zygomys/zygo"
+	"github.com/gookit/color"
 )
 
 const DefaultSeparator string = `(\s\s+|\t)`
-const Version string = "v1.0.13"
+const Version string = "v1.0.17"
+
+var DefaultLoadPath string = os.Getenv("HOME") + "/.config/tablizer/lisp"
 
 var VERSION string // maintained by -x
 
 type Config struct {
 	Debug          bool
 	NoNumbering    bool
+	NoHeaders      bool
 	Columns        string
 	UseColumns     []int
 	Separator      string
@@ -39,6 +45,7 @@ type Config struct {
 	InvertMatch    bool
 	Pattern        string
 	PatternR       *regexp.Regexp
+	UseFuzzySearch bool
 
 	SortMode       string
 	SortDescending bool
@@ -51,6 +58,13 @@ type Config struct {
 	ColorStyle color.Style
 
 	NoColor bool
+
+	// special  case: we use the  config struct to transport  the lisp
+	// env trough the program
+	Lisp *zygo.Zlisp
+
+	// a path containing lisp scripts to be loaded on startup
+	LispLoadPath string
 }
 
 // maps outputmode short flags to output mode, ie. -O => -o orgtbl
@@ -82,6 +96,9 @@ type Sortmode struct {
 	Age     bool
 }
 
+// valid lisp hooks
+var ValidHooks []string
+
 // default color schemes
 func Colors() map[color.Level]map[string]color.Color {
 	return map[color.Level]map[string]color.Color{
@@ -184,6 +201,8 @@ func (c *Config) ApplyDefaults() {
 	if c.OutputMode == Yaml || c.OutputMode == CSV {
 		c.NoNumbering = true
 	}
+
+	ValidHooks = []string{"filter", "process", "transpose", "append"}
 }
 
 func (c *Config) PreparePattern(pattern string) error {

cmd/root.go

@@ -20,13 +20,14 @@ import (
 	"bytes"
 	"errors"
 	"fmt"
-	"github.com/spf13/cobra"
-	"github.com/tlinden/tablizer/cfg"
-	"github.com/tlinden/tablizer/lib"
 	"log"
 	"os"
 	"os/exec"
 	"strings"
+
+	"github.com/spf13/cobra"
+	"github.com/tlinden/tablizer/cfg"
+	"github.com/tlinden/tablizer/lib"
 )
 
 func man() {
@@ -97,6 +98,12 @@ func Execute() {
 			conf.DetermineColormode()
 			conf.ApplyDefaults()
 
+			// setup lisp env, load plugins etc
+			err := lib.SetupLisp(&conf)
+			if err != nil {
+				return err
+			}
+
 			// actual execution starts here
 			return lib.ProcessFiles(&conf, args)
 		},
@@ -105,10 +112,12 @@ func Execute() {
 	// options
 	rootCmd.PersistentFlags().BoolVarP(&conf.Debug, "debug", "d", false, "Enable debugging")
 	rootCmd.PersistentFlags().BoolVarP(&conf.NoNumbering, "no-numbering", "n", false, "Disable header numbering")
+	rootCmd.PersistentFlags().BoolVarP(&conf.NoHeaders, "no-headers", "H", false, "Disable header display")
 	rootCmd.PersistentFlags().BoolVarP(&conf.NoColor, "no-color", "N", false, "Disable pattern highlighting")
 	rootCmd.PersistentFlags().BoolVarP(&ShowVersion, "version", "V", false, "Print program version")
 	rootCmd.PersistentFlags().BoolVarP(&conf.InvertMatch, "invert-match", "v", false, "select non-matching rows")
 	rootCmd.PersistentFlags().BoolVarP(&ShowManual, "man", "m", false, "Display manual page")
+	rootCmd.PersistentFlags().BoolVarP(&conf.UseFuzzySearch, "fuzzy", "z", false, "Use fuzzy searching")
 	rootCmd.PersistentFlags().StringVarP(&ShowCompletion, "completion", "", "", "Display completion code")
 	rootCmd.PersistentFlags().StringVarP(&conf.Separator, "separator", "s", cfg.DefaultSeparator, "Custom field separator")
 	rootCmd.PersistentFlags().StringVarP(&conf.Columns, "columns", "c", "", "Only show the speficied columns (separated by ,)")
@@ -121,7 +130,7 @@ func Execute() {
 	rootCmd.PersistentFlags().BoolVarP(&sortmode.Numeric, "sort-numeric", "i", false, "sort according to string numerical value")
 	rootCmd.PersistentFlags().BoolVarP(&sortmode.Time, "sort-time", "t", false, "sort according to time string")
 	rootCmd.PersistentFlags().BoolVarP(&sortmode.Age, "sort-age", "a", false, "sort according to age (duration) string")
-	rootCmd.MarkFlagsMutuallyExclusive("sort-desc", "sort-numeric", "sort-time", "sort-age")
+	rootCmd.MarkFlagsMutuallyExclusive("sort-numeric", "sort-time", "sort-age")
 
 	// output flags, only 1 allowed
 	rootCmd.PersistentFlags().BoolVarP(&modeflag.X, "extended", "X", false, "Enable extended output")
@@ -133,6 +142,9 @@ func Execute() {
 	rootCmd.PersistentFlags().BoolVarP(&modeflag.A, "ascii", "A", false, "Enable ASCII output (default)")
 	rootCmd.MarkFlagsMutuallyExclusive("extended", "markdown", "orgtbl", "shell", "yaml", "csv")
 
+	// lisp options
+	rootCmd.PersistentFlags().StringVarP(&conf.LispLoadPath, "load-path", "l", cfg.DefaultLoadPath, "Load path for lisp plugins (expects *.zy files)")
+
 	rootCmd.SetUsageTemplate(strings.TrimSpace(usage) + "\n")
 
 	err := rootCmd.Execute()

cmd/tablizer.go

@@ -13,8 +13,10 @@ SYNOPSIS
           -v, --invert-match       select non-matching rows
           -n, --no-numbering       Disable header numbering
           -N, --no-color           Disable pattern highlighting
+          -H, --no-headers         Disable headers display
           -s, --separator string   Custom field separator
           -k, --sort-by int        Sort by column (default: 1)
+          -z, --fuzzy              Use fuzzy seach [experimental]
 
         Output Flags (mutually exclusive):
           -X, --extended           Enable extended output
@@ -87,6 +89,11 @@ DESCRIPTION
 
     The numbering can be suppressed by using the -n option.
 
+    By default tablizer shows a header containing the names of each column.
+    This can be disabled using the -H option. Be aware that this only
+    affects tabular output modes. Shell, Extended, Yaml and CSV output modes
+    always use the column names.
+
     By default, if a pattern has been speficied, matches will be
     highlighted. You can disable this behavior with the -N option.
 
@@ -132,6 +139,10 @@ DESCRIPTION
 
         kubectl get pods -A | tablizer "(?i)account"
 
+    You can use the experimental fuzzy seach feature by providing the option
+    -z, in which case the pattern is regarded as a fuzzy search term, not a
+    regexp.
+
   COLUMNS
     The parameter -c can be used to specify, which columns to display. By
     default tablizer numerizes the header names and these numbers can be
@@ -255,9 +266,9 @@ LICENSE
     This software is licensed under the GNU GENERAL PUBLIC LICENSE version
     3.
 
-    Copyright (c) 2022 by Thomas von Dein
+    Copyright (c) 2023 by Thomas von Dein
 
-    This software uses the following GO libraries:
+    This software uses the following GO modules:
 
     repr (https://github.com/alecthomas/repr)
         Released under the MIT License, Copyright (c) 2016 Alec Thomas
@@ -266,6 +277,19 @@ LICENSE
         Released under the Apache 2.0 license, Copyright 2013-2022 The Cobra
         Authors
 
+    dateparse (github.com/araddon/dateparse)
+        Released under the MIT License, Copyright (c) 2015-2017 Aaron Raddon
+
+    color (github.com/gookit/color)
+        Released under the MIT License, Copyright (c) 2016 inhere
+
+    tablewriter (github.com/olekukonko/tablewriter)
+        Released under the MIT License, Copyright (c) 201 by Oleku Konko
+
+    yaml (gopkg.in/yaml.v3)
+        Released under the MIT License, Copyright (c) 2006-2011 Kirill
+        Simonov
+
 AUTHORS
     Thomas von Dein tom AT vondein DOT org
 
@@ -280,8 +304,10 @@ Operational Flags:
   -v, --invert-match       select non-matching rows
   -n, --no-numbering       Disable header numbering
   -N, --no-color           Disable pattern highlighting
+  -H, --no-headers         Disable headers display
   -s, --separator string   Custom field separator
   -k, --sort-by int        Sort by column (default: 1)
+  -z, --fuzzy              Use fuzzy seach [experimental]
 
 Output Flags (mutually exclusive):
   -X, --extended           Enable extended output
@@ -300,6 +326,8 @@ Sort Mode Flags (mutually exclusive):
 
 Other Flags:
       --completion <shell> Generate the autocompletion script for <shell>
+  -l  --load-path <path>   Where to search for lisp plugins. Maybe a file or
+                           a directory containing files with *.zy extension
   -d, --debug              Enable debugging
   -h, --help               help for tablizer
   -m, --man                Display manual page

go.mod

@@ -3,19 +3,36 @@ module github.com/tlinden/tablizer
 go 1.18
 
 require (
-	github.com/alecthomas/repr v0.0.0-20180818092828-117648cd9897
+	github.com/alecthomas/repr v0.1.1
 	github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de
 	github.com/gookit/color v1.5.2
 	github.com/olekukonko/tablewriter v0.0.5
-	github.com/spf13/cobra v1.5.0
+	github.com/spf13/cobra v1.6.1
 	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
-	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/glycerine/blake2b v0.0.0-20151022103502-3c8c640cd7be // indirect
-	github.com/mattn/go-runewidth v0.0.10 // indirect
+	github.com/glycerine/greenpack v5.1.1+incompatible // indirect
-	github.com/rivo/uniseg v0.1.0 // indirect
+	github.com/glycerine/liner v0.0.0-20160121172638-72909af234e0 // indirect
-	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/glycerine/zygomys v5.1.2+incompatible // indirect
-	github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 // indirect
+	github.com/lithammer/fuzzysearch v1.1.7 // indirect
-	golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44 // indirect
+	github.com/philhofer/fwd v1.1.2 // indirect
+	github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636 // indirect
+	github.com/shurcooL/go-goon v1.0.0 // indirect
+	github.com/tinylib/msgp v1.1.8 // indirect
+	github.com/ugorji/go/codec v1.2.11 // indirect
+	golang.org/x/text v0.8.0 // indirect
+)
+
+require (
+	github.com/inconshreveable/mousetrap v1.0.1 // indirect
+	github.com/mattn/go-runewidth v0.0.14 // indirect
+
+	// force release. > 0.4. doesnt build everywhere, see:
+	// https://github.com/TLINDEN/tablizer/actions/runs/3396457307/jobs/5647544615
+	github.com/rivo/uniseg v0.2.0 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
+	golang.org/x/sys v0.5.0 // indirect
 )

go.sum

@@ -1,28 +1,46 @@
-github.com/alecthomas/repr v0.0.0-20180818092828-117648cd9897 h1:p9Sln00KOTlrYkxI1zYWl1QLnEqAqEARBEYa8FQnQcY=
+github.com/alecthomas/repr v0.1.1 h1:87P60cSmareLAxMc4Hro0r2RBY4ROm0dYwkJNpS4pPs=
-github.com/alecthomas/repr v0.0.0-20180818092828-117648cd9897/go.mod h1:xTS7Pm1pD1mvyM075QCDSRqH6qRLXylzS24ZTpRiSzQ=
+github.com/alecthomas/repr v0.1.1/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
 github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de h1:FxWPpzIjnTlhPwqqXc4/vE0f7GvRjuAsbW+HOIe8KnA=
 github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de/go.mod h1:DCaWoUhZrYW9p1lxo/cm8EmUOOzAPSEZNGF2DK1dJgw=
 github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/glycerine/blake2b v0.0.0-20151022103502-3c8c640cd7be h1:XBJdPGgA3qqhW+p9CANCAVdF7ZIXdu3pZAkypMkKAjE=
+github.com/glycerine/blake2b v0.0.0-20151022103502-3c8c640cd7be/go.mod h1:OSCrScrFAjcBObrulk6BEQlytA462OkG1UGB5NYj9kE=
+github.com/glycerine/greenpack v5.1.1+incompatible h1:fDr9i6MkSGZmAy4VXPfJhW+SyK2/LNnzIp5nHyDiaIM=
+github.com/glycerine/greenpack v5.1.1+incompatible/go.mod h1:us0jVISAESGjsEuLlAfCd5nkZm6W6WQF18HPuOecIg4=
+github.com/glycerine/liner v0.0.0-20160121172638-72909af234e0 h1:4ZegphJXBTc4uFQ08UVoWYmQXorGa+ipXetUj83sMBc=
+github.com/glycerine/liner v0.0.0-20160121172638-72909af234e0/go.mod h1:AqJLs6UeoC65dnHxyCQ6MO31P5STpjcmgaANAU+No8Q=
+github.com/glycerine/zygomys v5.1.2+incompatible h1:jmcdmA3XPxgfOunAXFpipE9LQoUL6eX6d2mhYyjV4GE=
+github.com/glycerine/zygomys v5.1.2+incompatible/go.mod h1:i3SPKZpmy9dwF/3iWrXJ/ZLyzZucegwypwOmqRkUUaQ=
 github.com/gookit/color v1.5.2 h1:uLnfXcaFjlrDnQDT+NCBcfhrXqYTx/rcCa6xn01Y8yI=
 github.com/gookit/color v1.5.2/go.mod h1:w8h4bGiHeeBpvQVePTutdbERIUf3oJE5lZ8HM0UgXyg=
-github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
+github.com/inconshreveable/mousetrap v1.0.1 h1:U3uMjPSQEBMNp1lFxmllqCPM6P5u/Xq7Pgzkat/bFNc=
-github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/lithammer/fuzzysearch v1.1.7 h1:q8rZNmBIUkqxsxb/IlwsXVbCoPIH/0juxjFHY0UIwhU=
+github.com/lithammer/fuzzysearch v1.1.7/go.mod h1:ZhIlfRGxnD8qa9car/yplC6GmnM14CS07BYAKJJBK2I=
 github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
-github.com/mattn/go-runewidth v0.0.10 h1:CoZ3S2P7pvtP45xOtBw+/mDL2z0RKI576gSkzRRpdGg=
 github.com/mattn/go-runewidth v0.0.10/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
+github.com/mattn/go-runewidth v0.0.14 h1:+xnbZSEeDbOIg5/mE6JF0w6n9duR1l3/WmbinWVwUuU=
+github.com/mattn/go-runewidth v0.0.14/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
 github.com/olekukonko/tablewriter v0.0.5 h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec=
 github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY=
+github.com/philhofer/fwd v1.1.2 h1:bnDivRJ1EWPjUIRXV5KfORO897HTbpFAQddBdE8t7Gw=
+github.com/philhofer/fwd v1.1.2/go.mod h1:qkPdfjR2SIEbspLqpe1tO4n5yICnr2DY7mqEx2tUTP0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/rivo/uniseg v0.1.0 h1:+2KBaVoUmb9XzDsrx/Ct0W/EYOSFf/nWTauy++DprtY=
 github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
+github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
+github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4/go.mod h1:C1a7PQSMz9NShzorzCiG2fk9+xuCgLkPeCvMHYR2OWg=
-github.com/spf13/cobra v1.5.0 h1:X+jTBEBqF0bHN+9cSMgmfuvv2VHJ9ezmFNf9Y/XstYU=
+github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636 h1:aSISeOcal5irEhJd1M+IrApc0PdcN7e7Aj4yuEnOrfQ=
-github.com/spf13/cobra v1.5.0/go.mod h1:dWXEIy2H428czQCjInthrTRUg7yKbok+2Qi/yBIJoUM=
+github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk=
+github.com/shurcooL/go-goon v1.0.0 h1:BCQPvxGkHHJ4WpBO4m/9FXbITVIsvAm/T66cCcCGI7E=
+github.com/shurcooL/go-goon v1.0.0/go.mod h1:2wTHMsGo7qnpmqA8ADYZtP4I1DD94JpXGQ3Dxq2YQ5w=
+github.com/spf13/cobra v1.6.1 h1:o94oiPyS4KD1mPy2fmcYYHHfCxLqYjJOhGsCHFZtEzA=
+github.com/spf13/cobra v1.6.1/go.mod h1:IOw/AERYS7UzyrGinqmz6HLUo219MORXGxhbaJUqzrY=
 github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -31,13 +49,58 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
-github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 h1:QldyIu/L63oPpyvQmHgvgickp1Yw510KJOqX7H24mg8=
+github.com/tinylib/msgp v1.1.8 h1:FCXC1xanKO4I8plpHGH2P7koL/RzZs12l/+r7vakfm0=
+github.com/tinylib/msgp v1.1.8/go.mod h1:qkpG+2ldGg4xRFmx+jfTvZPxfGFhi64BcnL9vkCm/Tw=
+github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU=
+github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
 github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778/go.mod h1:2MuV+tbUrU1zIOPMxZ5EncGwgmMJsa+9ucAQZXxsObs=
-golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44 h1:Bli41pIlzTzf3KEY06n+xnzK/BESIg2ze4Pgfh/aI8c=
+github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
+github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
+github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
+golang.org/x/exp v0.0.0-20220909182711-5c715a9e8561 h1:MDc5xs78ZrZr3HMQugiXOAkSZtfTpbJLDr/lwfgO53E=
+golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
+golang.org/x/mod v0.7.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
+golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/net v0.3.0/go.mod h1:MBQ8lrhLObU/6UmLb4fmbmk5OcyYmqtbGd/9yIeKjEE=
+golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.1.0 h1:kunALQeHf1/185U1i0GOB/fy1IPRDDpuoOOqRReG57U=
+golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU=
+golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA=
+golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/text v0.5.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
+golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
+golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=
+golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
+golang.org/x/tools v0.4.0/go.mod h1:UE5sM2OK9E/d67R0ANs2xJizIymRP5gJU295PvKXxjQ=
+golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

lib/io.go

@@ -55,10 +55,26 @@ func ProcessFiles(c *cfg.Config, args []string) error {
 func determineIO(c *cfg.Config, args []string) ([]io.Reader, string, error) {
 	var pattern string
 	var fds []io.Reader
-	var havefiles bool
+	var haveio bool
 
+	stat, _ := os.Stdin.Stat()
+	if (stat.Mode() & os.ModeCharDevice) == 0 {
+		// we're reading from STDIN, which takes precedence over file args
+		fds = append(fds, os.Stdin)
+		if len(args) > 0 {
+			// ignore any args > 1
+			pattern = args[0]
+			c.Pattern = args[0] // used for colorization by printData()
+		}
+		haveio = true
+	} else {
 		if len(args) > 0 {
 			// threre were args left, take a look
+			if args[0] == "-" {
+				// in traditional unix programs a dash denotes STDIN (forced)
+				fds = append(fds, os.Stdin)
+				haveio = true
+			} else {
 				if _, err := os.Stat(args[0]); err != nil {
 					// first  one is  not a  file, consider  it as  regexp and
 					// shift arg list
@@ -68,8 +84,9 @@ func determineIO(c *cfg.Config, args []string) ([]io.Reader, string, error) {
 				}
 
 				if len(args) > 0 {
-			// only files
+					// consider any other args as files
 					for _, file := range args {
+
 						fd, err := os.OpenFile(file, os.O_RDONLY, 0755)
 
 						if err != nil {
@@ -77,19 +94,16 @@ func determineIO(c *cfg.Config, args []string) ([]io.Reader, string, error) {
 						}
 
 						fds = append(fds, fd)
+						haveio = true
+					}
+				}
 			}
-			havefiles = true
 		}
 	}
 
-	if !havefiles {
+	if !haveio {
-		stat, _ := os.Stdin.Stat()
-		if (stat.Mode() & os.ModeCharDevice) == 0 {
-			fds = append(fds, os.Stdin)
-		} else {
 		return nil, "", errors.New("No file specified and nothing to read on stdin!")
 	}
-	}
 
 	return fds, pattern, nil
 }

lib/lisp.go

@@ -0,0 +1,293 @@
+/*
+Copyright © 2023 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package lib
+
+import (
+	"errors"
+	"fmt"
+	"log"
+	"os"
+	"strings"
+
+	"github.com/glycerine/zygomys/zygo"
+	"github.com/tlinden/tablizer/cfg"
+)
+
+/*
+needs to be global because we can't feed an cfg object to AddHook()
+which is being called from user lisp code
+*/
+var Hooks map[string][]*zygo.SexpSymbol
+
+/*
+AddHook() (called addhook from lisp code)  can be used by the user to
+add a function to one of the available hooks provided by tablizer.
+*/
+func AddHook(env *zygo.Zlisp, name string, args []zygo.Sexp) (zygo.Sexp, error) {
+	var hookname string
+
+	if len(args) < 2 {
+		return zygo.SexpNull, errors.New("argument of %add-hook should be: %hook-name %your-function")
+	}
+
+	switch t := args[0].(type) {
+	case *zygo.SexpSymbol:
+		if !HookExists(t.Name()) {
+			return zygo.SexpNull, errors.New("Unknown hook " + t.Name())
+		}
+		hookname = t.Name()
+	default:
+		return zygo.SexpNull, errors.New("hook name must be a symbol!")
+	}
+
+	switch t := args[1].(type) {
+	case *zygo.SexpSymbol:
+		_, exists := Hooks[hookname]
+		if !exists {
+			Hooks[hookname] = []*zygo.SexpSymbol{t}
+		} else {
+			Hooks[hookname] = append(Hooks[hookname], t)
+		}
+	default:
+		return zygo.SexpNull, errors.New("hook function must be a symbol!")
+	}
+
+	return zygo.SexpNull, nil
+}
+
+/*
+Check if a hook exists
+*/
+func HookExists(key string) bool {
+	for _, hook := range cfg.ValidHooks {
+		if hook == key {
+			return true
+		}
+	}
+
+	return false
+}
+
+/*
+ * Basic sanity checks and load lisp file
+ */
+func LoadFile(env *zygo.Zlisp, path string) error {
+	if strings.HasSuffix(path, `.zy`) {
+		code, err := os.ReadFile(path)
+		if err != nil {
+			return err
+		}
+
+		// FIXME: check what res (_ here) could be and mean
+		_, err = env.EvalString(string(code))
+		if err != nil {
+			log.Fatalf(env.GetStackTrace(err))
+		}
+	}
+
+	return nil
+}
+
+/*
+ * Setup lisp interpreter environment
+ */
+func SetupLisp(c *cfg.Config) error {
+	Hooks = make(map[string][]*zygo.SexpSymbol)
+
+	env := zygo.NewZlispSandbox()
+	env.AddFunction("addhook", AddHook)
+
+	// iterate over load-path and evaluate all *.zy files there, if any
+	// we ignore if load-path does not exist, which is the default anyway
+	if path, err := os.Stat(c.LispLoadPath); !os.IsNotExist(err) {
+		if !path.IsDir() {
+			err := LoadFile(env, c.LispLoadPath)
+			if err != nil {
+				return err
+			}
+		} else {
+			dir, err := os.ReadDir(c.LispLoadPath)
+			if err != nil {
+				return err
+			}
+
+			for _, entry := range dir {
+				if !entry.IsDir() {
+					err := LoadFile(env, c.LispLoadPath+"/"+entry.Name())
+					if err != nil {
+						return err
+					}
+				}
+			}
+		}
+	}
+
+	RegisterLib(env)
+
+	c.Lisp = env
+	return nil
+}
+
+/*
+Execute every user lisp function registered as filter hook.
+
+Each function is given the current line as argument and is expected to
+return a boolean.   True indicates to keep the line,  false to skip
+it.
+
+If there  are multiple such  functions registered, then the  first one
+returning false wins,  that is if each function returns  true the line
+will  be kept,  if at  least one  of them  returns false,  it will  be
+skipped.
+*/
+func RunFilterHooks(c cfg.Config, line string) (bool, error) {
+	for _, hook := range Hooks["filter"] {
+		var result bool
+		c.Lisp.Clear()
+		res, err := c.Lisp.EvalString(fmt.Sprintf("(%s `%s`)", hook.Name(), line))
+		if err != nil {
+			return false, err
+		}
+
+		switch t := res.(type) {
+		case *zygo.SexpBool:
+			result = t.Val
+		default:
+			return false, errors.New("filter hook shall return BOOL!")
+		}
+
+		if !result {
+			// the first hook which returns false leads to complete false
+			return result, nil
+		}
+	}
+
+	// if no hook returned false, we succeed and accept the given line
+	return true, nil
+}
+
+/*
+These hooks get the data (Tabdata) readily processed by tablizer as
+argument. They are expected to  return a SexpPair containing a boolean
+denoting  if  the data  has  been  modified  and the  actual  modified
+data. Columns must  be the same, rows may differ.  Cells may also have
+been modified.
+
+Replaces the  internal data structure  Tabdata with the  user supplied
+version.
+
+Only one process hook function is supported.
+
+The somewhat complicated code is being  caused by the fact, that we
+need to convert our internal structure  to a lisp variable and vice
+versa afterwards.
+*/
+func RunProcessHooks(c cfg.Config, data Tabdata) (Tabdata, bool, error) {
+	var userdata Tabdata
+	lisplist := []zygo.Sexp{}
+
+	if len(Hooks["process"]) == 0 {
+		return userdata, false, nil
+	}
+
+	if len(Hooks["process"]) > 1 {
+		fmt.Println("Warning: only one process hook is allowed!")
+	}
+
+	// there are hook[s] installed, convert the go data structure 'data to lisp
+	for _, row := range data.entries {
+		var entry zygo.SexpHash
+
+		for idx, cell := range row {
+			err := entry.HashSet(&zygo.SexpStr{S: data.headers[idx]}, &zygo.SexpStr{S: cell})
+			if err != nil {
+				return userdata, false, err
+			}
+		}
+
+		lisplist = append(lisplist, &entry)
+	}
+
+	// we need to add it to the env so that the function can use the struct directly
+	c.Lisp.AddGlobal("data", &zygo.SexpArray{Val: lisplist, Env: c.Lisp})
+
+	// execute the actual hook
+	hook := Hooks["process"][0]
+	var result bool
+	c.Lisp.Clear()
+
+	res, err := c.Lisp.EvalString(fmt.Sprintf("(%s data)", hook.Name()))
+	if err != nil {
+		return userdata, false, err
+	}
+
+	// we expect (bool, array(hash)) as return from the function
+	switch t := res.(type) {
+	case *zygo.SexpPair:
+		switch th := t.Head.(type) {
+		case *zygo.SexpBool:
+			result = th.Val
+		default:
+			return userdata, false, errors.New("Expect (bool, array(hash)) as return value!")
+		}
+
+		switch tt := t.Tail.(type) {
+		case *zygo.SexpArray:
+			lisplist = tt.Val
+		default:
+			return userdata, false, errors.New("Expect (bool, array(hash)) as return value!")
+		}
+	default:
+		return userdata, false, errors.New("filter hook shall return array of hashes!")
+	}
+
+	if !result {
+		// no further processing required
+		return userdata, result, nil
+	}
+
+	// finally convert lispdata back to Tabdata
+	for _, item := range lisplist {
+		row := []string{}
+
+		switch hash := item.(type) {
+		case *zygo.SexpHash:
+			for _, header := range data.headers {
+				entry, err := hash.HashGetDefault(c.Lisp, &zygo.SexpStr{S: header}, &zygo.SexpStr{S: ""})
+				if err != nil {
+					return userdata, false, err
+				}
+
+				switch t := entry.(type) {
+				case *zygo.SexpStr:
+					row = append(row, t.S)
+				default:
+					return userdata, false, errors.New("Hash values should be string!")
+				}
+			}
+		default:
+			return userdata, false, errors.New("Returned array should contain hashes!")
+		}
+
+		userdata.entries = append(userdata.entries, row)
+	}
+
+	userdata.headers = data.headers
+
+	return userdata, result, nil
+}

lib/lisplib.go

@@ -0,0 +1,84 @@
+/*
+Copyright © 2023 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package lib
+
+import (
+	"errors"
+	"regexp"
+	"strconv"
+
+	"github.com/glycerine/zygomys/zygo"
+)
+
+func Splice2SexpList(list []string) zygo.Sexp {
+	slist := []zygo.Sexp{}
+
+	for _, item := range list {
+		slist = append(slist, &zygo.SexpStr{S: item})
+	}
+	return zygo.MakeList(slist)
+}
+
+func StringReSplit(env *zygo.Zlisp, name string, args []zygo.Sexp) (zygo.Sexp, error) {
+	if len(args) < 2 {
+		return zygo.SexpNull, errors.New("expecting 2 arguments!")
+	}
+
+	var separator string
+	var input string
+
+	switch t := args[0].(type) {
+	case *zygo.SexpStr:
+		input = t.S
+	default:
+		return zygo.SexpNull, errors.New("second argument must be a string!")
+	}
+
+	switch t := args[1].(type) {
+	case *zygo.SexpStr:
+		separator = t.S
+	default:
+		return zygo.SexpNull, errors.New("first argument must be a string!")
+	}
+
+	sep := regexp.MustCompile(separator)
+
+	return Splice2SexpList(sep.Split(input, -1)), nil
+}
+
+func String2Int(env *zygo.Zlisp, name string, args []zygo.Sexp) (zygo.Sexp, error) {
+	var number int
+
+	switch t := args[0].(type) {
+	case *zygo.SexpStr:
+		num, err := strconv.Atoi(t.S)
+		if err != nil {
+			return zygo.SexpNull, err
+		}
+		number = num
+	default:
+		return zygo.SexpNull, errors.New("argument must be a string!")
+	}
+
+	return &zygo.SexpInt{Val: int64(number)}, nil
+}
+
+func RegisterLib(env *zygo.Zlisp) {
+	env.AddFunction("resplit", StringReSplit)
+	env.AddFunction("atoi", String2Int)
+}

lib/parser.go

@@ -22,13 +22,31 @@ import (
 	"encoding/csv"
 	"errors"
 	"fmt"
-	"github.com/alecthomas/repr"
-	"github.com/tlinden/tablizer/cfg"
 	"io"
 	"regexp"
 	"strings"
+
+	"github.com/alecthomas/repr"
+	"github.com/lithammer/fuzzysearch/fuzzy"
+	"github.com/tlinden/tablizer/cfg"
 )
 
+/*
+ * [!]Match a  line, use fuzzy  search for normal pattern  strings and
+ * regexp otherwise.
+ */
+func matchPattern(c cfg.Config, line string) bool {
+	if len(c.Pattern) > 0 {
+		if c.UseFuzzySearch {
+			return fuzzy.MatchFold(c.Pattern, line)
+		} else {
+			return c.PatternR.MatchString(line)
+		}
+	}
+
+	return true
+}
+
 /*
 Parser switch
 */
@@ -55,13 +73,25 @@ func parseCSV(c cfg.Config, input io.Reader) (Tabdata, error) {
 			line := strings.TrimSpace(scanner.Text())
 			if hadFirst {
 				// don't match 1st line, it's the header
-				if c.PatternR.MatchString(line) == c.InvertMatch {
+				if matchPattern(c, line) == c.InvertMatch {
 					// by default  -v is false, so if a  line does NOT
 					// match the pattern, we will ignore it. However,
 					// if the user specified -v, the matching is inverted,
 					// so we ignore all lines, which DO match.
 					continue
 				}
+
+				// apply user defined lisp filters, if any
+				accept, err := RunFilterHooks(c, line)
+				if err != nil {
+					return data, errors.Unwrap(fmt.Errorf("Failed to apply filter hook: %w", err))
+				}
+
+				if !accept {
+					//  IF there  are filter  hook[s] and  IF one  of them
+					// returns false on the current line, reject it
+					continue
+				}
 			}
 			lines = append(lines, line)
 			hadFirst = true
@@ -94,6 +124,15 @@ func parseCSV(c cfg.Config, input io.Reader) (Tabdata, error) {
 		}
 	}
 
+	// apply user defined lisp process hooks, if any
+	userdata, changed, err := RunProcessHooks(c, data)
+	if err != nil {
+		return data, errors.Unwrap(fmt.Errorf("Failed to apply filter hook: %w", err))
+	}
+	if changed {
+		data = userdata
+	}
+
 	return data, nil
 }
 
@@ -141,14 +180,24 @@ func parseTabular(c cfg.Config, input io.Reader) (Tabdata, error) {
 			}
 		} else {
 			// data processing
-			if len(c.Pattern) > 0 {
+			if matchPattern(c, line) == c.InvertMatch {
-				if c.PatternR.MatchString(line) == c.InvertMatch {
 				// by default  -v is false, so if a  line does NOT
 				// match the pattern, we will ignore it. However,
 				// if the user specified -v, the matching is inverted,
 				// so we ignore all lines, which DO match.
 				continue
 			}
+
+			// apply user defined lisp filters, if any
+			accept, err := RunFilterHooks(c, line)
+			if err != nil {
+				return data, errors.Unwrap(fmt.Errorf("Failed to apply filter hook: %w", err))
+			}
+
+			if !accept {
+				//  IF there  are filter  hook[s] and  IF one  of them
+				// returns false on the current line, reject it
+				continue
 			}
 
 			idx := 0 // we cannot use the header index, because we could exclude columns
@@ -175,6 +224,15 @@ func parseTabular(c cfg.Config, input io.Reader) (Tabdata, error) {
 		return data, errors.Unwrap(fmt.Errorf("Failed to read from io.Reader: %w", scanner.Err()))
 	}
 
+	// apply user defined lisp process hooks, if any
+	userdata, changed, err := RunProcessHooks(c, data)
+	if err != nil {
+		return data, errors.Unwrap(fmt.Errorf("Failed to apply filter hook: %w", err))
+	}
+	if changed {
+		data = userdata
+	}
+
 	if c.Debug {
 		repr.Print(data)
 	}

lib/parser_test.go

@@ -19,10 +19,11 @@ package lib
 
 import (
 	"fmt"
-	"github.com/tlinden/tablizer/cfg"
 	"reflect"
 	"strings"
 	"testing"
+
+	"github.com/tlinden/tablizer/cfg"
 )
 
 var input = []struct {

lib/printer.go

@@ -75,7 +75,9 @@ func printOrgmodeData(w io.Writer, c cfg.Config, data *Tabdata) {
 	tableString := &strings.Builder{}
 	table := tablewriter.NewWriter(tableString)
 
+	if !c.NoHeaders {
 		table.SetHeader(data.headers)
+	}
 
 	for _, row := range data.entries {
 		table.Append(trimRow(row))
@@ -110,7 +112,9 @@ func printMarkdownData(w io.Writer, c cfg.Config, data *Tabdata) {
 	tableString := &strings.Builder{}
 	table := tablewriter.NewWriter(tableString)
 
+	if !c.NoHeaders {
 		table.SetHeader(data.headers)
+	}
 
 	for _, row := range data.entries {
 		table.Append(trimRow(row))
@@ -130,13 +134,11 @@ func printAsciiData(w io.Writer, c cfg.Config, data *Tabdata) {
 	tableString := &strings.Builder{}
 	table := tablewriter.NewWriter(tableString)
 
+	if !c.NoHeaders {
 		table.SetHeader(data.headers)
+	}
 	table.AppendBulk(data.entries)
 
-	// for _, row := range data.entries {
-	// 	table.Append(trimRow(row))
-	// }
-
 	table.SetAutoWrapText(false)
 	table.SetAutoFormatHeaders(true)
 	table.SetHeaderAlignment(tablewriter.ALIGN_LEFT)

mkrel.sh

@@ -43,7 +43,7 @@ for D in $DIST; do
     tardir="${tool}-${os}-${arch}-${version}"
     tarfile="releases/${tool}-${os}-${arch}-${version}.tar.gz"
     set -x
-    GOOS=${os} GOARCH=${arch} go build -o ${binfile} -ldflags "-X 'github.com/tlinden/tablizer/lib.VERSION=${version}'"
+    GOOS=${os} GOARCH=${arch} go build -o ${binfile} -ldflags "-X 'github.com/tlinden/tablizer/cfg.VERSION=${version}'"
     mkdir -p ${tardir}
     cp ${binfile} README.md LICENSE ${tardir}/
     echo 'tool = tablizer

t/plugintest.zy

@@ -0,0 +1,10 @@
+/*
+Simple filter hook function. Splits the argument by whitespace,
+fetches the 2nd element, converts it to an int and returns true
+if it s larger than 5, false otherwise.
+*/
+(defn uselarge [line]
+  (cond (> (atoi (second (resplit line `\s+`))) 5) true false))
+
+/* Register the filter hook */
+(addhook %filter %uselarge)

t/test.sh

@@ -0,0 +1,45 @@
+#!/bin/sh
+
+# simple commandline unit test script
+
+t="../tablizer"
+fail=0
+
+ex() {
+    # execute a test, report+exit on error, stay silent otherwise
+    log="/tmp/test-tablizer.$$.log"
+    name=$1
+    shift
+
+    echo -n "TEST $name "
+
+    $* > $log 2>&1
+
+    if test $? -ne 0; then
+        echo "failed, see $log"
+        fail=1
+    else
+        echo "ok"
+        rm -f $log
+    fi
+}
+
+# only use files in test dir
+cd $(dirname $0)
+
+echo "Executing commandline tests ..."
+
+# io pattern tests
+ex io-pattern-and-file $t bk7 testtable
+cat testtable | ex io-pattern-and-stdin $t bk7
+cat testtable | ex io-pattern-and-stdin-dash $t bk7 -
+
+# same w/o pattern
+ex io-just-file $t testtable
+cat testtable | ex io-just-stdin $t
+cat testtable | ex io-just-stdin-dash $t -
+
+if test $fail -ne 0; then
+    echo "!!! Some tests failed !!!"
+    exit 1
+fi

t/testtable

@@ -0,0 +1,6 @@
+NAME                                                 READY   STATUS    RESTARTS       AGE
+alertmanager-kube-prometheus-alertmanager-0          2/2     Running   35 (45m ago)   11d
+grafana-fcc54cbc9-bk7s8                              1/1     Running   17 (45m ago)   1d
+kube-prometheus-blackbox-exporter-5d85b5d8f4-tskh7   1/1     Running   17 (45m ago)   1h44m
+kube-prometheus-kube-state-metrics-b4cd9487-75p7f    1/1     Running   20 (45m ago)   45m
+kube-prometheus-node-exporter-bfzpl                  1/1     Running   17 (45m ago)   54s

t/testtable2

@@ -0,0 +1,6 @@
+NAME  DURATION
+x     10
+a     100
+z     0
+u     4
+k     6

tablizer.1

@@ -133,7 +133,7 @@
 .\" ========================================================================
 .\"
 .IX Title "TABLIZER 1"
-.TH TABLIZER 1 "2022-11-01" "1" "User Commands"
+.TH TABLIZER 1 "2023-05-06" "1" "User Commands"
 .\" For nroff, turn off justification.  Always turn off hyphenation; it makes
 .\" way too many mistakes in technical documents.
 .if n .ad l
@@ -151,8 +151,10 @@ tablizer \- Manipulate tabular output of other programs
 \&      \-v, \-\-invert\-match       select non\-matching rows
 \&      \-n, \-\-no\-numbering       Disable header numbering
 \&      \-N, \-\-no\-color           Disable pattern highlighting
+\&      \-H, \-\-no\-headers         Disable headers display
 \&      \-s, \-\-separator string   Custom field separator
 \&      \-k, \-\-sort\-by int        Sort by column (default: 1)
+\&      \-z, \-\-fuzzy              Use fuzzy seach [experimental]
 \&
 \&    Output Flags (mutually exclusive):
 \&      \-X, \-\-extended           Enable extended output
@@ -234,6 +236,11 @@ the original order.
 .PP
 The numbering can be suppressed by using the \fB\-n\fR option.
 .PP
+By  default tablizer  shows  a  header containing  the  names of  each
+column.  This can  be disabled using the \fB\-H\fR option.   Be aware that
+this only affects tabular output modes.  Shell, Extended, Yaml and \s-1CSV\s0
+output modes always use the column names.
+.PP
 By  default, if  a  \fBpattern\fR  has been  speficied,  matches will  be
 highlighted. You can disable this behavior with the \fB\-N\fR option.
 .PP
@@ -287,6 +294,10 @@ Example for a case insensitive search:
 .Vb 1
 \&    kubectl get pods \-A | tablizer "(?i)account"
 .Ve
+.PP
+You  can use  the experimental  fuzzy seach  feature by  providing the
+option \fB\-z\fR, in which case the  pattern is regarded as a fuzzy search
+term, not a regexp.
 .SS "\s-1COLUMNS\s0"
 .IX Subsection "COLUMNS"
 The  parameter  \fB\-c\fR  can  be  used  to  specify,  which  columns  to
@@ -442,15 +453,27 @@ or to submit a patch, please open an issue on github:
 .IX Header "LICENSE"
 This software is licensed under the \s-1GNU GENERAL PUBLIC LICENSE\s0 version 3.
 .PP
-Copyright (c) 2022 by Thomas von Dein
+Copyright (c) 2023 by Thomas von Dein
 .PP
-This software uses the following \s-1GO\s0 libraries:
+This software uses the following \s-1GO\s0 modules:
 .IP "repr (https://github.com/alecthomas/repr)" 4
 .IX Item "repr (https://github.com/alecthomas/repr)"
 Released under the \s-1MIT\s0 License, Copyright (c) 2016 Alec Thomas
 .IP "cobra (https://github.com/spf13/cobra)" 4
 .IX Item "cobra (https://github.com/spf13/cobra)"
 Released under the Apache 2.0 license, Copyright 2013\-2022 The Cobra Authors
+.IP "dateparse (github.com/araddon/dateparse)" 4
+.IX Item "dateparse (github.com/araddon/dateparse)"
+Released under the \s-1MIT\s0 License, Copyright (c) 2015\-2017 Aaron Raddon
+.IP "color (github.com/gookit/color)" 4
+.IX Item "color (github.com/gookit/color)"
+Released under the \s-1MIT\s0 License, Copyright (c) 2016 inhere
+.IP "tablewriter (github.com/olekukonko/tablewriter)" 4
+.IX Item "tablewriter (github.com/olekukonko/tablewriter)"
+Released under the \s-1MIT\s0 License, Copyright (c) 201 by Oleku Konko
+.IP "yaml (gopkg.in/yaml.v3)" 4
+.IX Item "yaml (gopkg.in/yaml.v3)"
+Released under the \s-1MIT\s0 License, Copyright (c) 2006\-2011 Kirill Simonov
 .SH "AUTHORS"
 .IX Header "AUTHORS"
 Thomas von Dein \fBtom \s-1AT\s0 vondein \s-1DOT\s0 org\fR

tablizer.pod

@@ -12,8 +12,10 @@ tablizer - Manipulate tabular output of other programs
       -v, --invert-match       select non-matching rows
       -n, --no-numbering       Disable header numbering
       -N, --no-color           Disable pattern highlighting
+      -H, --no-headers         Disable headers display
       -s, --separator string   Custom field separator
       -k, --sort-by int        Sort by column (default: 1)
+      -z, --fuzzy              Use fuzzy seach [experimental]
 
     Output Flags (mutually exclusive):
       -X, --extended           Enable extended output
@@ -90,6 +92,11 @@ the original order.
 
 The numbering can be suppressed by using the B<-n> option.
 
+By  default tablizer  shows  a  header containing  the  names of  each
+column.  This can  be disabled using the B<-H> option.   Be aware that
+this only affects tabular output modes.  Shell, Extended, Yaml and CSV
+output modes always use the column names.
+
 By  default, if  a  B<pattern>  has been  speficied,  matches will  be
 highlighted. You can disable this behavior with the B<-N> option.
 
@@ -146,6 +153,9 @@ Example for a case insensitive search:
 
     kubectl get pods -A | tablizer "(?i)account"
 
+You  can use  the experimental  fuzzy seach  feature by  providing the
+option B<-z>, in which case the  pattern is regarded as a fuzzy search
+term, not a regexp.
 
 =head2 COLUMNS
 
@@ -293,9 +303,9 @@ L<https://github.com/TLINDEN/tablizer/issues>.
 
 This software is licensed under the GNU GENERAL PUBLIC LICENSE version 3.
 
-Copyright (c) 2022 by Thomas von Dein
+Copyright (c) 2023 by Thomas von Dein
 
-This software uses the following GO libraries:
+This software uses the following GO modules:
 
 =over 4
 
@@ -307,6 +317,22 @@ Released under the MIT License, Copyright (c) 2016 Alec Thomas
 
 Released under the Apache 2.0 license, Copyright 2013-2022 The Cobra Authors
 
+=item dateparse (github.com/araddon/dateparse)
+
+Released under the MIT License, Copyright (c) 2015-2017 Aaron Raddon
+
+=item color (github.com/gookit/color)
+
+Released under the MIT License, Copyright (c) 2016 inhere
+
+=item tablewriter (github.com/olekukonko/tablewriter)
+
+Released under the MIT License, Copyright (c) 201 by Oleku Konko
+
+=item yaml (gopkg.in/yaml.v3)
+
+Released under the MIT License, Copyright (c) 2006-2011 Kirill Simonov
+
 =back
 
 =head1 AUTHORS