fix #27: check if parsed headers and columns match

Bumps [github.com/alecthomas/repr](https://github.com/alecthomas/repr) from 0.1.1 to 0.4.0.
- [Commits](https://github.com/alecthomas/repr/compare/v0.1.1...v0.4.0)

---
updated-dependencies:
- dependency-name: github.com/alecthomas/repr
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,31 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[bug-report]"
+labels: bug
+assignees: TLINDEN
+
+---
+
+**Description**
+<!-- Please provide a clear and concise description of the issue: -->
+
+
+**Steps To Reproduce**
+<!-- Please detail the steps to reproduce the behavior: -->
+
+
+**Expected behavior**
+<!-- What do you expected to happen instead? -->
+
+
+**Version information**
+<!--
+Please provide as much version information as possible:
+- if you have just installed a binary, provide the output of: tablizer --version
+- if you installed from source, provide the output of: make show-version
+- provide additional details: operating system and version and shell environment
+-->
+
+
+**Additional informations**

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,23 @@
+---
+name: Feature request
+about: Suggest a feature
+title: "[feature-request]"
+labels: feature-request
+assignees: TLINDEN
+
+---
+
+**Description**
+<!-- Please provide a clear and concise description of the feature you desire: -->
+
+
+
+**Version information**
+<!--
+Just in case the feature is already present, please provide as
+much version information as possible:
+- if you have just installed a binary, provide the output of: tablizer --version
+- if you installed from source, provide the output of: make show-version
+- provide additional details: operating system and version and shell environment
+-->
+

.github/dependabot.yml

@@ -0,0 +1,10 @@
+version: 2
+updates:
+  - package-ecosystem: "gomod"
+    directory: "/"
+    schedule:
+      interval: "monthly"
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+        interval: "monthly"

.github/workflows/ci.yaml

@@ -4,22 +4,35 @@ jobs:
   build:
     strategy:
       matrix:
-        version: [1.17, 1.18, 1.19]
+        version: ['1.22']
         os: [ubuntu-latest, windows-latest, macos-latest]
     name: Build
     runs-on: ${{ matrix.os }}
     steps:
-    - name: Set up Go 1.18
-      uses: actions/setup-go@v3
+    - name: Set up Go ${{ matrix.version }}
+      uses: actions/setup-go@v5
       with:
-        go-version: ${{ matrix.version }}
+        go-version: '${{ matrix.version }}'
       id: go
 
     - name: checkout
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     - name: build
       run: make
 
     - name: test
       run: make test
+
+  golangci:
+    name: lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/setup-go@v5
+        with:
+          go-version: 1.22
+      - uses: actions/checkout@v4
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@v6
+        with:
+          skip-cache: true

.gitignore

@@ -0,0 +1,2 @@
+releases
+tablizer

CHANGELOG.md

@@ -4,6 +4,125 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) and this project adheres to [Semantic Versioning](http://semver.org).
 
+## [v1.0.14](https://github.com/TLINDEN/tablizer/tree/v1.0.14) - 2023-01-23
+
+[Full Changelog](https://github.com/TLINDEN/tablizer/compare/v1.0.13...v1.0.14)
+
+### Fixed
+
+- The -D parameter could not be used together with -a.
+
+- Fixed invalid argv handling: when the user wanted to read from stdin
+  but gave an argument which was  meant as a pattern, but also existed
+  as a filename, then tablizer opened the file, ignored stdin.
+  
+- Makefile indentation
+
+  
+### Added
+
+- added licens notes about dependencies
+
+- using hard coded uniseq version, see actions#3396457307
+
+- updated dependencies (go module versions)
+
+
+## [v1.0.13](https://github.com/TLINDEN/tablizer/tree/v1.0.13) - 2022-11-03
+
+[Full Changelog](https://github.com/TLINDEN/tablizer/compare/v1.0.12...v1.0.13)
+
+### Added
+
+- Added command line flag to generate shell completion code
+
+- Added an animated demo gif to the README to demonstrate the tool
+
+### Fixed
+
+- The `-A` flag wasn't implemented (default output mode).
+
+- Fixed building from source on systems w/o perls pod tools,
+  which is not requrired anyway since I always commit the latest
+  manpage.
+
+
+## [v1.0.12](https://github.com/TLINDEN/tablizer/tree/v1.0.12) - 2022-10-25
+
+[Full Changelog](https://github.com/TLINDEN/tablizer/compare/v1.0.11...v1.0.12)
+
+### Added
+
+- Added support to parse CSV input
+
+- Added CSV output support
+
+- Added support for environment variables
+
+### Changed
+
+- We do not use the generated help message anymore, instead we use the
+  usage from the  manpage, which we have to maintain  anyway. It looks
+  better and has flag groups, which  cobra is still lacking as of this
+  writing.
+  
+-  More refactoring  and  re-organization,  runtime configuration  now
+  lives in the cfg module.
+  
+  
+### Fixed
+
+- Fixed  [Bug #5](https://github.com/TLINDEN/tablizer/issues/5), where
+  matches have not been highlighted correctly in some rare cases.
+
+
+
+## [v1.0.11](https://github.com/TLINDEN/tablizer/tree/v1.0.11) - 2022-10-19
+
+[Full Changelog](https://github.com/TLINDEN/tablizer/compare/v1.0.10...v1.0.11)
+
+### Added
+
+- Added CI job golinter to regularly check for common mistakes.
+
+- Added YAML output mode.
+
+- Added more unit tests, we're over 95% in the lib module.
+
+### Changed
+
+- do  not use any global  variables anymore, makes the  code easier to
+  maintain, understand and test
+
+- using io.Writer  in print* functions, which is easier  to test, also
+  re-implemented the print tests.
+
+- replaced go-str2duration with my own implementation `duration2int()`.
+
+
+
+## [v1.0.10](https://github.com/TLINDEN/tablizer/tree/v1.0.10) - 2022-10-15
+
+[Full Changelog](https://github.com/TLINDEN/tablizer/compare/v1.0.9...v1.0.10)
+
+### Added
+
+- Added various sort modes: sort by time, by duration, numerical (-a -t -i)
+
+- Added possibility to modify sort order to descending (-D)
+
+- Added support  to specify a regexp in column  selector -c, which can
+  also be mixed with numerical column spec
+  
+- More unit tests
+
+### Fixed
+
+- Column specification  allowed to specify duplicate  columns like `-c
+  1,2,1,2` unchecked. Now this list will be deduplicated before use.
+
+
+
 ## [v1.0.9](https://github.com/TLINDEN/tablizer/tree/v1.0.9) - 2022-10-14
 
 [Full Changelog](https://github.com/TLINDEN/tablizer/compare/v1.0.8...v1.0.9)
@@ -24,7 +143,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
 
 ### Added
 
-- Added sort support with the new parameter -k (like sort(1).
+- Added sort support with the new parameter -k (like sort(1)).
 
 
 

CODE_OF_CONDUCT.md

@@ -1,7 +1,9 @@
 # No Code of Conduct
 
-This project does  **NOT** have a so called Code  of Conduct, nor will
-it ever get one.
+*TL;DR:* This project  does **NOT** have a so called  Code of Conduct,
+nor will it ever have one.
+
+## The Rant
 
 The reasons are somewhat complicated and  I'll try my best to document
 them here.
@@ -42,28 +44,22 @@ might be  sarcastic connotations I  don't understand or  references to
 historical figures, events  or traditions I don't know  and never have
 heard of.
 
-Juding over other peoples online behavior looks like a titanic task to
-me. It  is just not my  job to judge  others. I am not  legitimized or
-authorized to do so.
+Judging over other  peoples online behavior looks like  a titanic task
+to me. It is just not my job  to judge others, I am not legitimized or
+authorized to do so and I am not interested in this kind of business.
 
 Another huge  problem with ethical rules  is that you need  to outline
-and enforce  sanctions on thos who  violate the rules. But  since I am
-not an  elected authority  how would  I be  able to  do this?  I don't
-know.  And what  happens if  someone complains  about myself?  Shall I
+and enforce sanctions  on those who violate the rules.  But since I am
+not an  elected authority  how would I  be able to  do this?   I don't
+know.  And  what happens if  someone complains about myself?   Shall I
 remove myself from my own project? Come on!
 
-Last but  not least  there's the  law. I  am a  german citizen  and am
-living  in relatively  freedom.  Unlike  many other  people living  in
-democracies these days,  I myself fought for this very  freedom on the
-streets of  Leipzig in 1989.  I  saw the tanks, the  Stasi officers, I
-felt the  fear. But the laws  under I live  today and which I  have to
-adhere to, are only limited to the small speck on earth I am living on.
-
-So, let's say someone in india  says something insulting to some other
-developer in an issue.  Of course  german law does not apply to indian
-people. More, the insult might actually  not be an insult in india. In
-the end, nothing would happen. Under normal circumstances, maintainers
-would delete the posting, ban the user or remove push privileges etc.
+Last but  not least there's  the law. So,  let's say someone  in india
+says  something insulting  to some  other developer  in an  issue.  Of
+course german  law does not apply  to indian people. More,  the insult
+might actually  not be an insult  in india. In the  end, nothing would
+happen.  Under  normal  circumstances, maintainers  would  delete  the
+posting, ban the user or remove push privileges etc.
 
 But then, is there a way for  the offending user to defend himself? Of
 course not, since neither indian or german law alone applies. I cannot
@@ -71,13 +67,18 @@ go to  a german court  and sue the  guy and he  cannot do the  same in
 india. Or -  we possibly could but the judges  on both countries would
 just laugh and close the case.
 
-And let's not  even start talking about  there undemocratic "comitees"
-many projects are forming to  circumvent this problem.
-
 That  being said,  I  don't have  the  power nor  the  tools, nor  the
 authority to enforce serious sanctions  of any meaningful kind against
 others. Therefore I cannot outline any rules whatsoever.
 
+And let's not  even start talking about  there undemocratic "comitees"
+many projects  are forming to  circumvent this problem.  Some projects
+even  include  external  entities  like a  lawer  or  some  bureaucrat
+somewhere  just to  have the  ability  to complain  against a  comitee
+member. What a mess!
+
+
+
 ## So, which are the ethical rules within this project then?
 
 Well, there are none.
@@ -86,7 +87,8 @@ This project is  about code, not society. It doesn't  matter where you
 come from,  how you look,  how you think,  what you believe,  who your
 friends are, whay you  said or did sometime in the  past. I don't even
 care if you are a human being. You are an alien so bored that you need
-to submit code on github? Fine with me.
+to submit code on github? Fine with me. You're a convicted criminal? I
+don't give a shit!
 
 **The only thing I am interested here is Code and only Code.**
 

Makefile

@@ -17,32 +17,40 @@
 
 #
 # no need to modify anything below
-tool    = tablizer
-version = $(shell egrep "= .v" lib/common.go | cut -d'=' -f2 | cut -d'"' -f 2)
-archs   = android darwin freebsd linux netbsd openbsd windows
-PREFIX  = /usr/local
-UID     = root
-GID     = 0
-BRANCH  = $(shell git describe --all | cut -d/ -f2)
-COMMIT  = $(shell git rev-parse --short=8 HEAD)
-BUILD   = $(shell date +%Y.%m.%d.%H%M%S) 
-VERSION:= $(if $(filter $(BRANCH), development),$(version)-$(BRANCH)-$(COMMIT)-$(BUILD),$(version))
-
+tool      = tablizer
+version   = $(shell egrep "= .v" cfg/config.go | cut -d'=' -f2 | cut -d'"' -f 2)
+archs     = android darwin freebsd linux netbsd openbsd windows
+PREFIX    = /usr/local
+UID       = root
+GID       = 0
+BRANCH    = $(shell git branch --show-current)
+COMMIT    = $(shell git rev-parse --short=8 HEAD)
+BUILD     = $(shell date +%Y.%m.%d.%H%M%S) 
+VERSION  := $(if $(filter $(BRANCH), development),$(version)-$(BRANCH)-$(COMMIT)-$(BUILD),$(version))
+HAVE_POD := $(shell pod2text -h 2>/dev/null)
 
 all: $(tool).1 cmd/$(tool).go buildlocal
 
 %.1: %.pod
+ifdef HAVE_POD
 	pod2man -c "User Commands" -r 1 -s 1 $*.pod > $*.1
+endif
 
 cmd/%.go: %.pod
+ifdef HAVE_POD
 	echo "package cmd" > cmd/$*.go
 	echo >> cmd/$*.go
 	echo "var manpage = \`" >> cmd/$*.go
 	pod2text $*.pod >> cmd/$*.go
 	echo "\`" >> cmd/$*.go
 
+	echo "var usage = \`" >> cmd/$*.go
+	awk '/SYNOPS/{f=1;next} /DESCR/{f=0} f' $*.pod  | sed 's/^    //' >> cmd/$*.go
+	echo "\`" >> cmd/$*.go
+endif
+
 buildlocal:
-	go build -ldflags "-X 'github.com/tlinden/tablizer/lib.VERSION=$(VERSION)'"
+	go build -ldflags "-X 'github.com/tlinden/tablizer/cfg.VERSION=$(VERSION)'"
 
 release:
 	./mkrel.sh $(tool) $(version)
@@ -55,7 +63,39 @@ install: buildlocal
 	install -o $(UID) -g $(GID) -m 444 $(tool).1 $(PREFIX)/man/man1/
 
 clean:
-	rm -rf $(tool) releases
+	rm -rf $(tool) releases coverage.out
 
 test:
 	go test -v ./...
+	bash t/test.sh
+
+singletest:
+	@echo "Call like this: ''make singletest TEST=TestPrepareColumns MOD=lib"
+	go test -run $(TEST) github.com/tlinden/tablizer/$(MOD)
+
+cover-report:
+	go test ./... -cover -coverprofile=coverage.out
+	go tool cover -html=coverage.out
+
+show-versions: buildlocal
+	@echo "### tablizer version:"
+	@./tablizer --version
+
+	@echo
+	@echo "### go module versions:"
+	@go list -m all
+
+	@echo
+	@echo "### go version used for building:"
+	@grep -m 1 go go.mod
+
+goupdate:
+	go get -t -u=patch ./...
+
+lint:
+	golangci-lint run
+
+# keep til ireturn
+lint-full:
+	golangci-lint run --enable-all --exclude-use-default --disable exhaustivestruct,exhaustruct,depguard,interfacer,deadcode,golint,structcheck,scopelint,varcheck,ifshort,maligned,nosnakecase,godot,funlen,gofumpt,cyclop,noctx,gochecknoglobals,paralleltest,forbidigo,gci,godox,goimports,ireturn,stylecheck,testpackage,mirror,nestif,revive,goerr113,gomnd
+	gocritic check -enableAll *.go

README.md

@@ -70,13 +70,28 @@ NAME(1)                      READY(2) STATUS(3)  RESTARTS(4)    AGE(5)
 repldepl-7bcd8d5b64-q2bf4    1/1      Running    1 (69m ago)    5h26m
 ```
 
+Sometimes a filter regex is to broad  and you wish to filter only on a
+particular column. This is possible using `-F`:
+```
+% kubectl get pods | tablizer -n -Fname=2
+NAME                            READY   STATUS  RESTARTS        AGE
+repldepl-7bcd8d5b64-q2bf4       1/1     Running 1 (69m ago)     5h26m
+```
+
+Here we filtered  the `NAME` column for `2`, which  would have matched
+otherwise on all rows.
+
 There are more output modes like org-mode (orgtbl) and markdown.
 
+## Demo
+
+[![asciicast](demo/tablizer-demo.gif)](https://asciinema.org/a/9FKc3HPnlg8D2X8otheleEa9t)
+
 ## Installation
 
 There are multiple ways to install **tablizer**:
 
-- Go to the [latest release page](https://github.com/muesli/mango/releases/latest),
+- Go to the [latest release page](https://github.com/tlinden/tablizer/releases/latest),
   locate the binary for your operating system and platform.
   
   Download it and put it into some directory within your `$PATH` variable.

TODO

@@ -1 +0,0 @@
-

TODO.md

@@ -0,0 +1,18 @@
+## Fixes to be implemented
+
+## Features to be implemented
+
+- add comment support (csf.NewReader().Comment = '#')
+
+- add --no-headers option
+
+### Lisp Plugin Infrastructure using zygo
+
+Hooks:
+
+| Filter    | Purpose                                                     | Args                | Return |
+|-----------|-------------------------------------------------------------|---------------------|--------|
+| filter    | include or exclude lines                                    | row as hash         | bool   |
+| process   | do calculations with data, store results in global lisp env | whole dataset       | nil    |
+| transpose | modify a cell                                               | headername and cell | cell   |
+| append    | add one or more rows to the dataset (use this to add stats) | nil                 | rows   |

cfg/config.go

@@ -0,0 +1,386 @@
+/*
+Copyright © 2022-2024 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+package cfg
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"regexp"
+	"strings"
+
+	"github.com/glycerine/zygomys/zygo"
+	"github.com/gookit/color"
+	"github.com/hashicorp/hcl/v2/hclsimple"
+)
+
+const DefaultSeparator string = `(\s\s+|\t)`
+const Version string = "v1.2.2"
+const MAXPARTS = 2
+
+var DefaultLoadPath = os.Getenv("HOME") + "/.config/tablizer/lisp"
+var DefaultConfigfile = os.Getenv("HOME") + "/.config/tablizer/config"
+
+var VERSION string // maintained by -x
+
+// public config, set via config file or using defaults
+type Settings struct {
+	FG             string `hcl:"FG"`
+	BG             string `hcl:"BG"`
+	HighlightFG    string `hcl:"HighlightFG"`
+	HighlightBG    string `hcl:"HighlightBG"`
+	NoHighlightFG  string `hcl:"NoHighlightFG"`
+	NoHighlightBG  string `hcl:"NoHighlightBG"`
+	HighlightHdrFG string `hcl:"HighlightHdrFG"`
+	HighlightHdrBG string `hcl:"HighlightHdrBG"`
+}
+
+// internal config
+type Config struct {
+	Debug          bool
+	NoNumbering    bool
+	NoHeaders      bool
+	Columns        string
+	UseColumns     []int
+	Separator      string
+	OutputMode     int
+	InvertMatch    bool
+	Pattern        string
+	PatternR       *regexp.Regexp
+	UseFuzzySearch bool
+	UseHighlight   bool
+
+	SortMode       string
+	SortDescending bool
+	SortByColumn   int
+
+	/*
+	 FIXME: make configurable somehow, config file or ENV
+	 see https://github.com/gookit/color.
+	*/
+	ColorStyle        color.Style
+	HighlightStyle    color.Style
+	NoHighlightStyle  color.Style
+	HighlightHdrStyle color.Style
+
+	NoColor bool
+
+	// special  case: we use the  config struct to transport  the lisp
+	// env trough the program
+	Lisp *zygo.Zlisp
+
+	// a path containing lisp scripts to be loaded on startup
+	LispLoadPath string
+
+	// config file, optional
+	Configfile string
+
+	Settings Settings
+
+	// used for field filtering
+	Rawfilters []string
+	Filters    map[string]*regexp.Regexp
+}
+
+// maps outputmode short flags to output mode, ie. -O => -o orgtbl
+type Modeflag struct {
+	X bool
+	O bool
+	M bool
+	S bool
+	Y bool
+	A bool
+	C bool
+}
+
+// used for switching printers
+const (
+	Extended = iota + 1
+	Orgtbl
+	Markdown
+	Shell
+	Yaml
+	CSV
+	ASCII
+)
+
+// various sort types
+type Sortmode struct {
+	Numeric bool
+	Time    bool
+	Age     bool
+}
+
+// valid lisp hooks
+var ValidHooks []string
+
+// default color schemes
+func (conf *Config) Colors() map[color.Level]map[string]color.Color {
+	colors := map[color.Level]map[string]color.Color{
+		color.Level16: {
+			"bg": color.BgGreen, "fg": color.FgWhite,
+			"hlbg": color.BgGray, "hlfg": color.FgWhite,
+		},
+		color.Level256: {
+			"bg": color.BgLightGreen, "fg": color.FgWhite,
+			"hlbg": color.BgLightBlue, "hlfg": color.FgWhite,
+		},
+		color.LevelRgb: {
+			"bg": color.BgLightGreen, "fg": color.FgWhite,
+			"hlbg": color.BgHiGreen, "hlfg": color.FgWhite,
+			"nohlbg": color.BgWhite, "nohlfg": color.FgLightGreen,
+			"hdrbg": color.BgBlue, "hdrfg": color.FgWhite,
+		},
+	}
+
+	if len(conf.Settings.BG) > 0 {
+		colors[color.Level16]["bg"] = ColorStringToBGColor(conf.Settings.BG)
+		colors[color.Level256]["bg"] = ColorStringToBGColor(conf.Settings.BG)
+		colors[color.LevelRgb]["bg"] = ColorStringToBGColor(conf.Settings.BG)
+	}
+
+	if len(conf.Settings.FG) > 0 {
+		colors[color.Level16]["fg"] = ColorStringToColor(conf.Settings.FG)
+		colors[color.Level256]["fg"] = ColorStringToColor(conf.Settings.FG)
+		colors[color.LevelRgb]["fg"] = ColorStringToColor(conf.Settings.FG)
+	}
+
+	if len(conf.Settings.HighlightBG) > 0 {
+		colors[color.Level16]["hlbg"] = ColorStringToBGColor(conf.Settings.HighlightBG)
+		colors[color.Level256]["hlbg"] = ColorStringToBGColor(conf.Settings.HighlightBG)
+		colors[color.LevelRgb]["hlbg"] = ColorStringToBGColor(conf.Settings.HighlightBG)
+	}
+
+	if len(conf.Settings.HighlightFG) > 0 {
+		colors[color.Level16]["hlfg"] = ColorStringToColor(conf.Settings.HighlightFG)
+		colors[color.Level256]["hlfg"] = ColorStringToColor(conf.Settings.HighlightFG)
+		colors[color.LevelRgb]["hlfg"] = ColorStringToColor(conf.Settings.HighlightFG)
+	}
+
+	if len(conf.Settings.NoHighlightBG) > 0 {
+		colors[color.Level16]["nohlbg"] = ColorStringToBGColor(conf.Settings.NoHighlightBG)
+		colors[color.Level256]["nohlbg"] = ColorStringToBGColor(conf.Settings.NoHighlightBG)
+		colors[color.LevelRgb]["nohlbg"] = ColorStringToBGColor(conf.Settings.NoHighlightBG)
+	}
+
+	if len(conf.Settings.NoHighlightFG) > 0 {
+		colors[color.Level16]["nohlfg"] = ColorStringToColor(conf.Settings.NoHighlightFG)
+		colors[color.Level256]["nohlfg"] = ColorStringToColor(conf.Settings.NoHighlightFG)
+		colors[color.LevelRgb]["nohlfg"] = ColorStringToColor(conf.Settings.NoHighlightFG)
+	}
+
+	if len(conf.Settings.HighlightHdrBG) > 0 {
+		colors[color.Level16]["hdrbg"] = ColorStringToBGColor(conf.Settings.HighlightHdrBG)
+		colors[color.Level256]["hdrbg"] = ColorStringToBGColor(conf.Settings.HighlightHdrBG)
+		colors[color.LevelRgb]["hdrbg"] = ColorStringToBGColor(conf.Settings.HighlightHdrBG)
+	}
+
+	if len(conf.Settings.HighlightHdrFG) > 0 {
+		colors[color.Level16]["hdrfg"] = ColorStringToColor(conf.Settings.HighlightHdrFG)
+		colors[color.Level256]["hdrfg"] = ColorStringToColor(conf.Settings.HighlightHdrFG)
+		colors[color.LevelRgb]["hdrfg"] = ColorStringToColor(conf.Settings.HighlightHdrFG)
+	}
+
+	return colors
+}
+
+// find supported color mode, modifies config based on constants
+func (conf *Config) DetermineColormode() {
+	if !isTerminal(os.Stdout) {
+		color.Disable()
+	} else {
+		level := color.TermColorLevel()
+		colors := conf.Colors()
+
+		conf.ColorStyle = color.New(colors[level]["bg"], colors[level]["fg"])
+		conf.HighlightStyle = color.New(colors[level]["hlbg"], colors[level]["hlfg"])
+		conf.NoHighlightStyle = color.New(colors[level]["nohlbg"], colors[level]["nohlfg"])
+		conf.HighlightHdrStyle = color.New(colors[level]["hdrbg"], colors[level]["hdrfg"])
+	}
+}
+
+// Return true if current terminal is interactive
+func isTerminal(f *os.File) bool {
+	o, _ := f.Stat()
+
+	return (o.Mode() & os.ModeCharDevice) == os.ModeCharDevice
+}
+
+// main program version
+// generated  version string, used  by -v contains  lib.Version on
+//
+//	main branch, and lib.Version-$branch-$lastcommit-$date on
+//
+// development branch
+func Getversion() string {
+	return fmt.Sprintf("This is tablizer version %s", VERSION)
+}
+
+func (conf *Config) PrepareSortFlags(flag Sortmode) {
+	switch {
+	case flag.Numeric:
+		conf.SortMode = "numeric"
+	case flag.Age:
+		conf.SortMode = "duration"
+	case flag.Time:
+		conf.SortMode = "time"
+	default:
+		conf.SortMode = "string"
+	}
+}
+
+func (conf *Config) PrepareModeFlags(flag Modeflag) {
+	switch {
+	case flag.X:
+		conf.OutputMode = Extended
+	case flag.O:
+		conf.OutputMode = Orgtbl
+	case flag.M:
+		conf.OutputMode = Markdown
+	case flag.S:
+		conf.OutputMode = Shell
+	case flag.Y:
+		conf.OutputMode = Yaml
+	case flag.C:
+		conf.OutputMode = CSV
+	default:
+		conf.OutputMode = ASCII
+	}
+}
+
+func (conf *Config) PrepareFilters() error {
+	conf.Filters = make(map[string]*regexp.Regexp, len(conf.Rawfilters))
+
+	for _, filter := range conf.Rawfilters {
+		parts := strings.Split(filter, "=")
+		if len(parts) != MAXPARTS {
+			return errors.New("filter field and value must be separated by =")
+		}
+
+		reg, err := regexp.Compile(parts[1])
+		if err != nil {
+			return fmt.Errorf("failed to compile filter regex for field %s: %w",
+				parts[0], err)
+		}
+
+		conf.Filters[strings.ToLower(parts[0])] = reg
+	}
+
+	return nil
+}
+
+func (conf *Config) CheckEnv() {
+	// check for environment vars, command line flags have precedence,
+	// NO_COLOR is being checked by the color module itself.
+	if !conf.NoNumbering {
+		_, set := os.LookupEnv("T_NO_HEADER_NUMBERING")
+		if set {
+			conf.NoNumbering = true
+		}
+	}
+
+	if len(conf.Columns) == 0 {
+		cols := os.Getenv("T_COLUMNS")
+		if len(cols) > 1 {
+			conf.Columns = cols
+		}
+	}
+}
+
+func (conf *Config) ApplyDefaults() {
+	// mode specific defaults
+	if conf.OutputMode == Yaml || conf.OutputMode == CSV {
+		conf.NoNumbering = true
+	}
+
+	ValidHooks = []string{"filter", "process", "transpose", "append"}
+}
+
+func (conf *Config) PreparePattern(pattern string) error {
+	PatternR, err := regexp.Compile(pattern)
+
+	if err != nil {
+		return fmt.Errorf("regexp pattern %s is invalid: %w", conf.Pattern, err)
+	}
+
+	conf.PatternR = PatternR
+	conf.Pattern = pattern
+
+	return nil
+}
+
+// Parse config file.  Ignore if the file doesn't exist  but return an
+// error if it exists but fails to read or parse
+func (conf *Config) ParseConfigfile() error {
+	path, err := os.Stat(conf.Configfile)
+
+	if os.IsNotExist(err) || path.IsDir() {
+		// ignore non-existent or dirs
+		return nil
+	}
+
+	configstring, err := os.ReadFile(path.Name())
+	if err != nil {
+		return fmt.Errorf("failed to read config file %s: %w", path.Name(), err)
+	}
+
+	err = hclsimple.Decode(
+		path.Name(),
+		configstring,
+		nil,
+		&conf.Settings)
+	if err != nil {
+		return fmt.Errorf("failed to load configuration file %s: %w",
+			path.Name(), err)
+	}
+
+	return nil
+}
+
+// translate color string to internal color value
+func ColorStringToColor(colorname string) color.Color {
+	for name, color := range color.FgColors {
+		if name == colorname {
+			return color
+		}
+	}
+
+	for name, color := range color.ExFgColors {
+		if name == colorname {
+			return color
+		}
+	}
+
+	return color.Normal
+}
+
+// same, for background colors
+func ColorStringToBGColor(colorname string) color.Color {
+	for name, color := range color.BgColors {
+		if name == colorname {
+			return color
+		}
+	}
+
+	for name, color := range color.ExBgColors {
+		if name == colorname {
+			return color
+		}
+	}
+
+	return color.Normal
+}

cfg/config_test.go

@@ -0,0 +1,104 @@
+/*
+Copyright © 2022 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package cfg
+
+import (
+	"fmt"
+	//	"reflect"
+	"testing"
+)
+
+func TestPrepareModeFlags(t *testing.T) {
+	var tests = []struct {
+		flag   Modeflag
+		expect int // output (constant enum)
+	}{
+		// short commandline flags like -M
+		{Modeflag{X: true}, Extended},
+		{Modeflag{S: true}, Shell},
+		{Modeflag{O: true}, Orgtbl},
+		{Modeflag{Y: true}, Yaml},
+		{Modeflag{M: true}, Markdown},
+		{Modeflag{}, ASCII},
+	}
+
+	// FIXME: use a map for easier printing
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("PrepareModeFlags-expect-%d", testdata.expect)
+		t.Run(testname, func(t *testing.T) {
+			conf := Config{}
+
+			conf.PrepareModeFlags(testdata.flag)
+			if conf.OutputMode != testdata.expect {
+				t.Errorf("got: %d, expect: %d", conf.OutputMode, testdata.expect)
+			}
+		})
+	}
+}
+
+func TestPrepareSortFlags(t *testing.T) {
+	var tests = []struct {
+		flag   Sortmode
+		expect string // output
+	}{
+		// short commandline flags like -M
+		{Sortmode{Numeric: true}, "numeric"},
+		{Sortmode{Age: true}, "duration"},
+		{Sortmode{Time: true}, "time"},
+		{Sortmode{}, "string"},
+	}
+
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("PrepareSortFlags-expect-%s", testdata.expect)
+		t.Run(testname, func(t *testing.T) {
+			conf := Config{}
+
+			conf.PrepareSortFlags(testdata.flag)
+
+			if conf.SortMode != testdata.expect {
+				t.Errorf("got: %s, expect: %s", conf.SortMode, testdata.expect)
+			}
+		})
+	}
+}
+
+func TestPreparePattern(t *testing.T) {
+	var tests = []struct {
+		pattern string
+		wanterr bool
+	}{
+		{"[A-Z]+", false},
+		{"[a-z", true},
+	}
+
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("PreparePattern-pattern-%s-wanterr-%t",
+			testdata.pattern, testdata.wanterr)
+		t.Run(testname, func(t *testing.T) {
+			conf := Config{}
+
+			err := conf.PreparePattern(testdata.pattern)
+
+			if err != nil {
+				if !testdata.wanterr {
+					t.Errorf("PreparePattern returned error: %s", err)
+				}
+			}
+		})
+	}
+}

cmd/root.go

@@ -1,5 +1,5 @@
 /*
-Copyright © 2022 Thomas von Dein
+Copyright © 2022-2024 Thomas von Dein
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
@@ -18,24 +18,27 @@ package cmd
 
 import (
 	"bytes"
+	"errors"
 	"fmt"
-	"github.com/spf13/cobra"
-	"github.com/tlinden/tablizer/lib"
 	"log"
 	"os"
 	"os/exec"
-)
+	"strings"
 
-var ShowManual = false
+	"github.com/spf13/cobra"
+	"github.com/tlinden/tablizer/cfg"
+	"github.com/tlinden/tablizer/lib"
+)
 
 func man() {
 	man := exec.Command("less", "-")
 
-	var b bytes.Buffer
-	b.Write([]byte(manpage))
+	var buffer bytes.Buffer
+
+	buffer.Write([]byte(manpage))
 
 	man.Stdout = os.Stdout
-	man.Stdin = &b
+	man.Stdin = &buffer
 	man.Stderr = os.Stderr
 
 	err := man.Run()
@@ -45,64 +48,158 @@ func man() {
 	}
 }
 
-var rootCmd = &cobra.Command{
-	Use:   "tablizer [regex] [file, ...]",
-	Short: "[Re-]tabularize tabular data",
-	Long:  `Manipulate tabular output of other programs`,
-	RunE: func(cmd *cobra.Command, args []string) error {
-		if lib.ShowVersion {
-			fmt.Printf("This is tablizer version %s\n", lib.VERSION)
-			return nil
-		}
+func completion(cmd *cobra.Command, mode string) error {
+	switch mode {
+	case "bash":
+		return cmd.Root().GenBashCompletion(os.Stdout)
+	case "zsh":
+		return cmd.Root().GenZshCompletion(os.Stdout)
+	case "fish":
+		return cmd.Root().GenFishCompletion(os.Stdout, true)
+	case "powershell":
+		return cmd.Root().GenPowerShellCompletionWithDesc(os.Stdout)
+	default:
+		return errors.New("invalid shell parameter! Valid ones: bash|zsh|fish|powershell")
+	}
+}
 
-		if ShowManual {
-			man()
-			return nil
-		}
-
-		err := lib.PrepareColumns()
-		if err != nil {
-			return err
-		}
-
-		err = lib.PrepareModeFlags()
-		if err != nil {
-			return err
-		}
-
-		return lib.ProcessFiles(args)
-	},
+// we die with exit 1 if there's an error
+func wrapE(err error) {
+	if err != nil {
+		fmt.Println(err.Error())
+		os.Exit(1)
+	}
 }
 
 func Execute() {
+	var (
+		conf           cfg.Config
+		ShowManual     bool
+		ShowVersion    bool
+		ShowCompletion string
+		modeflag       cfg.Modeflag
+		sortmode       cfg.Sortmode
+	)
+
+	var rootCmd = &cobra.Command{
+		Use:   "tablizer [regex] [file, ...]",
+		Short: "[Re-]tabularize tabular data",
+		Long:  `Manipulate tabular output of other programs`,
+
+		Run: func(cmd *cobra.Command, args []string) {
+			if ShowVersion {
+				fmt.Println(cfg.Getversion())
+
+				return
+			}
+
+			if ShowManual {
+				man()
+
+				return
+			}
+
+			if len(ShowCompletion) > 0 {
+				wrapE(completion(cmd, ShowCompletion))
+
+				return
+			}
+
+			// Setup
+			wrapE(conf.ParseConfigfile())
+
+			conf.CheckEnv()
+			conf.PrepareModeFlags(modeflag)
+			conf.PrepareSortFlags(sortmode)
+
+			wrapE(conf.PrepareFilters())
+
+			conf.DetermineColormode()
+			conf.ApplyDefaults()
+
+			// setup lisp env, load plugins etc
+			wrapE(lib.SetupLisp(&conf))
+
+			// actual execution starts here
+			wrapE(lib.ProcessFiles(&conf, args))
+		},
+	}
+
+	// options
+	rootCmd.PersistentFlags().BoolVarP(&conf.Debug, "debug", "d", false,
+		"Enable debugging")
+	rootCmd.PersistentFlags().BoolVarP(&conf.NoNumbering, "no-numbering", "n", false,
+		"Disable header numbering")
+	rootCmd.PersistentFlags().BoolVarP(&conf.NoHeaders, "no-headers", "H", false,
+		"Disable header display")
+	rootCmd.PersistentFlags().BoolVarP(&conf.NoColor, "no-color", "N", false,
+		"Disable pattern highlighting")
+	rootCmd.PersistentFlags().BoolVarP(&ShowVersion, "version", "V", false,
+		"Print program version")
+	rootCmd.PersistentFlags().BoolVarP(&conf.InvertMatch, "invert-match", "v", false,
+		"select non-matching rows")
+	rootCmd.PersistentFlags().BoolVarP(&ShowManual, "man", "m", false,
+		"Display manual page")
+	rootCmd.PersistentFlags().BoolVarP(&conf.UseFuzzySearch, "fuzzy", "z", false,
+		"Use fuzzy searching")
+	rootCmd.PersistentFlags().BoolVarP(&conf.UseHighlight, "highlight-lines", "L", false,
+		"Use alternating background colors")
+	rootCmd.PersistentFlags().StringVarP(&ShowCompletion, "completion", "", "",
+		"Display completion code")
+	rootCmd.PersistentFlags().StringVarP(&conf.Separator, "separator", "s", cfg.DefaultSeparator,
+		"Custom field separator")
+	rootCmd.PersistentFlags().StringVarP(&conf.Columns, "columns", "c", "",
+		"Only show the speficied columns (separated by ,)")
+
+	// sort options
+	rootCmd.PersistentFlags().IntVarP(&conf.SortByColumn, "sort-by", "k", 0,
+		"Sort by column (default: 1)")
+
+	// sort mode, only 1 allowed
+	rootCmd.PersistentFlags().BoolVarP(&conf.SortDescending, "sort-desc", "D", false,
+		"Sort in descending order (default: ascending)")
+	rootCmd.PersistentFlags().BoolVarP(&sortmode.Numeric, "sort-numeric", "i", false,
+		"sort according to string numerical value")
+	rootCmd.PersistentFlags().BoolVarP(&sortmode.Time, "sort-time", "t", false,
+		"sort according to time string")
+	rootCmd.PersistentFlags().BoolVarP(&sortmode.Age, "sort-age", "a", false,
+		"sort according to age (duration) string")
+	rootCmd.MarkFlagsMutuallyExclusive("sort-numeric", "sort-time",
+		"sort-age")
+
+	// output flags, only 1 allowed
+	rootCmd.PersistentFlags().BoolVarP(&modeflag.X, "extended", "X", false,
+		"Enable extended output")
+	rootCmd.PersistentFlags().BoolVarP(&modeflag.M, "markdown", "M", false,
+		"Enable markdown table output")
+	rootCmd.PersistentFlags().BoolVarP(&modeflag.O, "orgtbl", "O", false,
+		"Enable org-mode table output")
+	rootCmd.PersistentFlags().BoolVarP(&modeflag.S, "shell", "S", false,
+		"Enable shell mode output")
+	rootCmd.PersistentFlags().BoolVarP(&modeflag.Y, "yaml", "Y", false,
+		"Enable yaml output")
+	rootCmd.PersistentFlags().BoolVarP(&modeflag.C, "csv", "C", false,
+		"Enable CSV output")
+	rootCmd.PersistentFlags().BoolVarP(&modeflag.A, "ascii", "A", false,
+		"Enable ASCII output (default)")
+	rootCmd.MarkFlagsMutuallyExclusive("extended", "markdown", "orgtbl",
+		"shell", "yaml", "csv")
+
+	// lisp options
+	rootCmd.PersistentFlags().StringVarP(&conf.LispLoadPath, "load-path", "l", cfg.DefaultLoadPath,
+		"Load path for lisp plugins (expects *.zy files)")
+
+	// config file
+	rootCmd.PersistentFlags().StringVarP(&conf.Configfile, "config", "f", cfg.DefaultConfigfile,
+		"config file (default: ~/.config/tablizer/config)")
+
+	// filters
+	rootCmd.PersistentFlags().StringArrayVarP(&conf.Rawfilters, "filter", "F", nil, "Filter by field (field=regexp)")
+
+	rootCmd.SetUsageTemplate(strings.TrimSpace(usage) + "\n")
+
 	err := rootCmd.Execute()
 	if err != nil {
 		os.Exit(1)
 	}
 }
-
-func init() {
-	rootCmd.PersistentFlags().BoolVarP(&lib.Debug, "debug", "d", false, "Enable debugging")
-	rootCmd.PersistentFlags().BoolVarP(&lib.NoNumbering, "no-numbering", "n", false, "Disable header numbering")
-	rootCmd.PersistentFlags().BoolVarP(&lib.NoColor, "no-color", "N", false, "Disable pattern highlighting")
-	rootCmd.PersistentFlags().BoolVarP(&lib.ShowVersion, "version", "V", false, "Print program version")
-	rootCmd.PersistentFlags().BoolVarP(&lib.InvertMatch, "invert-match", "v", false, "select non-matching rows")
-	rootCmd.PersistentFlags().BoolVarP(&ShowManual, "man", "m", false, "Display manual page")
-	rootCmd.PersistentFlags().StringVarP(&lib.Separator, "separator", "s", lib.DefaultSeparator, "Custom field separator")
-	rootCmd.PersistentFlags().StringVarP(&lib.Columns, "columns", "c", "", "Only show the speficied columns (separated by ,)")
-	rootCmd.PersistentFlags().IntVarP(&lib.SortByColumn, "sort-by", "k", 0, "Sort by column (default: 1)")
-
-	// output flags, only 1 allowed, hidden, since just short cuts
-	rootCmd.PersistentFlags().BoolVarP(&lib.OutflagExtended, "extended", "X", false, "Enable extended output")
-	rootCmd.PersistentFlags().BoolVarP(&lib.OutflagMarkdown, "markdown", "M", false, "Enable markdown table output")
-	rootCmd.PersistentFlags().BoolVarP(&lib.OutflagOrgtable, "orgtbl", "O", false, "Enable org-mode table output")
-	rootCmd.PersistentFlags().BoolVarP(&lib.OutflagShell, "shell", "S", false, "Enable shell mode output")
-	rootCmd.MarkFlagsMutuallyExclusive("extended", "markdown", "orgtbl", "shell")
-	rootCmd.Flags().MarkHidden("extended")
-	rootCmd.Flags().MarkHidden("orgtbl")
-	rootCmd.Flags().MarkHidden("markdown")
-	rootCmd.Flags().MarkHidden("shell")
-
-	// same thing but more common, takes precedence over above group
-	rootCmd.PersistentFlags().StringVarP(&lib.OutputMode, "output", "o", "", "Output mode - one of: orgtbl, markdown, extended, shell, ascii(default)")
-}

cmd/tablizer.go

@@ -8,21 +8,40 @@ SYNOPSIS
         Usage:
           tablizer [regex] [file, ...] [flags]
     
-        Flags:
+        Operational Flags:
           -c, --columns string     Only show the speficied columns (separated by ,)
-          -d, --debug              Enable debugging
-          -h, --help               help for tablizer
           -v, --invert-match       select non-matching rows
-          -m, --man                Display manual page
           -n, --no-numbering       Disable header numbering
           -N, --no-color           Disable pattern highlighting
-          -o, --output string      Output mode - one of: orgtbl, markdown, extended, ascii(default)
+          -H, --no-headers         Disable headers display
+          -s, --separator string   Custom field separator
+          -k, --sort-by int        Sort by column (default: 1)
+          -z, --fuzzy              Use fuzzy search [experimental]
+          -F, --filter field=reg   Filter given field with regex, can be used multiple times
+
+        Output Flags (mutually exclusive):
           -X, --extended           Enable extended output
           -M, --markdown           Enable markdown table output
           -O, --orgtbl             Enable org-mode table output
-          -s, --separator string   Custom field separator
-          -k, --sort-by int        Sort by column (default: 1)
-          -v, --version            Print program version
+          -S, --shell              Enable shell evaluable output
+          -Y, --yaml               Enable yaml output
+          -C, --csv                Enable CSV output
+          -A, --ascii              Default output mode, ascii tabular
+          -L, --hightlight-lines   Use alternating background colors for tables
+
+        Sort Mode Flags (mutually exclusive):
+          -a, --sort-age           sort according to age (duration) string
+          -D, --sort-desc          Sort in descending order (default: ascending)
+          -i, --sort-numeric       sort according to string numerical value
+          -t, --sort-time          sort according to time string
+
+        Other Flags:
+              --completion <shell> Generate the autocompletion script for <shell>
+          -f, --config <file>      Configuration file (default: ~/.config/tablizer/config)
+          -d, --debug              Enable debugging
+          -h, --help               help for tablizer
+          -m, --man                Display manual page
+          -V, --version            Print program version
 
 DESCRIPTION
     Many programs generate tabular output. But sometimes you need to
@@ -64,7 +83,7 @@ DESCRIPTION
        NAME(1) READY(2) STATUS(3) RESTARTS(4) AGE(5)
 
     These numbers denote the column and you can use them to specify which
-    columns you want to have in your output:
+    columns you want to have in your output (see COLUMNS:
 
        kubectl get pods | tablizer -c1,3
 
@@ -73,17 +92,33 @@ DESCRIPTION
 
     The numbering can be suppressed by using the -n option.
 
+    By default tablizer shows a header containing the names of each column.
+    This can be disabled using the -H option. Be aware that this only
+    affects tabular output modes. Shell, Extended, Yaml and CSV output modes
+    always use the column names.
+
     By default, if a pattern has been speficied, matches will be
     highlighted. You can disable this behavior with the -N option.
 
     Use the -k option to specify by which column to sort the tabular data
     (as in GNU sort(1)). The default sort column is the first one. To
-    disable sorting at all, supply 0 (Zero) to -k.
+    disable sorting at all, supply 0 (Zero) to -k. The default sort order is
+    ascending. You can change this to descending order using the option -D.
+    The default sort order is by string, but there are other sort modes:
+
+    -a --sort-age
+        Sorts duration strings like "1d4h32m51s".
+
+    -i --sort-numeric
+        Sorts numeric fields.
+
+    -t --sort-time
+        Sorts timestamps.
 
     Finally the -d option enables debugging output which is mostly useful
     for the developer.
 
-  PATTERNS
+  PATTERNS AND FILTERING
     You can reduce the rows being displayed by using a regular expression
     pattern. The regexp is PCRE compatible, refer to the syntax cheat sheet
     here: <https://github.com/google/re2/wiki/Syntax>. If you want to read a
@@ -107,6 +142,49 @@ DESCRIPTION
 
         kubectl get pods -A | tablizer "(?i)account"
 
+    You can use the experimental fuzzy search feature by providing the
+    option -z, in which case the pattern is regarded as a fuzzy search term,
+    not a regexp.
+
+    Sometimes you want to filter by one or more columns. You can do that
+    using the -F option. The option can be specified multiple times and has
+    the following format:
+
+        fieldname=regexp
+
+    Fieldnames (== columns headers) are case insensitive.
+
+    If you specify more than one filter, both filters have to match (AND
+    operation).
+
+    If the option -v is specified, the filtering is inverted.
+
+  COLUMNS
+    The parameter -c can be used to specify, which columns to display. By
+    default tablizer numerizes the header names and these numbers can be
+    used to specify which header to display, see example above.
+
+    However, beside numbers, you can also use regular expressions with -c,
+    also separated by comma. And you can mix column numbers with regexps.
+
+    Lets take this table:
+
+            PID TTY          TIME CMD
+          14001 pts/0    00:00:00 bash
+          42871 pts/0    00:00:00 ps
+          42872 pts/0    00:00:00 sed
+
+    We want to see only the CMD column and use a regex for this:
+
+        ps | tablizer -s '\s+' -c C
+        CMD(4)
+        bash
+        ps
+        tablizer
+        sed
+
+    where "C" is our regexp which matches CMD.
+
   OUTPUT MODES
     There might be cases when the tabular output of a program is way too
     large for your current terminal but you still need to see every column.
@@ -137,7 +215,93 @@ DESCRIPTION
 
     Beside normal ascii mode (the default) and extended mode there are more
     output modes available: orgtbl which prints an Emacs org-mode table and
-    markdown which prints a Markdown table.
+    markdown which prints a Markdown table, yaml, which prints yaml encoding
+    and CSV mode, which prints a comma separated value file.
+
+  ENVIRONMENT VARIABLES
+    tablizer supports certain environment variables which use can use to
+    influence program behavior. Commandline flags have always precedence
+    over environment variables.
+
+    <T_NO_HEADER_NUMBERING> - disable numbering of header fields, like -n.
+    <T_COLUMNS> - comma separated list of columns to output, like -c
+    <NO_COLORS> - disable colorization of matches, like -N
+
+  COMPLETION
+    Shell completion for command line options can be enabled by using the
+    --completion flag. The required parameter is the name of your shell.
+    Currently supported are: bash, zsh, fish and powershell.
+
+    Detailed instructions:
+
+    Bash:
+           source <(tablizer --completion bash)
+
+        To load completions for each session, execute once:
+
+          # Linux:
+          $ tablizer --completion bash > /etc/bash_completion.d/tablizer
+
+          # macOS:
+          $ tablizer --completion bash > $(brew --prefix)/etc/bash_completion.d/tablizer
+
+    Zsh:
+        If shell completion is not already enabled in your environment, you
+        will need to enable it. You can execute the following once:
+
+          echo "autoload -U compinit; compinit" >> ~/.zshrc
+
+        To load completions for each session, execute once:
+
+          $ tablizer --completion zsh > "${fpath[1]}/_tablizer"
+
+        You will need to start a new shell for this setup to take effect.
+
+    fish:
+           tablizer --completion fish | source
+
+        To load completions for each session, execute once:
+
+           tablizer --completion fish > ~/.config/fish/completions/tablizer.fish
+
+    PowerShell:
+           tablizer --completion powershell | Out-String | Invoke-Expression
+
+        To load completions for every new session, run:
+
+           tablizer --completion powershell > tablizer.ps1
+
+        and source this file from your PowerShell profile.
+
+CONFIGURATION AND COLORS
+    YOu can put certain configuration values into a configuration file in
+    HCL format. By default tablizer looks for
+    "$HOME/.config/tablizer/config", but you can provide one using the
+    parameter "-f".
+
+    In the configuration the following variables can be defined:
+
+        BG             = "lightGreen"
+        FG             = "white"
+        HighlightBG    = "lightGreen"
+        HighlightFG    = "white"
+        NoHighlightBG  = "white"
+        NoHighlightFG  = "lightGreen"
+        HighlightHdrBG = "red"
+        HighlightHdrFG = "white"
+
+    The following color definitions are available:
+
+    black, blue, cyan, darkGray, default, green, lightBlue, lightCyan,
+    lightGreen, lightMagenta, lightRed, lightWhite, lightYellow, magenta,
+    red, white, yellow
+
+    The Variables FG and BG are being used to highlight matches. The other
+    *FG and *BG variables are for colored table output (enabled with the
+    "-L" parameter).
+
+    Colorization can be turned off completely either by setting the
+    parameter "-N" or the environment variable NO_COLOR to a true value.
 
 BUGS
     In order to report a bug, unexpected behavior, feature requests or to
@@ -148,9 +312,9 @@ LICENSE
     This software is licensed under the GNU GENERAL PUBLIC LICENSE version
     3.
 
-    Copyright (c) 2022 by Thomas von Dein
+    Copyright (c) 2022-2024 by Thomas von Dein
 
-    This software uses the following GO libraries:
+    This software uses the following GO modules:
 
     repr (https://github.com/alecthomas/repr)
         Released under the MIT License, Copyright (c) 2016 Alec Thomas
@@ -159,7 +323,62 @@ LICENSE
         Released under the Apache 2.0 license, Copyright 2013-2022 The Cobra
         Authors
 
+    dateparse (github.com/araddon/dateparse)
+        Released under the MIT License, Copyright (c) 2015-2017 Aaron Raddon
+
+    color (github.com/gookit/color)
+        Released under the MIT License, Copyright (c) 2016 inhere
+
+    tablewriter (github.com/olekukonko/tablewriter)
+        Released under the MIT License, Copyright (c) 201 by Oleku Konko
+
+    yaml (gopkg.in/yaml.v3)
+        Released under the MIT License, Copyright (c) 2006-2011 Kirill
+        Simonov
+
 AUTHORS
     Thomas von Dein tom AT vondein DOT org
 
 `
+var usage = `
+
+Usage:
+  tablizer [regex] [file, ...] [flags]
+
+Operational Flags:
+  -c, --columns string     Only show the speficied columns (separated by ,)
+  -v, --invert-match       select non-matching rows
+  -n, --no-numbering       Disable header numbering
+  -N, --no-color           Disable pattern highlighting
+  -H, --no-headers         Disable headers display
+  -s, --separator string   Custom field separator
+  -k, --sort-by int        Sort by column (default: 1)
+  -z, --fuzzy              Use fuzzy search [experimental]
+  -F, --filter field=reg   Filter given field with regex, can be used multiple times
+
+Output Flags (mutually exclusive):
+  -X, --extended           Enable extended output
+  -M, --markdown           Enable markdown table output
+  -O, --orgtbl             Enable org-mode table output
+  -S, --shell              Enable shell evaluable output
+  -Y, --yaml               Enable yaml output
+  -C, --csv                Enable CSV output
+  -A, --ascii              Default output mode, ascii tabular
+  -L, --hightlight-lines   Use alternating background colors for tables
+
+Sort Mode Flags (mutually exclusive):
+  -a, --sort-age           sort according to age (duration) string
+  -D, --sort-desc          Sort in descending order (default: ascending)
+  -i, --sort-numeric       sort according to string numerical value
+  -t, --sort-time          sort according to time string
+
+Other Flags:
+      --completion <shell> Generate the autocompletion script for <shell>
+  -f, --config <file>      Configuration file (default: ~/.config/tablizer/config)
+  -d, --debug              Enable debugging
+  -h, --help               help for tablizer
+  -m, --man                Display manual page
+  -V, --version            Print program version
+
+
+`

config.hcl

@@ -0,0 +1,12 @@
+# supported  colors:
+# black, blue,  cyan, darkGray, default, green,  lightBlue, lightCyan,
+# lightGreen,   lightMagenta,   lightRed,   lightWhite,   lightYellow,
+# magenta, red, white, yellow
+BG             = "lightGreen"
+FG             = "white"
+HighlightBG    = "lightGreen"
+HighlightFG    = "white"
+NoHighlightBG  = "white"
+NoHighlightFG  = "lightGreen"
+HighlightHdrBG = "red"
+HighlightHdrFG = "white"

demo/Makefile

@@ -0,0 +1,3 @@
+all:
+	LC_ALL=en_US.UTF-8 asciinema rec --cols 50 --row 30 -c ./demo.sh --overwrite tmp.cast
+	agg tmp.cast tmp.gif

demo/demo.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+prompt() {
+    if test -n "$1"; then
+        echo
+        echo -n "% $*"
+        sleep 1
+        echo
+        $*
+        echo
+        echo -n "% "
+    else
+        echo -n "% "
+    fi
+}
+
+PATH=..:$PATH
+clear
+while IFS=$'\t' read -r flags table msg source _; do
+    echo "#"
+    echo "#   source tabular data:"
+    cat $table
+    echo
+    echo "#"
+    echo "#   $msg:"
+    prompt "tablizer $flags $table"
+  
+    sleep 4
+    clear
+done < <(yq -r tables.yaml \
+         | yq -r '.tables[] | [.flags, .table, .msg, .source] | @tsv')

demo/table.demo1

@@ -0,0 +1,4 @@
+NAME   DURATION   COUNT  WHEN
+beta   1d10h5m1s  33     3/1/2014
+alpha  4h35m      170    2013-Feb-03
+ceta   33d12h     9      06/Jan/2008 15:04:05 -0700

demo/table.demo2

@@ -0,0 +1,3 @@
+    PID TTY          TIME CMD
+  30912 pts/0    00:00:00 bash
+  49526 pts/0    00:00:00 ps

demo/tables.yaml

@@ -0,0 +1,54 @@
+tables:
+# OUTPUTS
+  - flags: -A
+    table: table.demo1
+    msg: default output mode
+  - flags: -O
+    table: table.demo1
+    msg: orgmode output mode
+  - flags: -M
+    table: table.demo1
+    msg: markdown output mode
+  - flags: -S
+    table: table.demo1
+    msg: shell output mode
+  - flags: -X
+    table: table.demo1
+    msg: extended output mode
+  - flags: -Y
+    table: table.demo1
+    msg: yaml output mode
+  - flags: -C
+    table: table.demo1
+    msg: CSV output mode
+
+  # SORTS
+  - flags: -A -k 3
+    table: table.demo1
+    msg: sort by column 3
+  - flags: -A -k 4 -t
+    table: table.demo1
+    msg: sort by column 4 and sort type time
+  - flags: -A -k 2 -a
+    table: table.demo1
+    msg: sort by column 2 and sort type duration
+
+  # REDUCE
+  - flags: -A -c 1,3
+    table: table.demo1
+    msg: only display column 1 and 3
+  - flags: -A -c AM,RA
+    table: table.demo1
+    msg: only display columns matching /(RA|AM)/
+  - flags: -X -c 1,3
+    table: table.demo1
+    msg: only display column 1 and 3 in extended mode
+
+  # SEARCH
+  - flags: /20 -A
+    table: table.demo1
+    msg: only show rows matching /20
+  - flags: /20 -A -v
+    table: table.demo1
+    msg: only show rows NOT matching /20
+

demo/tablizer-demo.cast

@@ -0,0 +1,119 @@
+{"version": 2, "width": 80, "height": 25, "timestamp": 1666890777, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}}
+[0.004618, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[0.010297, "o", "#\r\n#   source tabular data:\r\n"]
+[0.010898, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[0.011125, "o", "\r\n#\r\n"]
+[0.011177, "o", "#   default output mode:\r\n"]
+[0.011219, "o", "\r\n% tablizer -A table.demo1"]
+[1.011851, "o", "\r\n"]
+[1.013635, "o", "NAME(1)\tDURATION(2)\tCOUNT(3)\tWHEN(4)                    \r\nbeta   \t1d10h5m1s  \t33      \t3/1/2014                  \t\r\nalpha  \t4h35m      \t170     \t2013-Feb-03               \t\r\nceta   \t33d12h     \t9       \t06/Jan/2008 15:04:05 -0700\t\r\n"]
+[1.014021, "o", "\r\n% "]
+[5.015241, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[5.015339, "o", "#\r\n#   source tabular data:\r\n"]
+[5.015688, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[5.015776, "o", "\r\n#\r\n#   orgmode output mode:\r\n\r\n% tablizer -O table.demo1"]
+[6.016322, "o", "\r\n"]
+[6.01823, "o", "+---------+-------------+----------+----------------------------+\r\n| NAME(1) | DURATION(2) | COUNT(3) |          WHEN(4)           |\r\n+---------+-------------+----------+----------------------------+\r\n| beta    | 1d10h5m1s   |       33 | 3/1/2014                   |\r\n| alpha   | 4h35m       |      170 | 2013-Feb-03                |\r\n| ceta    | 33d12h      |        9 | 06/Jan/2008 15:04:05 -0700 |\r\n+---------+-------------+----------+----------------------------+\r\n"]
+[6.018497, "o", "\r\n% "]
+[10.020014, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[10.020112, "o", "#\r\n#   source tabular data:\r\n"]
+[10.020573, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[10.020643, "o", "\r\n#\r\n"]
+[10.02068, "o", "#   markdown output mode:\r\n\r\n% tablizer -M table.demo1"]
+[11.021559, "o", "\r\n"]
+[11.023551, "o", "| NAME(1) | DURATION(2) | COUNT(3) |          WHEN(4)           |\r\n|---------|-------------|----------|----------------------------|\r\n| beta    | 1d10h5m1s   |       33 | 3/1/2014                   |\r\n| alpha   | 4h35m       |      170 | 2013-Feb-03                |\r\n| ceta    | 33d12h      |        9 | 06/Jan/2008 15:04:05 -0700 |\r\n"]
+[11.023838, "o", "\r\n% "]
+[15.025244, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[15.025345, "o", "#\r\n#   source tabular data:\r\n"]
+[15.025829, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[15.025915, "o", "\r\n#\r\n#   shell output mode:\r\n"]
+[15.025931, "o", "\r\n"]
+[15.025948, "o", "% tablizer -S table.demo1"]
+[16.026714, "o", "\r\n"]
+[16.028606, "o", "NAME(1)=\"beta\" DURATION(2)=\"1d10h5m1s\" COUNT(3)=\"33\" WHEN(4)=\"3/1/2014\"\r\nNAME(1)=\"alpha\" DURATION(2)=\"4h35m\" COUNT(3)=\"170\" WHEN(4)=\"2013-Feb-03\"\r\nNAME(1)=\"ceta\" DURATION(2)=\"33d12h\" COUNT(3)=\"9\" WHEN(4)=\"06/Jan/2008 15:04:05 -0700\"\r\n"]
+[16.029144, "o", "\r\n% "]
+[20.030593, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[20.030706, "o", "#\r\n#   source tabular data:\r\n"]
+[20.03121, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[20.031277, "o", "\r\n#\r\n#   extended output mode:\r\n"]
+[20.031327, "o", "\r\n% tablizer -X table.demo1"]
+[21.032053, "o", "\r\n"]
+[21.033787, "o", "    NAME(1): beta\r\nDURATION(2): 1d10h5m1s\r\n   COUNT(3): 33\r\n    WHEN(4): 3/1/2014\r\n\r\n    NAME(1): alpha\r\nDURATION(2): 4h35m\r\n   COUNT(3): 170\r\n    WHEN(4): 2013-Feb-03\r\n\r\n    NAME(1): ceta\r\nDURATION(2): 33d12h\r\n   COUNT(3): 9\r\n    WHEN(4): 06/Jan/2008 15:04:05 -0700\r\n\r\n"]
+[21.034132, "o", "\r\n% "]
+[25.035531, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[25.035585, "o", "#\r\n"]
+[25.035681, "o", "#   source tabular data:\r\n"]
+[25.036179, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[25.036232, "o", "\r\n#\r\n"]
+[25.036274, "o", "#   yaml output mode:\r\n\r\n% tablizer -Y table.demo1"]
+[26.036928, "o", "\r\n"]
+[26.038674, "o", "entries:\r\n    - count: 33\r\n      duration: \"1d10h5m1s\"\r\n      name: \"beta\"\r\n      when: \"3/1/2014\"\r\n    - count: 170\r\n      duration: \"4h35m\"\r\n      name: \"alpha\"\r\n      when: \"2013-Feb-03\"\r\n    - count: 9\r\n      duration: \"33d12h\"\r\n      name: \"ceta\"\r\n      when: \"06/Jan/2008 15:04:05 -0700\"\r\n"]
+[26.038975, "o", "\r\n% "]
+[30.040539, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[30.040659, "o", "#\r\n#   source tabular data:\r\n"]
+[30.041167, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[30.041246, "o", "\r\n#\r\n#   CSV output mode:\r\n\r\n% tablizer -C table.demo1"]
+[31.042088, "o", "\r\n"]
+[31.043721, "o", "NAME,DURATION,COUNT,WHEN\r\nbeta,1d10h5m1s,33,3/1/2014\r\nalpha,4h35m,170,2013-Feb-03\r\nceta,33d12h,9,06/Jan/2008 15:04:05 -0700\r\n"]
+[31.043997, "o", "\r\n% "]
+[35.045523, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[35.04563, "o", "#\r\n#   source tabular data:\r\n"]
+[35.046209, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[35.046275, "o", "\r\n#\r\n#   sort by column 3:\r\n\r\n% tablizer -A -k 3 table.demo1"]
+[36.047083, "o", "\r\n"]
+[36.048793, "o", "NAME(1)\tDURATION(2)\tCOUNT(3)\tWHEN(4)                    \r\nalpha  \t4h35m      \t170     \t2013-Feb-03               \t\r\nbeta   \t1d10h5m1s  \t33      \t3/1/2014                  \t\r\nceta   \t33d12h     \t9       \t06/Jan/2008 15:04:05 -0700\t\r\n"]
+[36.049077, "o", "\r\n% "]
+[40.050739, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[40.050925, "o", "#\r\n#   source tabular data:\r\n"]
+[40.051481, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[40.051671, "o", "\r\n#\r\n#   sort by column 4 and sort type time:\r\n\r\n% tablizer -A -k 4 -t table.demo1"]
+[41.052486, "o", "\r\n"]
+[41.05454, "o", "NAME(1)\tDURATION(2)\tCOUNT(3)\tWHEN(4)                    \r\nceta   \t33d12h     \t9       \t06/Jan/2008 15:04:05 -0700\t\r\nalpha  \t4h35m      \t170     \t2013-Feb-03               \t\r\nbeta   \t1d10h5m1s  \t33      \t3/1/2014                  \t\r\n"]
+[41.054864, "o", "\r\n% "]
+[45.056297, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[45.056405, "o", "#\r\n#   source tabular data:\r\n"]
+[45.056895, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[45.056978, "o", "\r\n#\r\n"]
+[45.057023, "o", "#   sort by column 2 and sort type duration:\r\n"]
+[45.057073, "o", "\r\n% tablizer -A -k 2 -a table.demo1"]
+[46.057895, "o", "\r\n"]
+[46.059684, "o", "NAME(1)\tDURATION(2)\tCOUNT(3)\tWHEN(4)                    \r\nalpha  \t4h35m      \t170     \t2013-Feb-03               \t\r\nbeta   \t1d10h5m1s  \t33      \t3/1/2014                  \t\r\nceta   \t33d12h     \t9       \t06/Jan/2008 15:04:05 -0700\t\r\n"]
+[46.059988, "o", "\r\n% "]
+[50.061514, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[50.061622, "o", "#\r\n#   source tabular data:\r\n"]
+[50.062091, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[50.062188, "o", "\r\n#\r\n#   only display column 1 and 3:\r\n\r\n% tablizer -A -c 1,3 table.demo1"]
+[51.062985, "o", "\r\n"]
+[51.066293, "o", "NAME(1)\tCOUNT(3) \r\nbeta   \t33      \t\r\nalpha  \t170     \t\r\nceta   \t9       \t\r\n"]
+[51.066843, "o", "\r\n% "]
+[55.070781, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[55.071327, "o", "#\r\n#   source tabular data:\r\n"]
+[55.073499, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[55.073822, "o", "\r\n#\r\n#   only display columns matching /(RA|AM)/:\r\n"]
+[55.074188, "o", "\r\n% tablizer -A -c AM,RA table.demo1"]
+[56.07636, "o", "\r\n"]
+[56.078603, "o", "NAME(1)\tDURATION(2) \r\nbeta   \t1d10h5m1s  \t\r\nalpha  \t4h35m      \t\r\nceta   \t33d12h     \t\r\n"]
+[56.078957, "o", "\r\n% "]
+[60.080574, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[60.080734, "o", "#\r\n#   source tabular data:\r\n"]
+[60.081286, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[60.081418, "o", "\r\n#\r\n#   only display column 1 and 3 in extended mode:\r\n\r\n% tablizer -X -c 1,3 table.demo1"]
+[61.082844, "o", "\r\n"]
+[61.089822, "o", " NAME(1): beta\r\nCOUNT(3): 33\r\n\r\n NAME(1): alpha\r\nCOUNT(3): 170\r\n\r\n NAME(1): ceta\r\nCOUNT(3): 9\r\n\r\n"]
+[61.090969, "o", "\r\n% "]
+[65.096092, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[65.096571, "o", "#\r\n#   source tabular data:\r\n"]
+[65.098736, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[65.099085, "o", "\r\n#\r\n#   only show rows matching /20:\r\n"]
+[65.099283, "o", "\r\n% tablizer /20 -A table.demo1"]
+[66.101537, "o", "\r\n"]
+[66.109112, "o", "NAME(1)\tDURATION(2)\tCOUNT(3)\tWHEN(4)                    \r\nbeta   \t1d10h5m1s  \t33      \t3/1\u001b[102;30m/20\u001b[0m14                  \t\r\nceta   \t33d12h     \t9       \t06/Jan\u001b[102;30m/20\u001b[0m08 15:04:05 -0700\t\r\n"]
+[66.109405, "o", "\r\n% "]
+[70.11076, "o", "\u001b[H\u001b[2J\u001b[3J"]
+[70.110873, "o", "#\r\n#   source tabular data:\r\n"]
+[70.111365, "o", "NAME   DURATION   COUNT  WHEN\r\nbeta   1d10h5m1s  33     3/1/2014\r\nalpha  4h35m      170    2013-Feb-03\r\nceta   33d12h     9      06/Jan/2008 15:04:05 -0700\r\n"]
+[70.111469, "o", "\r\n#\r\n#   only show rows NOT matching /20:\r\n\r\n% tablizer /20 -A -v table.demo1"]
+[71.112738, "o", "\r\n"]
+[71.120032, "o", "NAME(1)\tDURATION(2)\tCOUNT(3)\tWHEN(4)     \r\nalpha  \t4h35m      \t170     \t2013-Feb-03\t\r\n"]
+[71.121127, "o", "\r\n% "]
+[75.126199, "o", "\u001b[H\u001b[2J\u001b[3J"]

go.mod

@@ -1,18 +1,44 @@
 module github.com/tlinden/tablizer
 
-go 1.18
+go 1.22
 
 require (
-	github.com/alecthomas/repr v0.0.0-20180818092828-117648cd9897
-	github.com/gookit/color v1.5.2
+	github.com/alecthomas/repr v0.4.0
+	github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de
+	github.com/glycerine/zygomys v5.1.2+incompatible
+	github.com/gookit/color v1.5.4
+	github.com/hashicorp/hcl/v2 v2.22.0
+	github.com/lithammer/fuzzysearch v1.1.8
 	github.com/olekukonko/tablewriter v0.0.5
-	github.com/spf13/cobra v1.5.0
-	github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778
+	github.com/spf13/cobra v1.8.1
+	gopkg.in/yaml.v3 v3.0.1
 )
 
 require (
-	github.com/inconshreveable/mousetrap v1.0.0 // indirect
-	github.com/mattn/go-runewidth v0.0.9 // indirect
+	github.com/agext/levenshtein v1.2.1 // indirect
+	github.com/apparentlymart/go-textseg/v13 v13.0.0 // indirect
+	github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
+	github.com/glycerine/blake2b v0.0.0-20151022103502-3c8c640cd7be // indirect
+	github.com/glycerine/goconvey v0.0.0-20190410193231-58a59202ab31 // indirect
+	github.com/glycerine/greenpack v5.1.1+incompatible // indirect
+	github.com/glycerine/liner v0.0.0-20160121172638-72909af234e0 // indirect
+	github.com/google/go-cmp v0.6.0 // indirect
+	github.com/gopherjs/gopherjs v1.17.2 // indirect
+	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/jtolds/gls v4.20.0+incompatible // indirect
+	github.com/mattn/go-runewidth v0.0.10 // indirect
+	github.com/mitchellh/go-wordwrap v0.0.0-20150314170334-ad45545899c7 // indirect
+	github.com/philhofer/fwd v1.1.2 // indirect
+	github.com/rivo/uniseg v0.1.0 // indirect
+	github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636 // indirect
+	github.com/shurcooL/go-goon v1.0.0 // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
-	golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44 // indirect
+	github.com/tinylib/msgp v1.1.9 // indirect
+	github.com/ugorji/go/codec v1.2.11 // indirect
+	github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 // indirect
+	github.com/zclconf/go-cty v1.13.0 // indirect
+	golang.org/x/mod v0.13.0 // indirect
+	golang.org/x/sys v0.13.0 // indirect
+	golang.org/x/text v0.11.0 // indirect
+	golang.org/x/tools v0.14.0 // indirect
 )

go.sum

@@ -1,35 +1,118 @@
-github.com/alecthomas/repr v0.0.0-20180818092828-117648cd9897 h1:p9Sln00KOTlrYkxI1zYWl1QLnEqAqEARBEYa8FQnQcY=
-github.com/alecthomas/repr v0.0.0-20180818092828-117648cd9897/go.mod h1:xTS7Pm1pD1mvyM075QCDSRqH6qRLXylzS24ZTpRiSzQ=
-github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
+github.com/agext/levenshtein v1.2.1 h1:QmvMAjj2aEICytGiWzmxoE0x2KZvE0fvmqMOfy2tjT8=
+github.com/agext/levenshtein v1.2.1/go.mod h1:JEDfjyjHDjOF/1e4FlBE/PkbqA9OfWu2ki2W0IB5558=
+github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
+github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
+github.com/apparentlymart/go-textseg/v13 v13.0.0 h1:Y+KvPE1NYz0xl601PVImeQfFyEy6iT90AvPUL1NNfNw=
+github.com/apparentlymart/go-textseg/v13 v13.0.0/go.mod h1:ZK2fH7c4NqDTLtiYLvIkEghdlcqw7yxLeM89kiTRPUo=
+github.com/apparentlymart/go-textseg/v15 v15.0.0 h1:uYvfpb3DyLSCGWnctWKGj857c6ew1u1fNQOlOtuGxQY=
+github.com/apparentlymart/go-textseg/v15 v15.0.0/go.mod h1:K8XmNZdhEBkdlyDdvbmmsvpAG721bKi0joRfFdHIWJ4=
+github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de h1:FxWPpzIjnTlhPwqqXc4/vE0f7GvRjuAsbW+HOIe8KnA=
+github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de/go.mod h1:DCaWoUhZrYW9p1lxo/cm8EmUOOzAPSEZNGF2DK1dJgw=
+github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
-github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
-github.com/gookit/color v1.5.2 h1:uLnfXcaFjlrDnQDT+NCBcfhrXqYTx/rcCa6xn01Y8yI=
-github.com/gookit/color v1.5.2/go.mod h1:w8h4bGiHeeBpvQVePTutdbERIUf3oJE5lZ8HM0UgXyg=
-github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
-github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
-github.com/mattn/go-runewidth v0.0.9 h1:Lm995f3rfxdpd6TSmuVCHVb/QhupuXlYr8sCI/QdE+0=
+github.com/glycerine/blake2b v0.0.0-20151022103502-3c8c640cd7be h1:XBJdPGgA3qqhW+p9CANCAVdF7ZIXdu3pZAkypMkKAjE=
+github.com/glycerine/blake2b v0.0.0-20151022103502-3c8c640cd7be/go.mod h1:OSCrScrFAjcBObrulk6BEQlytA462OkG1UGB5NYj9kE=
+github.com/glycerine/goconvey v0.0.0-20190410193231-58a59202ab31 h1:gclg6gY70GLy3PbkQ1AERPfmLMMagS60DKF78eWwLn8=
+github.com/glycerine/goconvey v0.0.0-20190410193231-58a59202ab31/go.mod h1:Ogl1Tioa0aV7gstGFO7KhffUsb9M4ydbEbbxpcEDc24=
+github.com/glycerine/greenpack v5.1.1+incompatible h1:fDr9i6MkSGZmAy4VXPfJhW+SyK2/LNnzIp5nHyDiaIM=
+github.com/glycerine/greenpack v5.1.1+incompatible/go.mod h1:us0jVISAESGjsEuLlAfCd5nkZm6W6WQF18HPuOecIg4=
+github.com/glycerine/liner v0.0.0-20160121172638-72909af234e0 h1:4ZegphJXBTc4uFQ08UVoWYmQXorGa+ipXetUj83sMBc=
+github.com/glycerine/liner v0.0.0-20160121172638-72909af234e0/go.mod h1:AqJLs6UeoC65dnHxyCQ6MO31P5STpjcmgaANAU+No8Q=
+github.com/glycerine/zygomys v5.1.2+incompatible h1:jmcdmA3XPxgfOunAXFpipE9LQoUL6eX6d2mhYyjV4GE=
+github.com/glycerine/zygomys v5.1.2+incompatible/go.mod h1:i3SPKZpmy9dwF/3iWrXJ/ZLyzZucegwypwOmqRkUUaQ=
+github.com/go-test/deep v1.0.3 h1:ZrJSEWsXzPOxaZnFteGEfooLba+ju3FYIbOrS+rQd68=
+github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
+github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
+github.com/gookit/color v1.5.4 h1:FZmqs7XOyGgCAxmWyPslpiok1k05wmY3SJTytgvYFs0=
+github.com/gookit/color v1.5.4/go.mod h1:pZJOeOS8DM43rXbp4AZo1n9zCU2qjpcRko0b6/QJi9w=
+github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
+github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
+github.com/hashicorp/hcl/v2 v2.22.0 h1:hkZ3nCtqeJsDhPRFz5EA9iwcG1hNWGePOTw6oyul12M=
+github.com/hashicorp/hcl/v2 v2.22.0/go.mod h1:62ZYHrXgPoX8xBnzl8QzbWq4dyDsDtfCRgIq1rbJEvA=
+github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
+github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
+github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
+github.com/lithammer/fuzzysearch v1.1.8 h1:/HIuJnjHuXS8bKaiTMeeDlW2/AyIWk2brx1V8LFgLN4=
+github.com/lithammer/fuzzysearch v1.1.8/go.mod h1:IdqeyBClc3FFqSzYq/MXESsS4S0FsZ5ajtkr5xPLts4=
 github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
+github.com/mattn/go-runewidth v0.0.10 h1:CoZ3S2P7pvtP45xOtBw+/mDL2z0RKI576gSkzRRpdGg=
+github.com/mattn/go-runewidth v0.0.10/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
+github.com/mitchellh/go-wordwrap v0.0.0-20150314170334-ad45545899c7 h1:DpOJ2HYzCv8LZP15IdmG+YdwD2luVPHITV96TkirNBM=
+github.com/mitchellh/go-wordwrap v0.0.0-20150314170334-ad45545899c7/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo=
 github.com/olekukonko/tablewriter v0.0.5 h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec=
 github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY=
+github.com/philhofer/fwd v1.1.2 h1:bnDivRJ1EWPjUIRXV5KfORO897HTbpFAQddBdE8t7Gw=
+github.com/philhofer/fwd v1.1.2/go.mod h1:qkPdfjR2SIEbspLqpe1tO4n5yICnr2DY7mqEx2tUTP0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rivo/uniseg v0.1.0 h1:+2KBaVoUmb9XzDsrx/Ct0W/EYOSFf/nWTauy++DprtY=
+github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
-github.com/spf13/cobra v1.5.0 h1:X+jTBEBqF0bHN+9cSMgmfuvv2VHJ9ezmFNf9Y/XstYU=
-github.com/spf13/cobra v1.5.0/go.mod h1:dWXEIy2H428czQCjInthrTRUg7yKbok+2Qi/yBIJoUM=
+github.com/scylladb/termtables v0.0.0-20191203121021-c4c0b6d42ff4/go.mod h1:C1a7PQSMz9NShzorzCiG2fk9+xuCgLkPeCvMHYR2OWg=
+github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636 h1:aSISeOcal5irEhJd1M+IrApc0PdcN7e7Aj4yuEnOrfQ=
+github.com/shurcooL/go v0.0.0-20200502201357-93f07166e636/go.mod h1:TDJrrUr11Vxrven61rcy3hJMUqaf/CLWYhHNPmT14Lk=
+github.com/shurcooL/go-goon v1.0.0 h1:BCQPvxGkHHJ4WpBO4m/9FXbITVIsvAm/T66cCcCGI7E=
+github.com/shurcooL/go-goon v1.0.0/go.mod h1:2wTHMsGo7qnpmqA8ADYZtP4I1DD94JpXGQ3Dxq2YQ5w=
+github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM=
+github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
 github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
-github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
-github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
-github.com/stretchr/testify v1.8.0 h1:pSgiaMZlXftHpm5L7V1+rVB+AZJydKsMxsQBIJw4PKk=
-github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
+github.com/tinylib/msgp v1.1.9 h1:SHf3yoO2sGA0veCJeCBYLHuttAVFHGm2RHgNodW7wQU=
+github.com/tinylib/msgp v1.1.9/go.mod h1:BCXGB54lDD8qUEPmiG0cQQUANC4IUQyB2ItS2UDlO/k=
+github.com/ugorji/go/codec v1.2.11 h1:BMaWp1Bb6fHwEtbplGBGJ498wD+LKlNSl25MjdZY4dU=
+github.com/ugorji/go/codec v1.2.11/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
 github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 h1:QldyIu/L63oPpyvQmHgvgickp1Yw510KJOqX7H24mg8=
 github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778/go.mod h1:2MuV+tbUrU1zIOPMxZ5EncGwgmMJsa+9ucAQZXxsObs=
-golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44 h1:Bli41pIlzTzf3KEY06n+xnzK/BESIg2ze4Pgfh/aI8c=
-golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
+github.com/zclconf/go-cty v1.13.0 h1:It5dfKTTZHe9aeppbNOda3mN7Ag7sg6QkBNm6TkyFa0=
+github.com/zclconf/go-cty v1.13.0/go.mod h1:YKQzy/7pZ7iq2jNFzy5go57xdxdWoLLpaEp4u238AE0=
+github.com/zclconf/go-cty-debug v0.0.0-20240509010212-0d6042c53940 h1:4r45xpDWB6ZMSMNJFMOjqrGHynW3DIBuR2H9j0ug+Mo=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
+golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
+golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
+golang.org/x/mod v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY=
+golang.org/x/mod v0.13.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
+golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
+golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.4.0 h1:zxkM55ReGkDlKSM+Fu41A+zmbZuaPVbGMzvvdUPznYQ=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
+golang.org/x/sys v0.13.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
+golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
+golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
+golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
+golang.org/x/text v0.11.0 h1:LAntKIrcmeSKERyiOh0XMV39LXS8IE9UL2yP7+f5ij4=
+golang.org/x/text v0.11.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
+golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
+golang.org/x/tools v0.14.0 h1:jvNa2pY0M4r62jkRQ6RwEZZyPcymeL9XZMLBbV7U2nc=
+golang.org/x/tools v0.14.0/go.mod h1:uYBEerGOWcJyEORxN+Ek8+TT266gXkNlHdJBwexUsBg=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

lib/common.go

@@ -1,5 +1,5 @@
 /*
-Copyright © 2022 Thomas von Dein
+Copyright © 2022-2024 Thomas von Dein
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
@@ -17,73 +17,20 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 
 package lib
 
-import (
-	"github.com/gookit/color"
-	//"github.com/xo/terminfo"
-)
-
-var (
-	// command line flags
-	Debug            bool
-	XtendedOut       bool
-	NoNumbering      bool
-	ShowVersion      bool
-	Columns          string
-	UseColumns       []int
-	DefaultSeparator string = `(\s\s+|\t)`
-	Separator        string = `(\s\s+|\t)`
-	OutflagExtended  bool
-	OutflagMarkdown  bool
-	OutflagOrgtable  bool
-	OutflagShell     bool
-	OutputMode       string
-	InvertMatch      bool
-	Pattern          string
-
-	/*
-	 FIXME: make configurable somehow, config file or ENV
-	 see https://github.com/gookit/color will be set by
-	 io.ProcessFiles() according to currently supported
-	 color mode.
-	*/
-	MatchFG string
-	MatchBG string
-	NoColor bool
-
-	// colors to be used per supported color mode
-	Colors = map[color.Level]map[string]string{
-		color.Level16: {
-			"bg": "green", "fg": "black",
-		},
-		color.Level256: {
-			"bg": "lightGreen", "fg": "black",
-		},
-		color.LevelRgb: {
-			// FIXME: maybe use something nicer
-			"bg": "lightGreen", "fg": "black",
-		},
-	}
-
-	// used for validation
-	validOutputmodes = "(orgtbl|markdown|extended|ascii)"
-
-	// main program version
-	Version = "v1.0.9"
-
-	// generated  version string, used  by -v contains  lib.Version on
-	//  main  branch,   and  lib.Version-$branch-$lastcommit-$date  on
-	// development branch
-	VERSION string
-
-	// sorting
-	SortByColumn int
-)
-
 // contains a whole parsed table
 type Tabdata struct {
 	maxwidthHeader int      // longest header
-	maxwidthPerCol []int    // max width per column
 	columns        int      // count
 	headers        []string // [ "ID", "NAME", ...]
 	entries        [][]string
 }
+
+func (data *Tabdata) CloneEmpty() Tabdata {
+	newdata := Tabdata{
+		maxwidthHeader: data.maxwidthHeader,
+		columns:        data.columns,
+		headers:        data.headers,
+	}
+
+	return newdata
+}

lib/filter.go

@@ -0,0 +1,130 @@
+/*
+Copyright © 2022-2024 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package lib
+
+import (
+	"bufio"
+	"fmt"
+	"io"
+	"strings"
+
+	"github.com/lithammer/fuzzysearch/fuzzy"
+	"github.com/tlinden/tablizer/cfg"
+)
+
+/*
+ * [!]Match a  line, use fuzzy  search for normal pattern  strings and
+ * regexp otherwise.
+ */
+func matchPattern(conf cfg.Config, line string) bool {
+	if conf.UseFuzzySearch {
+		return fuzzy.MatchFold(conf.Pattern, line)
+	}
+
+	return conf.PatternR.MatchString(line)
+}
+
+/*
+ * Filter parsed data by fields. The  filter is positive, so if one or
+ * more filters match on a row, it  will be kept, otherwise it will be
+ * excluded.
+ */
+func FilterByFields(conf cfg.Config, data Tabdata) (Tabdata, bool, error) {
+	if len(conf.Filters) == 0 {
+		// no filters, no checking
+		return Tabdata{}, false, nil
+	}
+
+	newdata := data.CloneEmpty()
+
+	for _, row := range data.entries {
+		keep := true
+
+		for idx, header := range data.headers {
+			if !Exists(conf.Filters, strings.ToLower(header)) {
+				// do not filter by unspecified field
+				continue
+			}
+
+			if !conf.Filters[strings.ToLower(header)].MatchString(row[idx]) {
+				// there IS a filter, but it doesn't match
+				keep = false
+
+				break
+			}
+		}
+
+		if keep == !conf.InvertMatch {
+			// also apply -v
+			newdata.entries = append(newdata.entries, row)
+		}
+	}
+
+	return newdata, true, nil
+}
+
+/* generic map.Exists(key) */
+func Exists[K comparable, V any](m map[K]V, v K) bool {
+	if _, ok := m[v]; ok {
+		return true
+	}
+
+	return false
+}
+
+func FilterByPattern(conf cfg.Config, input io.Reader) (io.Reader, error) {
+	if conf.Pattern == "" {
+		return input, nil
+	}
+
+	scanner := bufio.NewScanner(input)
+	lines := []string{}
+	hadFirst := false
+
+	for scanner.Scan() {
+		line := strings.TrimSpace(scanner.Text())
+		if hadFirst {
+			// don't match 1st line, it's the header
+			if conf.Pattern != "" && matchPattern(conf, line) == conf.InvertMatch {
+				// by default  -v is false, so if a  line does NOT
+				// match the pattern, we will ignore it. However,
+				// if the user specified -v, the matching is inverted,
+				// so we ignore all lines, which DO match.
+				continue
+			}
+
+			// apply user defined lisp filters, if any
+			accept, err := RunFilterHooks(conf, line)
+			if err != nil {
+				return input, fmt.Errorf("failed to apply filter hook: %w", err)
+			}
+
+			if !accept {
+				//  IF there  are filter  hook[s] and  IF one  of them
+				// returns false on the current line, reject it
+				continue
+			}
+		}
+
+		lines = append(lines, line)
+
+		hadFirst = true
+	}
+
+	return strings.NewReader(strings.Join(lines, "\n")), nil
+}

lib/filter_test.go

@@ -0,0 +1,162 @@
+/*
+Copyright © 2024 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package lib
+
+import (
+	"fmt"
+	"reflect"
+	"testing"
+
+	"github.com/tlinden/tablizer/cfg"
+)
+
+func TestMatchPattern(t *testing.T) {
+	var input = []struct {
+		name    string
+		fuzzy   bool
+		pattern string
+		line    string
+	}{
+		{
+			name:    "normal",
+			pattern: "haus",
+			line:    "hausparty",
+		},
+		{
+			name:    "fuzzy",
+			pattern: "hpt",
+			line:    "haus-party-termin",
+			fuzzy:   true,
+		},
+	}
+
+	for _, inputdata := range input {
+		testname := fmt.Sprintf("match-pattern-%s", inputdata.name)
+
+		t.Run(testname, func(t *testing.T) {
+			conf := cfg.Config{}
+
+			if inputdata.fuzzy {
+				conf.UseFuzzySearch = true
+			}
+
+			err := conf.PreparePattern(inputdata.pattern)
+			if err != nil {
+				t.Errorf("PreparePattern returned error: %s", err)
+			}
+
+			if !matchPattern(conf, inputdata.line) {
+				t.Errorf("matchPattern() did not match\nExp: true\nGot: false\n")
+			}
+		})
+	}
+}
+
+func TestFilterByFields(t *testing.T) {
+	data := Tabdata{
+		headers: []string{
+			"ONE", "TWO", "THREE",
+		},
+		entries: [][]string{
+			{"asd", "igig", "cxxxncnc"},
+			{"19191", "EDD 1", "x"},
+			{"8d8", "AN 1", "y"},
+		},
+	}
+
+	var input = []struct {
+		name   string
+		filter []string
+		expect Tabdata
+		invert bool
+	}{
+		{
+			name:   "one-field",
+			filter: []string{"one=19"},
+			expect: Tabdata{
+				headers: []string{
+					"ONE", "TWO", "THREE",
+				},
+				entries: [][]string{
+					{"19191", "EDD 1", "x"},
+				},
+			},
+		},
+
+		{
+			name:   "one-field-inverted",
+			filter: []string{"one=19"},
+			invert: true,
+			expect: Tabdata{
+				headers: []string{
+					"ONE", "TWO", "THREE",
+				},
+				entries: [][]string{
+					{"asd", "igig", "cxxxncnc"},
+					{"8d8", "AN 1", "y"},
+				},
+			},
+		},
+
+		{
+			name:   "many-fields",
+			filter: []string{"one=19", "two=DD"},
+			expect: Tabdata{
+				headers: []string{
+					"ONE", "TWO", "THREE",
+				},
+				entries: [][]string{
+					{"19191", "EDD 1", "x"},
+				},
+			},
+		},
+
+		{
+			name:   "many-fields-inverted",
+			filter: []string{"one=19", "two=DD"},
+			invert: true,
+			expect: Tabdata{
+				headers: []string{
+					"ONE", "TWO", "THREE",
+				},
+				entries: [][]string{
+					{"asd", "igig", "cxxxncnc"},
+					{"8d8", "AN 1", "y"},
+				},
+			},
+		},
+	}
+
+	for _, inputdata := range input {
+		testname := fmt.Sprintf("filter-by-fields-%s", inputdata.name)
+
+		t.Run(testname, func(t *testing.T) {
+			conf := cfg.Config{Rawfilters: inputdata.filter, InvertMatch: inputdata.invert}
+
+			err := conf.PrepareFilters()
+			if err != nil {
+				t.Errorf("PrepareFilters returned error: %s", err)
+			}
+
+			data, _, _ := FilterByFields(conf, data)
+			if !reflect.DeepEqual(data, inputdata.expect) {
+				t.Errorf("Filtered data does not match expected data:\ngot: %+v\nexp: %+v", data, inputdata.expect)
+			}
+		})
+	}
+}

lib/helpers.go

@@ -20,11 +20,14 @@ package lib
 import (
 	"errors"
 	"fmt"
-	"github.com/gookit/color"
 	"os"
 	"regexp"
+	"sort"
 	"strconv"
 	"strings"
+
+	"github.com/gookit/color"
+	"github.com/tlinden/tablizer/cfg"
 )
 
 func contains(s []int, e int) bool {
@@ -33,42 +36,101 @@ func contains(s []int, e int) bool {
 			return true
 		}
 	}
+
 	return false
 }
 
-func PrepareColumns() error {
-	if len(Columns) > 0 {
-		for _, use := range strings.Split(Columns, ",") {
-			usenum, err := strconv.Atoi(use)
-			if err != nil {
-				msg := fmt.Sprintf("Could not parse columns list %s: %v", Columns, err)
-				return errors.New(msg)
-			}
-			UseColumns = append(UseColumns, usenum)
+// validate the consitency of parsed data
+func ValidateConsistency(data *Tabdata) error {
+	expectedfields := len(data.headers)
+
+	for idx, row := range data.entries {
+		if len(row) != expectedfields {
+			return fmt.Errorf("row %d does not contain expected %d elements, but %d",
+				idx, expectedfields, len(row))
 		}
 	}
+
 	return nil
 }
 
-func numberizeHeaders(data *Tabdata) {
-	// prepare headers: add numbers to headers
+// parse columns list given  with -c, modifies config.UseColumns based
+// on eventually given regex
+func PrepareColumns(conf *cfg.Config, data *Tabdata) error {
+	if conf.Columns == "" {
+		return nil
+	}
+
+	for _, use := range strings.Split(conf.Columns, ",") {
+		if len(use) == 0 {
+			return fmt.Errorf("could not parse columns list %s: empty column", conf.Columns)
+		}
+
+		usenum, err := strconv.Atoi(use)
+		if err != nil {
+			// might be a regexp
+			colPattern, err := regexp.Compile(use)
+			if err != nil {
+				msg := fmt.Sprintf("Could not parse columns list %s: %v", conf.Columns, err)
+
+				return errors.New(msg)
+			}
+
+			// find matching header fields
+			for i, head := range data.headers {
+				if colPattern.MatchString(head) {
+					conf.UseColumns = append(conf.UseColumns, i+1)
+				}
+			}
+		} else {
+			// we digress from go  best practises here, because if
+			// a colum spec is not a number, we process them above
+			// inside the err handler  for atoi(). so only add the
+			// number, if it's really just a number.
+			conf.UseColumns = append(conf.UseColumns, usenum)
+		}
+	}
+
+	// deduplicate: put all values into a map (value gets map key)
+	// thereby  removing duplicates,  extract keys into  new slice
+	// and sort it
+	imap := make(map[int]int, len(conf.UseColumns))
+	for _, i := range conf.UseColumns {
+		imap[i] = 0
+	}
+
+	conf.UseColumns = nil
+
+	for k := range imap {
+		conf.UseColumns = append(conf.UseColumns, k)
+	}
+
+	sort.Ints(conf.UseColumns)
+
+	return nil
+}
+
+// prepare headers: add numbers to headers
+func numberizeAndReduceHeaders(conf cfg.Config, data *Tabdata) {
 	numberedHeaders := []string{}
 	maxwidth := 0 // start from scratch, so we only look at displayed column widths
 
-	for i, head := range data.headers {
-		headlen := 0
-		if len(Columns) > 0 {
+	for idx, head := range data.headers {
+		var headlen int
+
+		if len(conf.Columns) > 0 {
 			// -c specified
-			if !contains(UseColumns, i+1) {
+			if !contains(conf.UseColumns, idx+1) {
 				// ignore this one
 				continue
 			}
 		}
-		if NoNumbering {
+
+		if conf.NoNumbering {
 			numberedHeaders = append(numberedHeaders, head)
 			headlen = len(head)
 		} else {
-			numhead := fmt.Sprintf("%s(%d)", head, i+1)
+			numhead := fmt.Sprintf("%s(%d)", head, idx+1)
 			headlen = len(numhead)
 			numberedHeaders = append(numberedHeaders, numhead)
 		}
@@ -77,89 +139,94 @@ func numberizeHeaders(data *Tabdata) {
 			maxwidth = headlen
 		}
 	}
+
 	data.headers = numberedHeaders
+
 	if data.maxwidthHeader != maxwidth && maxwidth > 0 {
 		data.maxwidthHeader = maxwidth
 	}
 }
 
-func reduceColumns(data *Tabdata) {
-	// exclude columns, if any
-	if len(Columns) > 0 {
+// exclude columns, if any
+func reduceColumns(conf cfg.Config, data *Tabdata) {
+	if len(conf.Columns) > 0 {
 		reducedEntries := [][]string{}
+
 		var reducedEntry []string
+
 		for _, entry := range data.entries {
 			reducedEntry = nil
+
 			for i, value := range entry {
-				if !contains(UseColumns, i+1) {
+				if !contains(conf.UseColumns, i+1) {
 					continue
 				}
 
 				reducedEntry = append(reducedEntry, value)
 			}
+
 			reducedEntries = append(reducedEntries, reducedEntry)
 		}
+
 		data.entries = reducedEntries
 	}
 }
 
-func PrepareModeFlags() error {
-	if len(OutputMode) == 0 {
-		// associate short flags like -X with mode selector
-		switch {
-		case OutflagExtended:
-			OutputMode = "extended"
-		case OutflagMarkdown:
-			OutputMode = "markdown"
-		case OutflagOrgtable:
-			OutputMode = "orgtbl"
-		case OutflagShell:
-			OutputMode = "shell"
-			NoNumbering = true
-		default:
-			OutputMode = "ascii"
-		}
-	} else {
-		r, err := regexp.Compile(validOutputmodes)
-
-		if err != nil {
-			return errors.New("Failed to validate output mode spec!")
-		}
-
-		match := r.MatchString(OutputMode)
-
-		if !match {
-			return errors.New("Invalid output mode!")
-		}
-	}
-
-	return nil
-}
-
+// FIXME: remove this when we only use Tablewriter and strip in ParseFile()!
 func trimRow(row []string) []string {
-	// FIXME: remove this when we only use Tablewriter and strip in ParseFile()!
-	var fixedrow []string
-	for _, cell := range row {
-		fixedrow = append(fixedrow, strings.TrimSpace(cell))
+	var fixedrow = make([]string, len(row))
+
+	for idx, cell := range row {
+		fixedrow[idx] = strings.TrimSpace(cell)
 	}
 
 	return fixedrow
 }
 
-func colorizeData(output string) string {
-	if len(Pattern) > 0 && !NoColor && color.IsConsole(os.Stdout) {
-		r := regexp.MustCompile("(" + Pattern + ")")
-		return r.ReplaceAllString(output, "<bg="+MatchBG+";fg="+MatchFG+">$1</>")
-	} else {
+// FIXME: refactor this beast!
+func colorizeData(conf cfg.Config, output string) string {
+	switch {
+	case conf.UseHighlight && color.IsConsole(os.Stdout):
+		highlight := true
+		colorized := ""
+		first := true
+
+		for _, line := range strings.Split(output, "\n") {
+			if highlight {
+				if first {
+					// we  need to add  two spaces to the  header line
+					//  because tablewriter omits them for some reason
+					//  in pprint mode. This doesn't matter as long as
+					//  we don't use colorization. But with colors the
+					// missing spaces can be seen.
+					if conf.OutputMode == cfg.ASCII {
+						line += "  "
+					}
+
+					line = conf.HighlightHdrStyle.Sprint(line)
+					first = false
+				} else {
+					line = conf.HighlightStyle.Sprint(line)
+				}
+			} else {
+				line = conf.NoHighlightStyle.Sprint(line)
+			}
+
+			highlight = !highlight
+
+			colorized += line + "\n"
+		}
+
+		return colorized
+
+	case len(conf.Pattern) > 0 && !conf.NoColor && color.IsConsole(os.Stdout):
+		r := regexp.MustCompile("(" + conf.Pattern + ")")
+
+		return r.ReplaceAllStringFunc(output, func(in string) string {
+			return conf.ColorStyle.Sprint(in)
+		})
+
+	default:
 		return output
 	}
 }
-
-func isTerminal(f *os.File) bool {
-	o, _ := f.Stat()
-	if (o.Mode() & os.ModeCharDevice) == os.ModeCharDevice {
-		return true
-	} else {
-		return false
-	}
-}

lib/helpers_test.go

@@ -21,9 +21,11 @@ import (
 	"fmt"
 	"reflect"
 	"testing"
+
+	"github.com/tlinden/tablizer/cfg"
 )
 
-func Testcontains(t *testing.T) {
+func TestContains(t *testing.T) {
 	var tests = []struct {
 		list   []int
 		search int
@@ -45,6 +47,19 @@ func Testcontains(t *testing.T) {
 }
 
 func TestPrepareColumns(t *testing.T) {
+	data := Tabdata{
+		maxwidthHeader: 5,
+		columns:        3,
+		headers: []string{
+			"ONE", "TWO", "THREE",
+		},
+		entries: [][]string{
+			{
+				"2", "3", "4",
+			},
+		},
+	}
+
 	var tests = []struct {
 		input     string
 		exp       []int
@@ -52,21 +67,23 @@ func TestPrepareColumns(t *testing.T) {
 	}{
 		{"1,2,3", []int{1, 2, 3}, false},
 		{"1,2,", []int{}, true},
-		{"a,b", []int{}, true},
+		{"T", []int{2, 3}, false},
+		{"T,2,3", []int{2, 3}, false},
+		{"[a-z,4,5", []int{4, 5}, true}, // invalid regexp
 	}
 
-	for _, tt := range tests {
-		testname := fmt.Sprintf("PrepareColumns-%s-%t", tt.input, tt.wanterror)
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("PrepareColumns-%s-%t", testdata.input, testdata.wanterror)
 		t.Run(testname, func(t *testing.T) {
-			Columns = tt.input
-			err := PrepareColumns()
+			conf := cfg.Config{Columns: testdata.input}
+			err := PrepareColumns(&conf, &data)
 			if err != nil {
-				if !tt.wanterror {
+				if !testdata.wanterror {
 					t.Errorf("got error: %v", err)
 				}
 			} else {
-				if !reflect.DeepEqual(UseColumns, tt.exp) {
-					t.Errorf("got: %v, expected: %v", UseColumns, tt.exp)
+				if !reflect.DeepEqual(conf.UseColumns, testdata.exp) {
+					t.Errorf("got: %v, expected: %v", conf.UseColumns, testdata.exp)
 				}
 			}
 		})
@@ -98,20 +115,48 @@ func TestReduceColumns(t *testing.T) {
 
 	input := [][]string{{"a", "b", "c"}}
 
-	Columns = "y" // used as a flag with len(Columns)...
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("reduce-columns-by-%+v", testdata.columns)
 
-	for _, tt := range tests {
-		testname := fmt.Sprintf("reduce-columns-by-%+v", tt.columns)
 		t.Run(testname, func(t *testing.T) {
-			UseColumns = tt.columns
+			c := cfg.Config{Columns: "x", UseColumns: testdata.columns}
 			data := Tabdata{entries: input}
-			reduceColumns(&data)
-			if !reflect.DeepEqual(data.entries, tt.expect) {
-				t.Errorf("reduceColumns returned invalid data:\ngot: %+v\nexp: %+v", data.entries, tt.expect)
+			reduceColumns(c, &data)
+			if !reflect.DeepEqual(data.entries, testdata.expect) {
+				t.Errorf("reduceColumns returned invalid data:\ngot: %+v\nexp: %+v",
+					data.entries, testdata.expect)
+			}
+		})
+	}
+}
+
+func TestNumberizeHeaders(t *testing.T) {
+	data := Tabdata{
+		headers: []string{"ONE", "TWO", "THREE"},
+	}
+
+	var tests = []struct {
+		expect  []string
+		columns []int
+		nonum   bool
+	}{
+		{[]string{"ONE(1)", "TWO(2)", "THREE(3)"}, []int{1, 2, 3}, false},
+		{[]string{"ONE(1)", "TWO(2)"}, []int{1, 2}, false},
+		{[]string{"ONE", "TWO"}, []int{1, 2}, true},
+	}
+
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("numberize-headers-columns-%+v-nonum-%t",
+			testdata.columns, testdata.nonum)
+
+		t.Run(testname, func(t *testing.T) {
+			conf := cfg.Config{Columns: "x", UseColumns: testdata.columns, NoNumbering: testdata.nonum}
+			usedata := data
+			numberizeAndReduceHeaders(conf, &usedata)
+			if !reflect.DeepEqual(usedata.headers, testdata.expect) {
+				t.Errorf("numberizeAndReduceHeaders returned invalid data:\ngot: %+v\nexp: %+v",
+					usedata.headers, testdata.expect)
 			}
 		})
 	}
-
-	Columns = "" // reset for other tests
-	UseColumns = nil
 }

lib/io.go

@@ -19,75 +19,100 @@ package lib
 
 import (
 	"errors"
-	"github.com/gookit/color"
+	"fmt"
 	"io"
 	"os"
+
+	"github.com/tlinden/tablizer/cfg"
 )
 
-func ProcessFiles(args []string) error {
-	fds, pattern, err := determineIO(args)
+const RWRR = 0755
 
-	if !isTerminal(os.Stdout) {
-		color.Disable()
-	} else {
-		level := color.TermColorLevel()
-		MatchFG = Colors[level]["fg"]
-		MatchBG = Colors[level]["bg"]
-	}
+func ProcessFiles(conf *cfg.Config, args []string) error {
+	fds, pattern, err := determineIO(conf, args)
 
 	if err != nil {
 		return err
 	}
 
+	if err := conf.PreparePattern(pattern); err != nil {
+		return err
+	}
+
 	for _, fd := range fds {
-		data, err := parseFile(fd, pattern)
+		data, err := Parse(*conf, fd)
 		if err != nil {
 			return err
 		}
-		printData(&data)
+
+		if err = ValidateConsistency(&data); err != nil {
+			return err
+		}
+
+		err = PrepareColumns(conf, &data)
+		if err != nil {
+			return err
+		}
+
+		printData(os.Stdout, *conf, &data)
 	}
 
 	return nil
 }
 
-func determineIO(args []string) ([]io.Reader, string, error) {
-	var pattern string
-	var fds []io.Reader
-	var havefiles bool
+func determineIO(conf *cfg.Config, args []string) ([]io.Reader, string, error) {
+	var filehandles []io.Reader
 
-	if len(args) > 0 {
-		// threre were args left, take a look
-		if _, err := os.Stat(args[0]); err != nil {
-			// first  one is  not a  file, consider  it as  regexp and
-			// shift arg list
-			pattern = args[0]
-			Pattern = args[0] // FIXME
-			args = args[1:]
-		}
+	var pattern string
+
+	var haveio bool
+
+	stat, _ := os.Stdin.Stat()
+	if (stat.Mode() & os.ModeCharDevice) == 0 {
+		// we're reading from STDIN, which takes precedence over file args
+		filehandles = append(filehandles, os.Stdin)
 
 		if len(args) > 0 {
-			// only files
-			for _, file := range args {
-				fd, err := os.OpenFile(file, os.O_RDONLY, 0755)
-
-				if err != nil {
-					return nil, "", err
-				}
-
-				fds = append(fds, fd)
-			}
-			havefiles = true
+			// ignore any args > 1
+			pattern = args[0]
+			conf.Pattern = args[0] // used for colorization by printData()
 		}
-	}
 
-	if !havefiles {
-		stat, _ := os.Stdin.Stat()
-		if (stat.Mode() & os.ModeCharDevice) == 0 {
-			fds = append(fds, os.Stdin)
+		haveio = true
+	} else if len(args) > 0 {
+		// threre were args left, take a look
+		if args[0] == "-" {
+			// in traditional unix programs a dash denotes STDIN (forced)
+			filehandles = append(filehandles, os.Stdin)
+			haveio = true
 		} else {
-			return nil, "", errors.New("No file specified and nothing to read on stdin!")
+			if _, err := os.Stat(args[0]); err != nil {
+				// first  one is  not a  file, consider  it as  regexp and
+				// shift arg list
+				pattern = args[0]
+				conf.Pattern = args[0] // used for colorization by printData()
+				args = args[1:]
+			}
+
+			if len(args) > 0 {
+				// consider any other args as files
+				for _, file := range args {
+					filehandle, err := os.OpenFile(file, os.O_RDONLY, RWRR)
+
+					if err != nil {
+						return nil, "", fmt.Errorf("failed to read input file %s: %w", file, err)
+					}
+
+					filehandles = append(filehandles, filehandle)
+					haveio = true
+				}
+			}
 		}
 	}
 
-	return fds, pattern, nil
+	if !haveio {
+		return nil, "", errors.New("no file specified and nothing to read on stdin")
+	}
+
+	return filehandles, pattern, nil
 }

lib/lisp.go

@@ -0,0 +1,313 @@
+/*
+Copyright © 2023 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package lib
+
+import (
+	"errors"
+	"fmt"
+	"log"
+	"os"
+	"strings"
+
+	"github.com/glycerine/zygomys/zygo"
+	"github.com/tlinden/tablizer/cfg"
+)
+
+/*
+needs to be global because we can't feed an cfg object to AddHook()
+which is being called from user lisp code
+*/
+var Hooks map[string][]*zygo.SexpSymbol
+
+/*
+AddHook() (called addhook from lisp code)  can be used by the user to
+add a function to one of the available hooks provided by tablizer.
+*/
+func AddHook(env *zygo.Zlisp, name string, args []zygo.Sexp) (zygo.Sexp, error) {
+	var hookname string
+
+	if len(args) < 2 {
+		return zygo.SexpNull, errors.New("argument of %add-hook should be: %hook-name %your-function")
+	}
+
+	switch sexptype := args[0].(type) {
+	case *zygo.SexpSymbol:
+		if !HookExists(sexptype.Name()) {
+			return zygo.SexpNull, errors.New("Unknown hook " + sexptype.Name())
+		}
+
+		hookname = sexptype.Name()
+
+	default:
+		return zygo.SexpNull, errors.New("hook name must be a symbol ")
+	}
+
+	switch sexptype := args[1].(type) {
+	case *zygo.SexpSymbol:
+		_, exists := Hooks[hookname]
+		if !exists {
+			Hooks[hookname] = []*zygo.SexpSymbol{sexptype}
+		} else {
+			Hooks[hookname] = append(Hooks[hookname], sexptype)
+		}
+
+	default:
+		return zygo.SexpNull, errors.New("hook function must be a symbol ")
+	}
+
+	return zygo.SexpNull, nil
+}
+
+/*
+Check if a hook exists
+*/
+func HookExists(key string) bool {
+	for _, hook := range cfg.ValidHooks {
+		if hook == key {
+			return true
+		}
+	}
+
+	return false
+}
+
+/*
+ * Basic sanity checks and load lisp file
+ */
+func LoadAndEvalFile(env *zygo.Zlisp, path string) error {
+	if strings.HasSuffix(path, `.zy`) {
+		code, err := os.ReadFile(path)
+		if err != nil {
+			return fmt.Errorf("failed to read lisp file %s: %w", path, err)
+		}
+
+		// FIXME: check what res (_ here) could be and mean
+		_, err = env.EvalString(string(code))
+		if err != nil {
+			log.Fatal(env.GetStackTrace(err))
+		}
+	}
+
+	return nil
+}
+
+/*
+ * Setup lisp interpreter environment
+ */
+func SetupLisp(conf *cfg.Config) error {
+	// iterate over load-path and evaluate all *.zy files there, if any
+	// we ignore if load-path does not exist, which is the default anyway
+	path, err := os.Stat(conf.LispLoadPath)
+	if os.IsNotExist(err) {
+		return nil
+	}
+
+	// init global hooks
+	Hooks = make(map[string][]*zygo.SexpSymbol)
+
+	// init sandbox
+	env := zygo.NewZlispSandbox()
+	env.AddFunction("addhook", AddHook)
+
+	if !path.IsDir() {
+		// load single lisp file
+		err = LoadAndEvalFile(env, conf.LispLoadPath)
+		if err != nil {
+			return err
+		}
+	} else {
+		// load all lisp file in load dir
+		dir, err := os.ReadDir(conf.LispLoadPath)
+		if err != nil {
+			return fmt.Errorf("failed to read lisp dir %s: %w",
+				conf.LispLoadPath, err)
+		}
+
+		for _, entry := range dir {
+			if !entry.IsDir() {
+				err := LoadAndEvalFile(env, conf.LispLoadPath+"/"+entry.Name())
+				if err != nil {
+					return err
+				}
+			}
+		}
+	}
+
+	RegisterLib(env)
+
+	conf.Lisp = env
+
+	return nil
+}
+
+/*
+Execute every user lisp function registered as filter hook.
+
+Each function is given the current line as argument and is expected to
+return a boolean.   True indicates to keep the line,  false to skip
+it.
+
+If there  are multiple such  functions registered, then the  first one
+returning false wins,  that is if each function returns  true the line
+will  be kept,  if at  least one  of them  returns false,  it will  be
+skipped.
+*/
+func RunFilterHooks(conf cfg.Config, line string) (bool, error) {
+	for _, hook := range Hooks["filter"] {
+		var result bool
+
+		conf.Lisp.Clear()
+
+		res, err := conf.Lisp.EvalString(fmt.Sprintf("(%s `%s`)", hook.Name(), line))
+		if err != nil {
+			return false, fmt.Errorf("failed to evaluate hook loader: %w", err)
+		}
+
+		switch sexptype := res.(type) {
+		case *zygo.SexpBool:
+			result = sexptype.Val
+		default:
+			return false, fmt.Errorf("filter hook shall return bool")
+		}
+
+		if !result {
+			// the first hook which returns false leads to complete false
+			return result, nil
+		}
+	}
+
+	// if no hook returned false, we succeed and accept the given line
+	return true, nil
+}
+
+/*
+These hooks get the data (Tabdata) readily processed by tablizer as
+argument. They are expected to  return a SexpPair containing a boolean
+denoting  if  the data  has  been  modified  and the  actual  modified
+data. Columns must  be the same, rows may differ.  Cells may also have
+been modified.
+
+Replaces the  internal data structure  Tabdata with the  user supplied
+version.
+
+Only one process hook function is supported.
+
+The somewhat complicated code is being  caused by the fact, that we
+need to convert our internal structure  to a lisp variable and vice
+versa afterwards.
+*/
+func RunProcessHooks(conf cfg.Config, data Tabdata) (Tabdata, bool, error) {
+	var userdata Tabdata
+
+	lisplist := []zygo.Sexp{}
+
+	if len(Hooks["process"]) == 0 {
+		return userdata, false, nil
+	}
+
+	if len(Hooks["process"]) > 1 {
+		fmt.Println("Warning: only one process hook is allowed!")
+	}
+
+	// there are hook[s] installed, convert the go data structure 'data to lisp
+	for _, row := range data.entries {
+		var entry zygo.SexpHash
+
+		for idx, cell := range row {
+			err := entry.HashSet(&zygo.SexpStr{S: data.headers[idx]}, &zygo.SexpStr{S: cell})
+			if err != nil {
+				return userdata, false, fmt.Errorf("failed to convert to lisp data: %w", err)
+			}
+		}
+
+		lisplist = append(lisplist, &entry)
+	}
+
+	// we need to add it to the env so that the function can use the struct directly
+	conf.Lisp.AddGlobal("data", &zygo.SexpArray{Val: lisplist, Env: conf.Lisp})
+
+	// execute the actual hook
+	hook := Hooks["process"][0]
+
+	conf.Lisp.Clear()
+
+	var result bool
+
+	res, err := conf.Lisp.EvalString(fmt.Sprintf("(%s data)", hook.Name()))
+	if err != nil {
+		return userdata, false, fmt.Errorf("failed to eval lisp loader: %w", err)
+	}
+
+	// we expect (bool, array(hash)) as return from the function
+	switch sexptype := res.(type) {
+	case *zygo.SexpPair:
+		switch th := sexptype.Head.(type) {
+		case *zygo.SexpBool:
+			result = th.Val
+		default:
+			return userdata, false, errors.New("xpect (bool, array(hash)) as return value")
+		}
+
+		switch sexptailtype := sexptype.Tail.(type) {
+		case *zygo.SexpArray:
+			lisplist = sexptailtype.Val
+		default:
+			return userdata, false, errors.New("expect (bool, array(hash)) as return value ")
+		}
+	default:
+		return userdata, false, errors.New("filter hook shall return array of hashes ")
+	}
+
+	if !result {
+		// no further processing required
+		return userdata, result, nil
+	}
+
+	// finally convert lispdata back to Tabdata
+	for _, item := range lisplist {
+		row := []string{}
+
+		switch hash := item.(type) {
+		case *zygo.SexpHash:
+			for _, header := range data.headers {
+				entry, err := hash.HashGetDefault(
+					conf.Lisp,
+					&zygo.SexpStr{S: header},
+					&zygo.SexpStr{S: ""})
+				if err != nil {
+					return userdata, false, fmt.Errorf("failed to get lisp hash entry: %w", err)
+				}
+
+				switch sexptype := entry.(type) {
+				case *zygo.SexpStr:
+					row = append(row, sexptype.S)
+				default:
+					return userdata, false, errors.New("hsh values should be string ")
+				}
+			}
+		default:
+			return userdata, false, errors.New("rturned array should contain hashes ")
+		}
+
+		userdata.entries = append(userdata.entries, row)
+	}
+
+	userdata.headers = data.headers
+
+	return userdata, result, nil
+}

lib/lisplib.go

@@ -0,0 +1,88 @@
+/*
+Copyright © 2023 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package lib
+
+import (
+	"errors"
+	"fmt"
+	"regexp"
+	"strconv"
+
+	"github.com/glycerine/zygomys/zygo"
+)
+
+func Splice2SexpList(list []string) zygo.Sexp {
+	slist := []zygo.Sexp{}
+
+	for _, item := range list {
+		slist = append(slist, &zygo.SexpStr{S: item})
+	}
+
+	return zygo.MakeList(slist)
+}
+
+func StringReSplit(env *zygo.Zlisp, name string, args []zygo.Sexp) (zygo.Sexp, error) {
+	if len(args) < 2 {
+		return zygo.SexpNull, errors.New("expecting 2 arguments")
+	}
+
+	var separator, input string
+
+	switch t := args[0].(type) {
+	case *zygo.SexpStr:
+		input = t.S
+	default:
+		return zygo.SexpNull, errors.New("second argument must be a string")
+	}
+
+	switch t := args[1].(type) {
+	case *zygo.SexpStr:
+		separator = t.S
+	default:
+		return zygo.SexpNull, errors.New("first argument must be a string")
+	}
+
+	sep := regexp.MustCompile(separator)
+
+	return Splice2SexpList(sep.Split(input, -1)), nil
+}
+
+func String2Int(env *zygo.Zlisp, name string, args []zygo.Sexp) (zygo.Sexp, error) {
+	var number int
+
+	switch t := args[0].(type) {
+	case *zygo.SexpStr:
+		num, err := strconv.Atoi(t.S)
+
+		if err != nil {
+			return zygo.SexpNull, fmt.Errorf("failed to convert string to number: %w", err)
+		}
+
+		number = num
+
+	default:
+		return zygo.SexpNull, errors.New("argument must be a string")
+	}
+
+	return &zygo.SexpInt{Val: int64(number)}, nil
+}
+
+func RegisterLib(env *zygo.Zlisp) {
+	env.AddFunction("resplit", StringReSplit)
+	env.AddFunction("atoi", String2Int)
+}

lib/parser.go

@@ -1,5 +1,5 @@
 /*
-Copyright © 2022 Thomas von Dein
+Copyright © 2022-2024 Thomas von Dein
 
 This program is free software: you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by
@@ -19,28 +19,87 @@ package lib
 
 import (
 	"bufio"
-	"errors"
+	"encoding/csv"
 	"fmt"
-	"github.com/alecthomas/repr"
 	"io"
 	"regexp"
 	"strings"
+
+	"github.com/alecthomas/repr"
+	"github.com/tlinden/tablizer/cfg"
 )
 
 /*
-   Parse tabular input.
+Parser switch
 */
-func parseFile(input io.Reader, pattern string) (Tabdata, error) {
+func Parse(conf cfg.Config, input io.Reader) (Tabdata, error) {
+	if len(conf.Separator) == 1 {
+		return parseCSV(conf, input)
+	}
+
+	return parseTabular(conf, input)
+}
+
+/*
+Parse CSV input.
+*/
+func parseCSV(conf cfg.Config, input io.Reader) (Tabdata, error) {
+	data := Tabdata{}
+
+	// apply pattern, if any
+	content, err := FilterByPattern(conf, input)
+	if err != nil {
+		return data, err
+	}
+
+	csvreader := csv.NewReader(content)
+	csvreader.Comma = rune(conf.Separator[0])
+
+	records, err := csvreader.ReadAll()
+	if err != nil {
+		return data, fmt.Errorf("could not parse CSV input: %w", err)
+	}
+
+	if len(records) >= 1 {
+		data.headers = records[0]
+		data.columns = len(records)
+
+		for _, head := range data.headers {
+			// register widest header field
+			headerlen := len(head)
+			if headerlen > data.maxwidthHeader {
+				data.maxwidthHeader = headerlen
+			}
+		}
+
+		if len(records) > 1 {
+			data.entries = records[1:]
+		}
+	}
+
+	// apply user defined lisp process hooks, if any
+	userdata, changed, err := RunProcessHooks(conf, data)
+	if err != nil {
+		return data, fmt.Errorf("failed to apply filter hook: %w", err)
+	}
+
+	if changed {
+		data = userdata
+	}
+
+	return data, nil
+}
+
+/*
+Parse tabular input.
+*/
+func parseTabular(conf cfg.Config, input io.Reader) (Tabdata, error) {
 	data := Tabdata{}
 
 	var scanner *bufio.Scanner
 
 	hadFirst := false
-	separate := regexp.MustCompile(Separator)
-	patternR, err := regexp.Compile(pattern)
-	if err != nil {
-		return data, errors.Unwrap(fmt.Errorf("Regexp pattern %s is invalid: %w", pattern, err))
-	}
+	separate := regexp.MustCompile(conf.Separator)
 
 	scanner = bufio.NewScanner(input)
 
@@ -57,10 +116,6 @@ func parseFile(input io.Reader, pattern string) (Tabdata, error) {
 
 			// process all header fields
 			for _, part := range parts {
-				// if Debug {
-				// 	fmt.Printf("Part: <%s>\n", string(line[beg:part[0]]))
-				//}
-
 				// register widest header field
 				headerlen := len(part)
 				if headerlen > data.maxwidthHeader {
@@ -75,29 +130,29 @@ func parseFile(input io.Reader, pattern string) (Tabdata, error) {
 			}
 		} else {
 			// data processing
-			if len(pattern) > 0 {
-				if patternR.MatchString(line) == InvertMatch {
-					// by default  -v is false, so if a  line does NOT
-					// match the pattern, we will ignore it. However,
-					// if the user specified -v, the matching is inverted,
-					// so we ignore all lines, which DO match.
-					continue
-				}
+			if conf.Pattern != "" && matchPattern(conf, line) == conf.InvertMatch {
+				// by default  -v is false, so if a  line does NOT
+				// match the pattern, we will ignore it. However,
+				// if the user specified -v, the matching is inverted,
+				// so we ignore all lines, which DO match.
+				continue
+			}
+
+			// apply user defined lisp filters, if any
+			accept, err := RunFilterHooks(conf, line)
+			if err != nil {
+				return data, fmt.Errorf("failed to apply filter hook: %w", err)
+			}
+
+			if !accept {
+				//  IF there  are filter  hook[s] and  IF one  of them
+				// returns false on the current line, reject it
+				continue
 			}
 
 			idx := 0 // we cannot use the header index, because we could exclude columns
 			values := []string{}
 			for _, part := range parts {
-				width := len(strings.TrimSpace(part))
-
-				if len(data.maxwidthPerCol)-1 < idx {
-					data.maxwidthPerCol = append(data.maxwidthPerCol, width)
-				} else {
-					if width > data.maxwidthPerCol[idx] {
-						data.maxwidthPerCol[idx] = width
-					}
-				}
-
 				// if Debug {
 				// 	fmt.Printf("<%s> ", value)
 				// }
@@ -105,15 +160,41 @@ func parseFile(input io.Reader, pattern string) (Tabdata, error) {
 
 				idx++
 			}
+
+			// fill up missing fields, if any
+			for i := len(values); i < len(data.headers); i++ {
+				values = append(values, "")
+			}
+
 			data.entries = append(data.entries, values)
 		}
 	}
 
 	if scanner.Err() != nil {
-		return data, errors.Unwrap(fmt.Errorf("Failed to read from io.Reader: %w", scanner.Err()))
+		return data, fmt.Errorf("failed to read from io.Reader: %w", scanner.Err())
 	}
 
-	if Debug {
+	// filter by field filters, if any
+	filtereddata, changed, err := FilterByFields(conf, data)
+	if err != nil {
+		return data, fmt.Errorf("failed to filter fields: %w", err)
+	}
+
+	if changed {
+		data = filtereddata
+	}
+
+	// apply user defined lisp process hooks, if any
+	userdata, changed, err := RunProcessHooks(conf, data)
+	if err != nil {
+		return data, fmt.Errorf("failed to apply filter hook: %w", err)
+	}
+
+	if changed {
+		data = userdata
+	}
+
+	if conf.Debug {
 		repr.Print(data)
 	}
 

lib/parser_test.go

@@ -22,42 +22,62 @@ import (
 	"reflect"
 	"strings"
 	"testing"
+
+	"github.com/tlinden/tablizer/cfg"
 )
 
+var input = []struct {
+	name      string
+	text      string
+	separator string
+}{
+	{
+		name:      "tabular-data",
+		separator: cfg.DefaultSeparator,
+		text: `
+ONE    TWO    THREE  
+asd    igig   cxxxncnc  
+19191  EDD 1  X`,
+	},
+	{
+		name:      "csv-data",
+		separator: ",",
+		text: `
+ONE,TWO,THREE
+asd,igig,cxxxncnc
+19191,"EDD 1",X`,
+	},
+}
+
 func TestParser(t *testing.T) {
 	data := Tabdata{
 		maxwidthHeader: 5,
-		maxwidthPerCol: []int{
-			5, 5, 8,
-		},
-		columns: 3,
+		columns:        3,
 		headers: []string{
 			"ONE", "TWO", "THREE",
 		},
 		entries: [][]string{
-			{
-				"asd", "igig", "cxxxncnc",
-			},
-			{
-				"19191", "EDD 1", "X",
-			},
+			{"asd", "igig", "cxxxncnc"},
+			{"19191", "EDD 1", "X"},
 		},
 	}
 
-	table := `ONE    TWO    THREE  
-asd    igig   cxxxncnc  
-19191  EDD 1  X`
+	for _, testdata := range input {
+		testname := fmt.Sprintf("parse-%s", testdata.name)
+		t.Run(testname, func(t *testing.T) {
+			readFd := strings.NewReader(strings.TrimSpace(testdata.text))
+			conf := cfg.Config{Separator: testdata.separator}
+			gotdata, err := Parse(conf, readFd)
 
-	readFd := strings.NewReader(table)
-	gotdata, err := parseFile(readFd, "")
-	Separator = DefaultSeparator
+			if err != nil {
+				t.Errorf("Parser returned error: %s\nData processed so far: %+v", err, gotdata)
+			}
 
-	if err != nil {
-		t.Errorf("Parser returned error: %s\nData processed so far: %+v", err, gotdata)
-	}
-
-	if !reflect.DeepEqual(data, gotdata) {
-		t.Errorf("Parser returned invalid data, Regex: %s\nExp: %+v\nGot: %+v\n", Separator, data, gotdata)
+			if !reflect.DeepEqual(data, gotdata) {
+				t.Errorf("Parser returned invalid data\nExp: %+v\nGot: %+v\n",
+					data, gotdata)
+			}
+		})
 	}
 }
 
@@ -66,47 +86,81 @@ func TestParserPatternmatching(t *testing.T) {
 		entries [][]string
 		pattern string
 		invert  bool
+		want    bool
 	}{
 		{
 			entries: [][]string{
-				{
-					"asd", "igig", "cxxxncnc",
-				},
+				{"asd", "igig", "cxxxncnc"},
 			},
 			pattern: "ig",
 			invert:  false,
 		},
 		{
 			entries: [][]string{
-				{
-					"19191", "EDD 1", "X",
-				},
+				{"19191", "EDD 1", "X"},
 			},
 			pattern: "ig",
 			invert:  true,
 		},
 	}
 
-	table := `ONE    TWO    THREE  
-asd    igig   cxxxncnc  
-19191  EDD 1  X`
+	for _, inputdata := range input {
+		for _, testdata := range tests {
+			testname := fmt.Sprintf("parse-%s-with-pattern-%s-inverted-%t",
+				inputdata.name, testdata.pattern, testdata.invert)
+			t.Run(testname, func(t *testing.T) {
+				conf := cfg.Config{InvertMatch: testdata.invert, Pattern: testdata.pattern,
+					Separator: inputdata.separator}
 
-	for _, tt := range tests {
-		testname := fmt.Sprintf("parse-with-inverted-pattern-%t", tt.invert)
-		t.Run(testname, func(t *testing.T) {
-			InvertMatch = tt.invert
+				_ = conf.PreparePattern(testdata.pattern)
 
-			readFd := strings.NewReader(table)
-			gotdata, err := parseFile(readFd, tt.pattern)
+				readFd := strings.NewReader(strings.TrimSpace(inputdata.text))
+				gotdata, err := Parse(conf, readFd)
 
-			if err != nil {
-				t.Errorf("Parser returned error: %s\nData processed so far: %+v", err, gotdata)
-			}
-
-			if !reflect.DeepEqual(tt.entries, gotdata.entries) {
-				t.Errorf("Parser returned invalid data (pattern: %s, invert: %t)\nExp: %+v\nGot: %+v\n",
-					tt.pattern, tt.invert, tt.entries, gotdata.entries)
-			}
-		})
+				if err != nil {
+					if !testdata.want {
+						t.Errorf("Parser returned error: %s\nData processed so far: %+v",
+							err, gotdata)
+					}
+				} else {
+					if !reflect.DeepEqual(testdata.entries, gotdata.entries) {
+						t.Errorf("Parser returned invalid data (pattern: %s, invert: %t)\nExp: %+v\nGot: %+v\n",
+							testdata.pattern, testdata.invert, testdata.entries, gotdata.entries)
+					}
+				}
+			})
+		}
+	}
+}
+
+func TestParserIncompleteRows(t *testing.T) {
+	data := Tabdata{
+		maxwidthHeader: 5,
+		columns:        3,
+		headers: []string{
+			"ONE", "TWO", "THREE",
+		},
+		entries: [][]string{
+			{"asd", "igig", ""},
+			{"19191", "EDD 1", "X"},
+		},
+	}
+
+	table := `
+ONE    TWO    THREE  
+asd    igig
+19191  EDD 1  X`
+
+	readFd := strings.NewReader(strings.TrimSpace(table))
+	conf := cfg.Config{Separator: cfg.DefaultSeparator}
+	gotdata, err := Parse(conf, readFd)
+
+	if err != nil {
+		t.Errorf("Parser returned error: %s\nData processed so far: %+v", err, gotdata)
+	}
+
+	if !reflect.DeepEqual(data, gotdata) {
+		t.Errorf("Parser returned invalid data, Regex: %s\nExp: %+v\nGot: %+v\n",
+			conf.Separator, data, gotdata)
 	}
 }

lib/printer.go

@@ -18,51 +18,64 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 package lib
 
 import (
+	"encoding/csv"
 	"fmt"
+	"io"
+	"log"
+	"regexp"
+	"strconv"
+	"strings"
+
 	"github.com/gookit/color"
 	"github.com/olekukonko/tablewriter"
-	"regexp"
-	"strings"
+	"github.com/tlinden/tablizer/cfg"
+	"gopkg.in/yaml.v3"
 )
 
-func printData(data *Tabdata) {
-	// some output preparations:
-
-	if OutputMode != "shell" {
-		// not needed in eval string
-		numberizeHeaders(data)
-	}
+func printData(writer io.Writer, conf cfg.Config, data *Tabdata) {
+	// add numbers to headers and remove this we're not interested in
+	numberizeAndReduceHeaders(conf, data)
 
 	// remove unwanted columns, if any
-	reduceColumns(data)
+	reduceColumns(conf, data)
 
 	// sort the data
-	sortTable(data, SortByColumn)
+	sortTable(conf, data)
 
-	switch OutputMode {
-	case "extended":
-		printExtendedData(data)
-	case "ascii":
-		printAsciiData(data)
-	case "orgtbl":
-		printOrgmodeData(data)
-	case "markdown":
-		printMarkdownData(data)
-	case "shell":
-		printShellData(data)
+	switch conf.OutputMode {
+	case cfg.Extended:
+		printExtendedData(writer, conf, data)
+	case cfg.ASCII:
+		printASCIIData(writer, conf, data)
+	case cfg.Orgtbl:
+		printOrgmodeData(writer, conf, data)
+	case cfg.Markdown:
+		printMarkdownData(writer, conf, data)
+	case cfg.Shell:
+		printShellData(writer, data)
+	case cfg.Yaml:
+		printYamlData(writer, data)
+	case cfg.CSV:
+		printCSVData(writer, data)
 	default:
-		printAsciiData(data)
+		printASCIIData(writer, conf, data)
 	}
 }
 
+func output(writer io.Writer, str string) {
+	fmt.Fprint(writer, str)
+}
+
 /*
-   Emacs org-mode compatible table (also orgtbl-mode)
+Emacs org-mode compatible table (also orgtbl-mode)
 */
-func printOrgmodeData(data *Tabdata) {
+func printOrgmodeData(writer io.Writer, conf cfg.Config, data *Tabdata) {
 	tableString := &strings.Builder{}
 	table := tablewriter.NewWriter(tableString)
 
-	table.SetHeader(data.headers)
+	if !conf.NoHeaders {
+		table.SetHeader(data.headers)
+	}
 
 	for _, row := range data.entries {
 		table.Append(trimRow(row))
@@ -81,23 +94,25 @@ func printOrgmodeData(data *Tabdata) {
 	   | cell | cell |
 	   |------+------|
 	*/
-	leftR := regexp.MustCompile("(?m)^\\+")
-	rightR := regexp.MustCompile("\\+(?m)$")
+	leftR := regexp.MustCompile(`(?m)^\\+`)
+	rightR := regexp.MustCompile(`\\+(?m)$`)
 
-	color.Print(
-		colorizeData(
+	output(writer, color.Sprint(
+		colorizeData(conf,
 			rightR.ReplaceAllString(
-				leftR.ReplaceAllString(tableString.String(), "|"), "|")))
+				leftR.ReplaceAllString(tableString.String(), "|"), "|"))))
 }
 
 /*
-   Markdown table
+Markdown table
 */
-func printMarkdownData(data *Tabdata) {
+func printMarkdownData(writer io.Writer, conf cfg.Config, data *Tabdata) {
 	tableString := &strings.Builder{}
 	table := tablewriter.NewWriter(tableString)
 
-	table.SetHeader(data.headers)
+	if !conf.NoHeaders {
+		table.SetHeader(data.headers)
+	}
 
 	for _, row := range data.entries {
 		table.Append(trimRow(row))
@@ -107,22 +122,21 @@ func printMarkdownData(data *Tabdata) {
 	table.SetCenterSeparator("|")
 
 	table.Render()
-	color.Print(colorizeData(tableString.String()))
+	output(writer, color.Sprint(colorizeData(conf, tableString.String())))
 }
 
 /*
-   Simple ASCII table without any borders etc, just like the input we expect
+Simple ASCII table without any borders etc, just like the input we expect
 */
-func printAsciiData(data *Tabdata) {
+func printASCIIData(writer io.Writer, conf cfg.Config, data *Tabdata) {
 	tableString := &strings.Builder{}
 	table := tablewriter.NewWriter(tableString)
 
-	table.SetHeader(data.headers)
-	table.AppendBulk(data.entries)
+	if !conf.NoHeaders {
+		table.SetHeader(data.headers)
+	}
 
-	// for _, row := range data.entries {
-	// 	table.Append(trimRow(row))
-	// }
+	table.AppendBulk(data.entries)
 
 	table.SetAutoWrapText(false)
 	table.SetAutoFormatHeaders(true)
@@ -133,51 +147,116 @@ func printAsciiData(data *Tabdata) {
 	table.SetRowSeparator("")
 	table.SetHeaderLine(false)
 	table.SetBorder(false)
-	table.SetTablePadding("\t") // pad with tabs
 	table.SetNoWhiteSpace(true)
 
+	if !conf.UseHighlight {
+		// the tabs destroy the highlighting
+		table.SetTablePadding("\t") // pad with tabs
+	} else {
+		table.SetTablePadding("   ")
+	}
+
 	table.Render()
-	color.Print(colorizeData(tableString.String()))
+	output(writer, color.Sprint(colorizeData(conf, tableString.String())))
 }
 
 /*
-   We simulate the \x command of psql (the PostgreSQL client)
+We simulate the \x command of psql (the PostgreSQL client)
 */
-func printExtendedData(data *Tabdata) {
+func printExtendedData(writer io.Writer, conf cfg.Config, data *Tabdata) {
 	// needed for data output
 	format := fmt.Sprintf("%%%ds: %%s\n", data.maxwidthHeader)
+	out := ""
 
 	if len(data.entries) > 0 {
 		for _, entry := range data.entries {
 			for i, value := range entry {
-				color.Printf(format, data.headers[i], value)
+				out += color.Sprintf(format, data.headers[i], value)
 			}
 
-			fmt.Println()
+			out += "\n"
 		}
 	}
+
+	output(writer, colorizeData(conf, out))
 }
 
 /*
-   Shell output, ready to be eval'd. Just like FreeBSD stat(1)
+Shell output, ready to be eval'd. Just like FreeBSD stat(1)
 */
-func printShellData(data *Tabdata) {
-	if len(data.entries) > 0 {
-		var idx int
-		for _, entry := range data.entries {
-			idx = 0
-			shentries := []string{}
-			for i, value := range entry {
-				if len(Columns) > 0 {
-					if !contains(UseColumns, i+1) {
-						continue
-					}
-				}
+func printShellData(writer io.Writer, data *Tabdata) {
+	out := ""
 
-				shentries = append(shentries, fmt.Sprintf("%s=\"%s\"", data.headers[idx], value))
-				idx++
+	if len(data.entries) > 0 {
+		for _, entry := range data.entries {
+			shentries := []string{}
+
+			for idx, value := range entry {
+				shentries = append(shentries, fmt.Sprintf("%s=\"%s\"",
+					data.headers[idx], value))
 			}
-			fmt.Println(strings.Join(shentries, " "))
+
+			out += strings.Join(shentries, " ") + "\n"
 		}
 	}
+
+	// no colorization here
+	output(writer, out)
+}
+
+func printYamlData(writer io.Writer, data *Tabdata) {
+	type Data struct {
+		Entries []map[string]interface{} `yaml:"entries"`
+	}
+
+	yamlout := Data{}
+
+	for _, entry := range data.entries {
+		yamldata := map[string]interface{}{}
+
+		for idx, entry := range entry {
+			style := yaml.TaggedStyle
+
+			_, err := strconv.Atoi(entry)
+			if err != nil {
+				style = yaml.DoubleQuotedStyle
+			}
+
+			yamldata[strings.ToLower(data.headers[idx])] =
+				&yaml.Node{
+					Kind:  yaml.ScalarNode,
+					Style: style,
+					Value: entry}
+		}
+
+		yamlout.Entries = append(yamlout.Entries, yamldata)
+	}
+
+	yamlstr, err := yaml.Marshal(&yamlout)
+
+	if err != nil {
+		log.Fatal(err)
+	}
+
+	output(writer, string(yamlstr))
+}
+
+func printCSVData(writer io.Writer, data *Tabdata) {
+	csvout := csv.NewWriter(writer)
+
+	if err := csvout.Write(data.headers); err != nil {
+		log.Fatalln("error writing record to csv:", err)
+	}
+
+	for _, entry := range data.entries {
+		if err := csvout.Write(entry); err != nil {
+			log.Fatalln("error writing record to csv:", err)
+		}
+	}
+
+	csvout.Flush()
+
+	if err := csvout.Error(); err != nil {
+		log.Fatal(err)
+	}
 }

lib/printer_test.go

@@ -18,197 +18,276 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 package lib
 
 import (
+	"bytes"
 	"fmt"
-	"github.com/gookit/color"
-	"os"
 	"strings"
 	"testing"
+
+	"github.com/tlinden/tablizer/cfg"
 )
 
-func stdout2pipe(t *testing.T) (*os.File, *os.File) {
-	reader, writer, err := os.Pipe()
-	if err != nil {
-		t.Fatal(err)
+func newData() Tabdata {
+	return Tabdata{
+		maxwidthHeader: 8,
+		columns:        4,
+		headers: []string{
+			"NAME",
+			"DURATION",
+			"COUNT",
+			"WHEN",
+		},
+		entries: [][]string{
+			{
+				"beta",
+				"1d10h5m1s",
+				"33",
+				"3/1/2014",
+			},
+			{
+				"alpha",
+				"4h35m",
+				"170",
+				"2013-Feb-03",
+			},
+			{
+				"ceta",
+				"33d12h",
+				"9",
+				"06/Jan/2008 15:04:05 -0700",
+			},
+		},
 	}
-	origStdout := os.Stdout
-	os.Stdout = writer
+}
 
-	// we need to tell the color mode the io.Writer, even if we don't usw colorization
-	color.SetOutput(writer)
+var tests = []struct {
+	name string // so we can identify which one fails, can be the same
+	// for multiple tests, because flags will be appended to the name
+	sortby    string // empty == default
+	column    int    // sort by this column, 0 == default first or NO Sort
+	desc      bool   // sort in descending order, default == ascending
+	nonum     bool   // hide numbering
+	mode      int    // shell, orgtbl, etc. empty == default: ascii
+	usecol    []int  // columns to display, empty == display all
+	usecolstr string // for testname, must match usecol
+	expect    string // rendered output we expect
+}{
+	// --------------------- Default settings mode tests ``
+	{
+		mode: cfg.ASCII,
+		name: "default",
+		expect: `
+NAME(1)	DURATION(2)	COUNT(3)	WHEN(4)                    
+beta   	1d10h5m1s  	33      	3/1/2014                  	
+alpha  	4h35m      	170     	2013-Feb-03               	
+ceta   	33d12h     	9       	06/Jan/2008 15:04:05 -0700`,
+	},
+	{
+		mode: cfg.CSV,
+		name: "csv",
+		expect: `
+NAME,DURATION,COUNT,WHEN
+beta,1d10h5m1s,33,3/1/2014
+alpha,4h35m,170,2013-Feb-03
+ceta,33d12h,9,06/Jan/2008 15:04:05 -0700`,
+	},
+	{
+		name: "default",
+		mode: cfg.Orgtbl,
+		expect: `
++---------+-------------+----------+----------------------------+
+| NAME(1) | DURATION(2) | COUNT(3) |          WHEN(4)           |
++---------+-------------+----------+----------------------------+
+| beta    | 1d10h5m1s   |       33 | 3/1/2014                   |
+| alpha   | 4h35m       |      170 | 2013-Feb-03                |
+| ceta    | 33d12h      |        9 | 06/Jan/2008 15:04:05 -0700 |
++---------+-------------+----------+----------------------------+`,
+	},
+	{
+		name: "default",
+		mode: cfg.Markdown,
+		expect: `
+| NAME(1) | DURATION(2) | COUNT(3) |          WHEN(4)           |
+|---------|-------------|----------|----------------------------|
+| beta    | 1d10h5m1s   |       33 | 3/1/2014                   |
+| alpha   | 4h35m       |      170 | 2013-Feb-03                |
+| ceta    | 33d12h      |        9 | 06/Jan/2008 15:04:05 -0700 |`,
+	},
+	{
+		name:  "default",
+		mode:  cfg.Shell,
+		nonum: true,
+		expect: `
+NAME="beta" DURATION="1d10h5m1s" COUNT="33" WHEN="3/1/2014"
+NAME="alpha" DURATION="4h35m" COUNT="170" WHEN="2013-Feb-03"
+NAME="ceta" DURATION="33d12h" COUNT="9" WHEN="06/Jan/2008 15:04:05 -0700"`,
+	},
+	{
+		name:  "default",
+		mode:  cfg.Yaml,
+		nonum: true,
+		expect: `
+entries:
+    - count: 33
+      duration: "1d10h5m1s"
+      name: "beta"
+      when: "3/1/2014"
+    - count: 170
+      duration: "4h35m"
+      name: "alpha"
+      when: "2013-Feb-03"
+    - count: 9
+      duration: "33d12h"
+      name: "ceta"
+      when: "06/Jan/2008 15:04:05 -0700"`,
+	},
+	{
+		name: "default",
+		mode: cfg.Extended,
+		expect: `
+    NAME(1): beta
+DURATION(2): 1d10h5m1s
+   COUNT(3): 33
+    WHEN(4): 3/1/2014
 
-	return origStdout, reader
+    NAME(1): alpha
+DURATION(2): 4h35m
+   COUNT(3): 170
+    WHEN(4): 2013-Feb-03
+
+    NAME(1): ceta
+DURATION(2): 33d12h
+   COUNT(3): 9
+    WHEN(4): 06/Jan/2008 15:04:05 -0700`,
+	},
+
+	//------------------------ SORT TESTS
+	{
+		name:   "sortbycolumn",
+		column: 3,
+		sortby: "numeric",
+		desc:   false,
+		expect: `
+NAME(1)	DURATION(2)	COUNT(3)	WHEN(4)                    
+ceta   	33d12h     	9       	06/Jan/2008 15:04:05 -0700	
+beta   	1d10h5m1s  	33      	3/1/2014                  	
+alpha  	4h35m      	170     	2013-Feb-03`,
+	},
+	{
+		name:   "sortbycolumn",
+		column: 4,
+		sortby: "time",
+		desc:   false,
+		expect: `
+NAME(1)	DURATION(2)	COUNT(3)	WHEN(4)                    
+ceta   	33d12h     	9       	06/Jan/2008 15:04:05 -0700	
+alpha  	4h35m      	170     	2013-Feb-03               	
+beta   	1d10h5m1s  	33      	3/1/2014`,
+	},
+	{
+		name:   "sortbycolumn",
+		column: 2,
+		sortby: "duration",
+		desc:   false,
+		expect: `
+NAME(1)	DURATION(2)	COUNT(3)	WHEN(4)                    
+alpha  	4h35m      	170     	2013-Feb-03               	
+beta   	1d10h5m1s  	33      	3/1/2014                  	
+ceta   	33d12h     	9       	06/Jan/2008 15:04:05 -0700`,
+	},
+
+	//  -----------------------  UseColumns Tests
+	{
+		name:      "usecolumns",
+		usecol:    []int{1, 4},
+		usecolstr: "1,4",
+		expect: `
+NAME(1)	WHEN(4)                    
+beta   	3/1/2014                  	
+alpha  	2013-Feb-03               	
+ceta   	06/Jan/2008 15:04:05 -0700`,
+	},
+	{
+		name:      "usecolumns",
+		usecol:    []int{2},
+		usecolstr: "2",
+		expect: `
+DURATION(2) 
+1d10h5m1s  	
+4h35m      	
+33d12h`,
+	},
+	{
+		name:      "usecolumns",
+		usecol:    []int{3},
+		usecolstr: "3",
+		expect: `
+COUNT(3) 
+33      	
+170     	
+9`,
+	},
+	{
+		name:      "usecolumns",
+		column:    0,
+		usecol:    []int{1, 3},
+		usecolstr: "1,3",
+		expect: `
+NAME(1)	COUNT(3) 
+beta   	33      	
+alpha  	170     	
+ceta   	9`,
+	},
+	{
+		name:      "usecolumns",
+		usecol:    []int{2, 4},
+		usecolstr: "2,4",
+		expect: `
+DURATION(2)	WHEN(4)                    
+1d10h5m1s  	3/1/2014                  	
+4h35m      	2013-Feb-03               	
+33d12h     	06/Jan/2008 15:04:05 -0700`,
+	},
 }
 
 func TestPrinter(t *testing.T) {
-	startdata := Tabdata{
-		maxwidthHeader: 5,
-		maxwidthPerCol: []int{
-			5,
-			5,
-			8,
-		},
-		columns: 3,
-		headers: []string{
-			"ONE", "TWO", "THREE",
-		},
-		entries: [][]string{
-			{
-				"asd", "igig", "cxxxncnc",
-			},
-			{
-				"19191", "EDD 1", "X",
-			},
-		},
-	}
-
-	expects := map[string]string{
-		"ascii": `ONE(1)	TWO(2)	THREE(3) 
-asd   	igig  	cxxxncnc	
-19191 	EDD 1 	X`,
-
-		"orgtbl": `|--------+--------+----------|
-| ONE(1) | TWO(2) | THREE(3) |
-|--------+--------+----------|
-| asd    | igig   | cxxxncnc |
-|  19191 | EDD 1  | X        |
-|--------+--------+----------|`,
-
-		"markdown": `| ONE(1) | TWO(2) | THREE(3) |
-|--------|--------|----------|
-| asd    | igig   | cxxxncnc |
-|  19191 | EDD 1  | X        |`,
-
-		"shell": `ONE="asd" TWO="igig" THREE="cxxxncnc"
-ONE="19191" TWO="EDD 1" THREE="X"`,
-
-		"extended": `ONE(1): asd
-  TWO(2): igig
-THREE(3): cxxxncnc
-
-  ONE(1): 19191
-  TWO(2): EDD 1
-THREE(3): X`,
-	}
-
-	NoColor = true
-	SortByColumn = 0 // disable sorting
-
-	origStdout, reader := stdout2pipe(t)
-
-	for mode, expect := range expects {
-		testname := fmt.Sprintf("print-%s", mode)
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("print-sortcol-%d-desc-%t-sortby-%s-mode-%d-usecolumns-%s",
+			testdata.column, testdata.desc, testdata.sortby, testdata.mode, testdata.usecolstr)
 		t.Run(testname, func(t *testing.T) {
+			// replaces os.Stdout, but we ignore it
+			var writer bytes.Buffer
 
-			OutputMode = mode
-			//  we need  to reset  our  mock data,  since it's  being
-			// modified in printData()
-			data := startdata
-			printData(&data)
-
-			buf := make([]byte, 1024)
-			n, err := reader.Read(buf)
-			if err != nil {
-				t.Fatal(err)
+			// cmd flags
+			conf := cfg.Config{
+				SortByColumn:   testdata.column,
+				SortDescending: testdata.desc,
+				SortMode:       testdata.sortby,
+				OutputMode:     testdata.mode,
+				NoNumbering:    testdata.nonum,
+				UseColumns:     testdata.usecol,
+				NoColor:        true,
 			}
-			buf = buf[:n]
-			output := strings.TrimSpace(string(buf))
 
-			if output != expect {
-				t.Errorf("output mode: %s, got:\n%s\nwant:\n%s\n (%d <=> %d)",
-					mode, output, expect, len(output), len(expect))
+			conf.ApplyDefaults()
+
+			// the test checks the len!
+			if len(testdata.usecol) > 0 {
+				conf.Columns = "yes"
+			} else {
+				conf.Columns = ""
+			}
+
+			data := newData()
+			exp := strings.TrimSpace(testdata.expect)
+
+			printData(&writer, conf, &data)
+
+			got := strings.TrimSpace(writer.String())
+
+			if got != exp {
+				t.Errorf("not rendered correctly:\n+++ got:\n%s\n+++ want:\n%s",
+					got, exp)
 			}
 		})
 	}
-
-	// Restore
-	os.Stdout = origStdout
-
-}
-
-func TestSortPrinter(t *testing.T) {
-	startdata := Tabdata{
-		maxwidthHeader: 5,
-		maxwidthPerCol: []int{
-			3,
-			3,
-			2,
-		},
-		columns: 3,
-		headers: []string{
-			"ONE", "TWO", "THREE",
-		},
-		entries: [][]string{
-			{
-				"abc", "345", "b1",
-			},
-			{
-				"bcd", "234", "a2",
-			},
-			{
-				"cde", "123", "c3",
-			},
-		},
-	}
-
-	var tests = []struct {
-		data   Tabdata
-		sortby int
-		expect string
-	}{
-		{
-			data:   startdata,
-			sortby: 1,
-			expect: `ONE(1)	TWO(2)	THREE(3) 
-abc   	345   	b1      	
-bcd   	234   	a2      	
-cde   	123   	c3`,
-		},
-
-		{
-			data:   startdata,
-			sortby: 2,
-			expect: `ONE(1)	TWO(2)	THREE(3) 
-cde   	123   	c3      	
-bcd   	234   	a2      	
-abc   	345   	b1`,
-		},
-
-		{
-			data:   startdata,
-			sortby: 3,
-			expect: `ONE(1)	TWO(2)	THREE(3) 
-bcd   	234   	a2      	
-abc   	345   	b1      	
-cde   	123   	c3`,
-		},
-	}
-
-	NoColor = true
-	OutputMode = "ascii"
-	origStdout, reader := stdout2pipe(t)
-
-	for _, tt := range tests {
-		testname := fmt.Sprintf("print-sorted-table-%d", tt.sortby)
-		t.Run(testname, func(t *testing.T) {
-			SortByColumn = tt.sortby
-
-			printData(&tt.data)
-
-			buf := make([]byte, 1024)
-			n, err := reader.Read(buf)
-			if err != nil {
-				t.Fatal(err)
-			}
-			buf = buf[:n]
-			output := strings.TrimSpace(string(buf))
-
-			if output != tt.expect {
-				t.Errorf("sort column: %d, got:\n%s\nwant:\n%s",
-					tt.sortby, output, tt.expect)
-			}
-		})
-	}
-
-	// Restore
-	os.Stdout = origStdout
 }

lib/sort.go

@@ -18,15 +18,23 @@ along with this program. If not, see <http://www.gnu.org/licenses/>.
 package lib
 
 import (
+	"regexp"
 	"sort"
+	"strconv"
+
+	"github.com/araddon/dateparse"
+	"github.com/tlinden/tablizer/cfg"
 )
 
-func sortTable(data *Tabdata, col int) {
-	if col <= 0 {
+func sortTable(conf cfg.Config, data *Tabdata) {
+	if conf.SortByColumn <= 0 {
 		// no sorting wanted
 		return
 	}
 
+	// slightly modified here to match internal array indicies
+	col := conf.SortByColumn
+
 	col-- // ui starts counting by 1, but use 0 internally
 
 	// sanity checks
@@ -41,6 +49,79 @@ func sortTable(data *Tabdata, col int) {
 
 	// actual sorting
 	sort.SliceStable(data.entries, func(i, j int) bool {
-		return data.entries[i][col] < data.entries[j][col]
+		return compare(&conf, data.entries[i][col], data.entries[j][col])
 	})
 }
+
+// config is not modified here, but it would be inefficient to copy it every loop
+func compare(conf *cfg.Config, left string, right string) bool {
+	var comp bool
+
+	switch conf.SortMode {
+	case "numeric":
+		left, err := strconv.Atoi(left)
+		if err != nil {
+			left = 0
+		}
+
+		right, err := strconv.Atoi(right)
+		if err != nil {
+			right = 0
+		}
+
+		comp = left < right
+	case "duration":
+		left := duration2int(left)
+		right := duration2int(right)
+
+		comp = left < right
+	case "time":
+		left, _ := dateparse.ParseAny(left)
+		right, _ := dateparse.ParseAny(right)
+
+		comp = left.Unix() < right.Unix()
+	default:
+		comp = left < right
+	}
+
+	if conf.SortDescending {
+		comp = !comp
+	}
+
+	return comp
+}
+
+/*
+We could use time.ParseDuration(), but this doesn't support days.
+
+We  could also  use github.com/xhit/go-str2duration/v2,  which does
+the job,  but it's  just another dependency,  just for  this little
+gem. And  we don't need a  time.Time value. And int  is good enough
+for duration comparison.
+
+Convert a  duration into  an integer.  Valid  time units  are "s",
+"m", "h" and "d".
+*/
+func duration2int(duration string) int {
+	re := regexp.MustCompile(`(\d+)([dhms])`)
+	seconds := 0
+
+	for _, match := range re.FindAllStringSubmatch(duration, -1) {
+		if len(match) == 3 {
+			durationvalue, _ := strconv.Atoi(match[1])
+
+			switch match[2][0] {
+			case 'd':
+				seconds += durationvalue * 86400
+			case 'h':
+				seconds += durationvalue * 3600
+			case 'm':
+				seconds += durationvalue * 60
+			case 's':
+				seconds += durationvalue
+			}
+		}
+	}
+
+	return seconds
+}

lib/sort_test.go

@@ -0,0 +1,82 @@
+/*
+Copyright © 2022 Thomas von Dein
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+*/
+
+package lib
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/tlinden/tablizer/cfg"
+)
+
+func TestDuration2Seconds(t *testing.T) {
+	var tests = []struct {
+		dur    string
+		expect int
+	}{
+		{"1d", 60 * 60 * 24},
+		{"1h", 60 * 60},
+		{"10m", 60 * 10},
+		{"2h4m10s", (60 * 120) + (4 * 60) + 10},
+		{"88u", 0},
+		{"19t77X what?4s", 4},
+	}
+
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("duration-%s", testdata.dur)
+		t.Run(testname, func(t *testing.T) {
+			seconds := duration2int(testdata.dur)
+			if seconds != testdata.expect {
+				t.Errorf("got %d, want %d", seconds, testdata.expect)
+			}
+		})
+	}
+}
+
+func TestCompare(t *testing.T) {
+	var tests = []struct {
+		mode string
+		a    string
+		b    string
+		want bool
+		desc bool
+	}{
+		// ascending
+		{"numeric", "10", "20", true, false},
+		{"duration", "2d4h5m", "45m", false, false},
+		{"time", "12/24/2022", "1/1/1970", false, false},
+
+		// descending
+		{"numeric", "10", "20", false, true},
+		{"duration", "2d4h5m", "45m", true, true},
+		{"time", "12/24/2022", "1/1/1970", true, true},
+	}
+
+	for _, testdata := range tests {
+		testname := fmt.Sprintf("compare-mode-%s-a-%s-b-%s-desc-%t",
+			testdata.mode, testdata.a, testdata.b, testdata.desc)
+
+		t.Run(testname, func(t *testing.T) {
+			c := cfg.Config{SortMode: testdata.mode, SortDescending: testdata.desc}
+			got := compare(&c, testdata.a, testdata.b)
+			if got != testdata.want {
+				t.Errorf("got %t, want %t", got, testdata.want)
+			}
+		})
+	}
+}

mkrel.sh

@@ -43,7 +43,7 @@ for D in $DIST; do
     tardir="${tool}-${os}-${arch}-${version}"
     tarfile="releases/${tool}-${os}-${arch}-${version}.tar.gz"
     set -x
-    GOOS=${os} GOARCH=${arch} go build -o ${binfile} -ldflags "-X 'github.com/tlinden/tablizer/lib.VERSION=${version}'"
+    GOOS=${os} GOARCH=${arch} go build -o ${binfile} -ldflags "-X 'github.com/tlinden/tablizer/cfg.VERSION=${version}'"
     mkdir -p ${tardir}
     cp ${binfile} README.md LICENSE ${tardir}/
     echo 'tool = tablizer

t/plugintest.zy

@@ -0,0 +1,10 @@
+/*
+Simple filter hook function. Splits the argument by whitespace,
+fetches the 2nd element, converts it to an int and returns true
+if it s larger than 5, false otherwise.
+*/
+(defn uselarge [line]
+  (cond (> (atoi (second (resplit line `\s+`))) 5) true false))
+
+/* Register the filter hook */
+(addhook %filter %uselarge)

t/test.sh

@@ -0,0 +1,45 @@
+#!/bin/sh
+
+# simple commandline unit test script
+
+t="../tablizer"
+fail=0
+
+ex() {
+    # execute a test, report+exit on error, stay silent otherwise
+    log="/tmp/test-tablizer.$$.log"
+    name=$1
+    shift
+
+    echo -n "TEST $name "
+
+    $* > $log 2>&1
+
+    if test $? -ne 0; then
+        echo "failed, see $log"
+        fail=1
+    else
+        echo "ok"
+        rm -f $log
+    fi
+}
+
+# only use files in test dir
+cd $(dirname $0)
+
+echo "Executing commandline tests ..."
+
+# io pattern tests
+ex io-pattern-and-file $t bk7 testtable
+cat testtable | ex io-pattern-and-stdin $t bk7
+cat testtable | ex io-pattern-and-stdin-dash $t bk7 -
+
+# same w/o pattern
+ex io-just-file $t testtable
+cat testtable | ex io-just-stdin $t
+cat testtable | ex io-just-stdin-dash $t -
+
+if test $fail -ne 0; then
+    echo "!!! Some tests failed !!!"
+    exit 1
+fi

t/testtable

@@ -0,0 +1,6 @@
+NAME                                                 READY   STATUS    RESTARTS       AGE
+alertmanager-kube-prometheus-alertmanager-0          2/2     Running   35 (45m ago)   11d
+grafana-fcc54cbc9-bk7s8                              1/1     Running   17 (45m ago)   1d
+kube-prometheus-blackbox-exporter-5d85b5d8f4-tskh7   1/1     Running   17 (45m ago)   1h44m
+kube-prometheus-kube-state-metrics-b4cd9487-75p7f    1/1     Running   20 (45m ago)   45m
+kube-prometheus-node-exporter-bfzpl                  1/1     Running   17 (45m ago)   54s

t/testtable2

@@ -0,0 +1,6 @@
+NAME  DURATION
+x     10
+a     100
+z     0
+u     4
+k     6

tablizer.1

@@ -133,7 +133,7 @@
 .\" ========================================================================
 .\"
 .IX Title "TABLIZER 1"
-.TH TABLIZER 1 "2022-10-14" "1" "User Commands"
+.TH TABLIZER 1 "2024-05-07" "1" "User Commands"
 .\" For nroff, turn off justification.  Always turn off hyphenation; it makes
 .\" way too many mistakes in technical documents.
 .if n .ad l
@@ -146,21 +146,40 @@ tablizer \- Manipulate tabular output of other programs
 \&    Usage:
 \&      tablizer [regex] [file, ...] [flags]
 \&    
-\&    Flags:
+\&    Operational Flags:
 \&      \-c, \-\-columns string     Only show the speficied columns (separated by ,)
-\&      \-d, \-\-debug              Enable debugging
-\&      \-h, \-\-help               help for tablizer
 \&      \-v, \-\-invert\-match       select non\-matching rows
-\&      \-m, \-\-man                Display manual page
 \&      \-n, \-\-no\-numbering       Disable header numbering
 \&      \-N, \-\-no\-color           Disable pattern highlighting
-\&      \-o, \-\-output string      Output mode \- one of: orgtbl, markdown, extended, ascii(default)
+\&      \-H, \-\-no\-headers         Disable headers display
+\&      \-s, \-\-separator string   Custom field separator
+\&      \-k, \-\-sort\-by int        Sort by column (default: 1)
+\&      \-z, \-\-fuzzy              Use fuzzy search [experimental]
+\&      \-F, \-\-filter field=reg   Filter given field with regex, can be used multiple times
+\&
+\&    Output Flags (mutually exclusive):
 \&      \-X, \-\-extended           Enable extended output
 \&      \-M, \-\-markdown           Enable markdown table output
 \&      \-O, \-\-orgtbl             Enable org\-mode table output
-\&      \-s, \-\-separator string   Custom field separator
-\&      \-k, \-\-sort\-by int        Sort by column (default: 1)
-\&      \-v, \-\-version            Print program version
+\&      \-S, \-\-shell              Enable shell evaluable output
+\&      \-Y, \-\-yaml               Enable yaml output
+\&      \-C, \-\-csv                Enable CSV output
+\&      \-A, \-\-ascii              Default output mode, ascii tabular
+\&      \-L, \-\-hightlight\-lines   Use alternating background colors for tables
+\&
+\&    Sort Mode Flags (mutually exclusive):
+\&      \-a, \-\-sort\-age           sort according to age (duration) string
+\&      \-D, \-\-sort\-desc          Sort in descending order (default: ascending)
+\&      \-i, \-\-sort\-numeric       sort according to string numerical value
+\&      \-t, \-\-sort\-time          sort according to time string
+\&
+\&    Other Flags:
+\&          \-\-completion <shell> Generate the autocompletion script for <shell>
+\&      \-f, \-\-config <file>      Configuration file (default: ~/.config/tablizer/config)
+\&      \-d, \-\-debug              Enable debugging
+\&      \-h, \-\-help               help for tablizer
+\&      \-m, \-\-man                Display manual page
+\&      \-V, \-\-version            Print program version
 .Ve
 .SH "DESCRIPTION"
 .IX Header "DESCRIPTION"
@@ -209,7 +228,7 @@ have a numer associated with it, e.g.:
 .Ve
 .PP
 These numbers denote the column and  you can use them to specify which
-columns you want to have in your output:
+columns you want to have in your output (see \s-1COLUMNS\s0:
 .PP
 .Vb 1
 \&   kubectl get pods | tablizer \-c1,3
@@ -220,17 +239,34 @@ the original order.
 .PP
 The numbering can be suppressed by using the \fB\-n\fR option.
 .PP
+By  default tablizer  shows  a  header containing  the  names of  each
+column.  This can  be disabled using the \fB\-H\fR option.   Be aware that
+this only affects tabular output modes.  Shell, Extended, Yaml and \s-1CSV\s0
+output modes always use the column names.
+.PP
 By  default, if  a  \fBpattern\fR  has been  speficied,  matches will  be
 highlighted. You can disable this behavior with the \fB\-N\fR option.
 .PP
 Use the  \fB\-k\fR option to specify  by which column to  sort the tabular
 data (as in \s-1GNU\s0 \fBsort\fR\|(1)). The default sort column is the first one. To
-disable sorting at all, supply 0 (Zero) to \-k.
+disable sorting at all, supply 0  (Zero) to \-k. The default sort order
+is ascending. You can change this to descending order using the option
+\&\fB\-D\fR. The default  sort order is by string, but  there are other sort
+modes:
+.IP "\fB\-a \-\-sort\-age\fR" 4
+.IX Item "-a --sort-age"
+Sorts duration strings like \*(L"1d4h32m51s\*(R".
+.IP "\fB\-i \-\-sort\-numeric\fR" 4
+.IX Item "-i --sort-numeric"
+Sorts numeric fields.
+.IP "\fB\-t \-\-sort\-time\fR" 4
+.IX Item "-t --sort-time"
+Sorts timestamps.
 .PP
 Finally the  \fB\-d\fR option  enables debugging  output which  is mostly
 useful for the developer.
-.SS "\s-1PATTERNS\s0"
-.IX Subsection "PATTERNS"
+.SS "\s-1PATTERNS AND FILTERING\s0"
+.IX Subsection "PATTERNS AND FILTERING"
 You can reduce the rows being  displayed by using a regular expression
 pattern.  The  regexp is  \s-1PCRE\s0 compatible, refer  to the  syntax cheat
 sheet here: <https://github.com/google/re2/wiki/Syntax>.  If you want
@@ -261,6 +297,57 @@ Example for a case insensitive search:
 .Vb 1
 \&    kubectl get pods \-A | tablizer "(?i)account"
 .Ve
+.PP
+You  can use  the experimental  fuzzy search  feature by  providing the
+option \fB\-z\fR, in which case the  pattern is regarded as a fuzzy search
+term, not a regexp.
+.PP
+Sometimes you want to  filter by one or more columns.  You can do that
+using the \fB\-F\fR option. The option can be specified multiple times and
+has the following format:
+.PP
+.Vb 1
+\&    fieldname=regexp
+.Ve
+.PP
+Fieldnames (== columns headers) are case insensitive.
+.PP
+If you specify more than one filter, both filters have to match (\s-1AND\s0
+operation).
+.PP
+If the option \fB\-v\fR is specified, the filtering is inverted.
+.SS "\s-1COLUMNS\s0"
+.IX Subsection "COLUMNS"
+The  parameter  \fB\-c\fR  can  be  used  to  specify,  which  columns  to
+display.  By default  tablizer numerizes  the header  names and  these
+numbers can  be used to specify  which header to display,  see example
+above.
+.PP
+However, beside  numbers, you  can also  use regular  expressions with
+\&\fB\-c\fR, also  separated by comma. And  you can mix column  numbers with
+regexps.
+.PP
+Lets take this table:
+.PP
+.Vb 4
+\&        PID TTY          TIME CMD
+\&      14001 pts/0    00:00:00 bash
+\&      42871 pts/0    00:00:00 ps
+\&      42872 pts/0    00:00:00 sed
+.Ve
+.PP
+We want to see only the \s-1CMD\s0 column and use a regex for this:
+.PP
+.Vb 6
+\&    ps | tablizer \-s \*(Aq\es+\*(Aq \-c C
+\&    CMD(4)
+\&    bash
+\&    ps
+\&    tablizer
+\&    sed
+.Ve
+.PP
+where \*(L"C\*(R" is our regexp which matches \s-1CMD.\s0
 .SS "\s-1OUTPUT MODES\s0"
 .IX Subsection "OUTPUT MODES"
 There might be cases  when the tabular output of a  program is way too
@@ -297,7 +384,116 @@ You can use this in an eval loop.
 .PP
 Beside normal  ascii mode  (the default) and  extended mode  there are
 more output modes available: \fBorgtbl\fR  which prints an Emacs org-mode
-table and \fBmarkdown\fR which prints a Markdown table.
+table and  \fBmarkdown\fR which prints  a Markdown table,  \fByaml\fR, which
+prints  yaml encoding  and \s-1CSV\s0  mode, which  prints a  comma separated
+value file.
+.SS "\s-1ENVIRONMENT VARIABLES\s0"
+.IX Subsection "ENVIRONMENT VARIABLES"
+\&\fBtablizer\fR supports  certain environment variables which  use can use
+to  influence   program  behavior.   Commandline  flags   have  always
+precedence over environment variables.
+.IP "<T_NO_HEADER_NUMBERING> \- disable numbering of header fields, like \fB\-n\fR." 4
+.IX Item "<T_NO_HEADER_NUMBERING> - disable numbering of header fields, like -n."
+.PD 0
+.IP "<T_COLUMNS> \- comma separated list of columns to output, like \fB\-c\fR" 4
+.IX Item "<T_COLUMNS> - comma separated list of columns to output, like -c"
+.IP "<\s-1NO_COLORS\s0> \- disable colorization of matches, like \fB\-N\fR" 4
+.IX Item "<NO_COLORS> - disable colorization of matches, like -N"
+.PD
+.SS "\s-1COMPLETION\s0"
+.IX Subsection "COMPLETION"
+Shell completion for command line options  can be enabled by using the
+\&\fB\-\-completion\fR  flag. The  required  parameter is  the  name of  your
+shell. Currently supported are: bash, zsh, fish and powershell.
+.PP
+Detailed instructions:
+.IP "Bash:" 4
+.IX Item "Bash:"
+.Vb 1
+\&   source <(tablizer \-\-completion bash)
+.Ve
+.Sp
+To load completions for each session, execute once:
+.Sp
+.Vb 2
+\&  # Linux:
+\&  $ tablizer \-\-completion bash > /etc/bash_completion.d/tablizer
+\&
+\&  # macOS:
+\&  $ tablizer \-\-completion bash > $(brew \-\-prefix)/etc/bash_completion.d/tablizer
+.Ve
+.IP "Zsh:" 4
+.IX Item "Zsh:"
+If shell completion is not already enabled in your environment,
+you will need to enable it.  You can execute the following once:
+.Sp
+.Vb 1
+\&  echo "autoload \-U compinit; compinit" >> ~/.zshrc
+.Ve
+.Sp
+To load completions for each session, execute once:
+.Sp
+.Vb 1
+\&  $ tablizer \-\-completion zsh > "${fpath[1]}/_tablizer"
+.Ve
+.Sp
+You will need to start a new shell for this setup to take effect.
+.IP "fish:" 4
+.IX Item "fish:"
+.Vb 1
+\&   tablizer \-\-completion fish | source
+.Ve
+.Sp
+To load completions for each session, execute once:
+.Sp
+.Vb 1
+\&   tablizer \-\-completion fish > ~/.config/fish/completions/tablizer.fish
+.Ve
+.IP "PowerShell:" 4
+.IX Item "PowerShell:"
+.Vb 1
+\&   tablizer \-\-completion powershell | Out\-String | Invoke\-Expression
+.Ve
+.Sp
+To load completions for every new session, run:
+.Sp
+.Vb 1
+\&   tablizer \-\-completion powershell > tablizer.ps1
+.Ve
+.Sp
+and source this file from your PowerShell profile.
+.SH "CONFIGURATION AND COLORS"
+.IX Header "CONFIGURATION AND COLORS"
+YOu can put certain configuration values into a configuration file in
+\&\s-1HCL\s0 format. By default tablizer looks for
+\&\f(CW\*(C`$HOME/.config/tablizer/config\*(C'\fR, but you can provide one using the
+parameter \f(CW\*(C`\-f\*(C'\fR.
+.PP
+In the configuration the following variables can be defined:
+.PP
+.Vb 8
+\&    BG             = "lightGreen"
+\&    FG             = "white"
+\&    HighlightBG    = "lightGreen"
+\&    HighlightFG    = "white"
+\&    NoHighlightBG  = "white"
+\&    NoHighlightFG  = "lightGreen"
+\&    HighlightHdrBG = "red"
+\&    HighlightHdrFG = "white"
+.Ve
+.PP
+The following color definitions are available:
+.PP
+black, blue,  cyan, darkGray, default, green,  lightBlue, lightCyan,
+lightGreen,   lightMagenta,   lightRed,   lightWhite,   lightYellow,
+magenta, red, white, yellow
+.PP
+The Variables \fB\s-1FG\s0\fR and \fB\s-1BG\s0\fR are being used to highlight matches. The
+other *FG and *BG variables are for colored table output (enabled with
+the \f(CW\*(C`\-L\*(C'\fR parameter).
+.PP
+Colorization can be turned off completely either by setting the
+parameter \f(CW\*(C`\-N\*(C'\fR or the environment variable \fB\s-1NO_COLOR\s0\fR to a true value.
 .SH "BUGS"
 .IX Header "BUGS"
 In order to report a bug, unexpected behavior, feature requests
@@ -307,15 +503,27 @@ or to submit a patch, please open an issue on github:
 .IX Header "LICENSE"
 This software is licensed under the \s-1GNU GENERAL PUBLIC LICENSE\s0 version 3.
 .PP
-Copyright (c) 2022 by Thomas von Dein
+Copyright (c) 2022\-2024 by Thomas von Dein
 .PP
-This software uses the following \s-1GO\s0 libraries:
+This software uses the following \s-1GO\s0 modules:
 .IP "repr (https://github.com/alecthomas/repr)" 4
 .IX Item "repr (https://github.com/alecthomas/repr)"
 Released under the \s-1MIT\s0 License, Copyright (c) 2016 Alec Thomas
 .IP "cobra (https://github.com/spf13/cobra)" 4
 .IX Item "cobra (https://github.com/spf13/cobra)"
 Released under the Apache 2.0 license, Copyright 2013\-2022 The Cobra Authors
+.IP "dateparse (github.com/araddon/dateparse)" 4
+.IX Item "dateparse (github.com/araddon/dateparse)"
+Released under the \s-1MIT\s0 License, Copyright (c) 2015\-2017 Aaron Raddon
+.IP "color (github.com/gookit/color)" 4
+.IX Item "color (github.com/gookit/color)"
+Released under the \s-1MIT\s0 License, Copyright (c) 2016 inhere
+.IP "tablewriter (github.com/olekukonko/tablewriter)" 4
+.IX Item "tablewriter (github.com/olekukonko/tablewriter)"
+Released under the \s-1MIT\s0 License, Copyright (c) 201 by Oleku Konko
+.IP "yaml (gopkg.in/yaml.v3)" 4
+.IX Item "yaml (gopkg.in/yaml.v3)"
+Released under the \s-1MIT\s0 License, Copyright (c) 2006\-2011 Kirill Simonov
 .SH "AUTHORS"
 .IX Header "AUTHORS"
 Thomas von Dein \fBtom \s-1AT\s0 vondein \s-1DOT\s0 org\fR

tablizer.pod

@@ -7,21 +7,40 @@ tablizer - Manipulate tabular output of other programs
     Usage:
       tablizer [regex] [file, ...] [flags]
     
-    Flags:
+    Operational Flags:
       -c, --columns string     Only show the speficied columns (separated by ,)
-      -d, --debug              Enable debugging
-      -h, --help               help for tablizer
       -v, --invert-match       select non-matching rows
-      -m, --man                Display manual page
       -n, --no-numbering       Disable header numbering
       -N, --no-color           Disable pattern highlighting
-      -o, --output string      Output mode - one of: orgtbl, markdown, extended, ascii(default)
+      -H, --no-headers         Disable headers display
+      -s, --separator string   Custom field separator
+      -k, --sort-by int        Sort by column (default: 1)
+      -z, --fuzzy              Use fuzzy search [experimental]
+      -F, --filter field=reg   Filter given field with regex, can be used multiple times
+
+    Output Flags (mutually exclusive):
       -X, --extended           Enable extended output
       -M, --markdown           Enable markdown table output
       -O, --orgtbl             Enable org-mode table output
-      -s, --separator string   Custom field separator
-      -k, --sort-by int        Sort by column (default: 1)
-      -v, --version            Print program version
+      -S, --shell              Enable shell evaluable output
+      -Y, --yaml               Enable yaml output
+      -C, --csv                Enable CSV output
+      -A, --ascii              Default output mode, ascii tabular
+      -L, --hightlight-lines   Use alternating background colors for tables
+
+    Sort Mode Flags (mutually exclusive):
+      -a, --sort-age           sort according to age (duration) string
+      -D, --sort-desc          Sort in descending order (default: ascending)
+      -i, --sort-numeric       sort according to string numerical value
+      -t, --sort-time          sort according to time string
+
+    Other Flags:
+          --completion <shell> Generate the autocompletion script for <shell>
+      -f, --config <file>      Configuration file (default: ~/.config/tablizer/config)
+      -d, --debug              Enable debugging
+      -h, --help               help for tablizer
+      -m, --man                Display manual page
+      -V, --version            Print program version
 
 
 =head1 DESCRIPTION
@@ -67,7 +86,7 @@ have a numer associated with it, e.g.:
    NAME(1) READY(2) STATUS(3) RESTARTS(4) AGE(5)
 
 These numbers denote the column and  you can use them to specify which
-columns you want to have in your output:
+columns you want to have in your output (see L<COLUMNS>:
 
    kubectl get pods | tablizer -c1,3
 
@@ -76,17 +95,41 @@ the original order.
 
 The numbering can be suppressed by using the B<-n> option.
 
+By  default tablizer  shows  a  header containing  the  names of  each
+column.  This can  be disabled using the B<-H> option.   Be aware that
+this only affects tabular output modes.  Shell, Extended, Yaml and CSV
+output modes always use the column names.
+
 By  default, if  a  B<pattern>  has been  speficied,  matches will  be
 highlighted. You can disable this behavior with the B<-N> option.
 
 Use the  B<-k> option to specify  by which column to  sort the tabular
 data (as in GNU sort(1)). The default sort column is the first one. To
-disable sorting at all, supply 0 (Zero) to -k.
+disable sorting at all, supply 0  (Zero) to -k. The default sort order
+is ascending. You can change this to descending order using the option
+B<-D>. The default  sort order is by string, but  there are other sort
+modes:
+
+=over
+
+=item B<-a --sort-age>
+
+Sorts duration strings like "1d4h32m51s".
+
+=item B<-i --sort-numeric>
+
+Sorts numeric fields.
+
+=item B<-t --sort-time>
+
+Sorts timestamps.
+
+=back
 
 Finally the  B<-d> option  enables debugging  output which  is mostly
 useful for the developer.
 
-=head2 PATTERNS
+=head2 PATTERNS AND FILTERING
 
 You can reduce the rows being  displayed by using a regular expression
 pattern.  The  regexp is  PCRE compatible, refer  to the  syntax cheat
@@ -113,6 +156,52 @@ Example for a case insensitive search:
 
     kubectl get pods -A | tablizer "(?i)account"
 
+You  can use  the experimental  fuzzy search  feature by  providing the
+option B<-z>, in which case the  pattern is regarded as a fuzzy search
+term, not a regexp.
+
+Sometimes you want to  filter by one or more columns.  You can do that
+using the B<-F> option. The option can be specified multiple times and
+has the following format:
+
+    fieldname=regexp
+
+Fieldnames (== columns headers) are case insensitive.
+
+If you specify more than one filter, both filters have to match (AND
+operation).
+
+If the option B<-v> is specified, the filtering is inverted.
+
+
+=head2 COLUMNS
+
+The  parameter  B<-c>  can  be  used  to  specify,  which  columns  to
+display.  By default  tablizer numerizes  the header  names and  these
+numbers can  be used to specify  which header to display,  see example
+above.
+
+However, beside  numbers, you  can also  use regular  expressions with
+B<-c>, also  separated by comma. And  you can mix column  numbers with
+regexps.
+
+Lets take this table:
+
+        PID TTY          TIME CMD
+      14001 pts/0    00:00:00 bash
+      42871 pts/0    00:00:00 ps
+      42872 pts/0    00:00:00 sed
+
+We want to see only the CMD column and use a regex for this:
+
+    ps | tablizer -s '\s+' -c C
+    CMD(4)
+    bash
+    ps
+    tablizer
+    sed
+
+where "C" is our regexp which matches CMD.
 
 =head2 OUTPUT MODES
 
@@ -146,7 +235,111 @@ You can use this in an eval loop.
 
 Beside normal  ascii mode  (the default) and  extended mode  there are
 more output modes available: B<orgtbl>  which prints an Emacs org-mode
-table and B<markdown> which prints a Markdown table.
+table and  B<markdown> which prints  a Markdown table,  B<yaml>, which
+prints  yaml encoding  and CSV  mode, which  prints a  comma separated
+value file.
+
+=head2 ENVIRONMENT VARIABLES
+
+B<tablizer> supports  certain environment variables which  use can use
+to  influence   program  behavior.   Commandline  flags   have  always
+precedence over environment variables.
+
+=over
+
+=item <T_NO_HEADER_NUMBERING> - disable numbering of header fields, like B<-n>.
+
+=item <T_COLUMNS> - comma separated list of columns to output, like B<-c>
+
+=item <NO_COLORS> - disable colorization of matches, like B<-N>
+
+=back
+
+=head2 COMPLETION
+
+Shell completion for command line options  can be enabled by using the
+B<--completion>  flag. The  required  parameter is  the  name of  your
+shell. Currently supported are: bash, zsh, fish and powershell.
+
+Detailed instructions:
+
+=over
+
+=item Bash:
+
+   source <(tablizer --completion bash)
+
+To load completions for each session, execute once:
+
+  # Linux:
+  $ tablizer --completion bash > /etc/bash_completion.d/tablizer
+
+  # macOS:
+  $ tablizer --completion bash > $(brew --prefix)/etc/bash_completion.d/tablizer
+
+=item Zsh:
+
+If shell completion is not already enabled in your environment,
+you will need to enable it.  You can execute the following once:
+
+  echo "autoload -U compinit; compinit" >> ~/.zshrc
+
+To load completions for each session, execute once:
+
+  $ tablizer --completion zsh > "${fpath[1]}/_tablizer"
+
+You will need to start a new shell for this setup to take effect.
+
+=item fish:
+
+   tablizer --completion fish | source
+
+To load completions for each session, execute once:
+
+   tablizer --completion fish > ~/.config/fish/completions/tablizer.fish
+
+=item PowerShell:
+
+   tablizer --completion powershell | Out-String | Invoke-Expression
+
+To load completions for every new session, run:
+
+   tablizer --completion powershell > tablizer.ps1
+
+and source this file from your PowerShell profile.
+
+=back
+
+=head1 CONFIGURATION AND COLORS
+
+YOu can put certain configuration values into a configuration file in
+HCL format. By default tablizer looks for
+C<$HOME/.config/tablizer/config>, but you can provide one using the
+parameter C<-f>.
+
+In the configuration the following variables can be defined:
+
+    BG             = "lightGreen"
+    FG             = "white"
+    HighlightBG    = "lightGreen"
+    HighlightFG    = "white"
+    NoHighlightBG  = "white"
+    NoHighlightFG  = "lightGreen"
+    HighlightHdrBG = "red"
+    HighlightHdrFG = "white"
+
+The following color definitions are available:
+
+black, blue,  cyan, darkGray, default, green,  lightBlue, lightCyan,
+lightGreen,   lightMagenta,   lightRed,   lightWhite,   lightYellow,
+magenta, red, white, yellow
+
+The Variables B<FG> and B<BG> are being used to highlight matches. The
+other *FG and *BG variables are for colored table output (enabled with
+the C<-L> parameter).
+
+Colorization can be turned off completely either by setting the
+parameter C<-N> or the environment variable B<NO_COLOR> to a true value.
 
 =head1 BUGS
 
@@ -158,9 +351,9 @@ L<https://github.com/TLINDEN/tablizer/issues>.
 
 This software is licensed under the GNU GENERAL PUBLIC LICENSE version 3.
 
-Copyright (c) 2022 by Thomas von Dein
+Copyright (c) 2022-2024 by Thomas von Dein
 
-This software uses the following GO libraries:
+This software uses the following GO modules:
 
 =over 4
 
@@ -172,6 +365,22 @@ Released under the MIT License, Copyright (c) 2016 Alec Thomas
 
 Released under the Apache 2.0 license, Copyright 2013-2022 The Cobra Authors
 
+=item dateparse (github.com/araddon/dateparse)
+
+Released under the MIT License, Copyright (c) 2015-2017 Aaron Raddon
+
+=item color (github.com/gookit/color)
+
+Released under the MIT License, Copyright (c) 2016 inhere
+
+=item tablewriter (github.com/olekukonko/tablewriter)
+
+Released under the MIT License, Copyright (c) 201 by Oleku Konko
+
+=item yaml (gopkg.in/yaml.v3)
+
+Released under the MIT License, Copyright (c) 2006-2011 Kirill Simonov
+
 =back
 
 =head1 AUTHORS