3f91b4415b
fix #77 : use processed ad dir for duplicate checking, not slug
2024-02-10 15:11:37 +01:00
5a2c277f0e
fix #71 and #73 : add support for outdir template and enhance docs
2024-02-10 14:44:09 +01:00
612ed2aa79
fix #74 : warn if about to write to already visited ad, overwrite if -f
2024-02-10 14:44:09 +01:00
ed78731b3c
check seek error
2024-01-27 17:34:44 +01:00
a84f0e1436
get rid of duplicate bytes.Buffer, use bytes.Reader instead, #39
2024-01-27 17:34:44 +01:00
d8d5be5c7d
fix #58 : add missing dashes to self issue template
2024-01-27 17:34:44 +01:00
bcf920c91e
correct #39 add --ignoreerrors flag
2024-01-27 17:34:44 +01:00
T.v.Dein
14f8c3fd43
Fix/linter ( #66 )
...
* added lint targets
* fix linter errors
* enhance error handling
* !!BREAKING!! rename Id to ID in tpls
2024-01-25 19:04:15 +01:00
9cd1fc0596
behavior changes: UserAgent configurable, test cookies, check errors
2024-01-24 19:22:31 +01:00
8df3ebfa6d
add throttling to image download
2024-01-24 19:22:31 +01:00
de82127223
first step in fixing #49 :
...
fetch cookies from 1st response and use them in subsequent requests.
2024-01-24 19:22:31 +01:00
a79a28f4a1
add contribution guidelines and non-code-of-conduct
2024-01-23 18:01:14 +01:00
95b1172b7f
fix typo
2024-01-23 17:26:06 +01:00
6f3954f1c0
added issue templates and fix make show-versions
2024-01-23 14:20:33 +01:00
a465bbfaa8
Added german README, update docker docs, add latest image tag
2024-01-23 14:20:33 +01:00
ddae6ed8be
bump version (feature update)
2024-01-22 18:43:44 +01:00
8cced1f9e1
Fix #47 : add mock http server for testing w/o hitting prod site
2024-01-22 18:43:44 +01:00
3245438564
update yadu to v0.1.1, fix #46
2024-01-22 18:43:44 +01:00
71c528114f
fix linter error
2024-01-22 18:43:44 +01:00
e2afc1350b
added -f to override d-hash, better debug and error handling
2024-01-22 18:43:44 +01:00
e971070f9f
added image diff hash distance caching to not overwrite similar images
2024-01-22 18:43:44 +01:00
T.v.Dein
8771ec1108
added support to calculate and store the ad expire date ( #43 )
2024-01-19 18:38:35 +01:00
T.v.Dein
1896209b96
use new yadu log handler, +tests, +upd modules, +version
2024-01-19 18:38:35 +01:00
T.v.Dein
3c93c9fce0
added docker image support
...
* added environment variable support
* added docker instructions
* added .env hint
2024-01-19 18:38:35 +01:00
T.v.Dein
42a958fc4c
refactored out http fetching code into Fetcher{}/fetch.go
2024-01-19 18:38:35 +01:00
T.v.Dein
5fa46ff106
Add HTTP retries and the possibility to ignore image download errors ( #33 )
...
added HTTP retry and --ignoreerrors which ignores image download errors, fix #30
2024-01-19 18:38:35 +01:00
T.v.Dein
cca3211023
Enhancement/http ( #32 )
...
* added HTTP debug logging using `-d` or `DEBUGHTTP=1` (headers only)
2024-01-19 18:38:35 +01:00
T.v.Dein
dce7604afb
fix #30 : revert default adnamedir to just use the slug as before ( #31 )
2024-01-19 18:38:35 +01:00
0fd9b519d1
fixed changes on kleinanzeigen.de:
...
- Meta did not contain condition and category together anymore, they
removed the category. Therefore fetching (that is, validation)
failed.
- Now we extract the condition and category directly.
- On top, category now includes the whole category tree.
- unit tests had to be tweaked for this measure.
2024-01-19 18:38:35 +01:00
6b7f727449
fixed utf8
2024-01-19 18:38:35 +01:00
5abbab9527
added template for ad directory, by default include id now
2024-01-19 18:38:35 +01:00
T.v.Dein
e03c7debb6
remove duplicate license badge ( #28 )
...
* remove duplicate license badge
* fix badges
2024-01-19 18:38:35 +01:00
1d2483d18f
portable error check
2024-01-19 18:38:35 +01:00
b17f4f0f3e
also added coverage report+badge
2024-01-19 18:38:35 +01:00
4a91167871
put all tests into main_test.go, more failure mode tests and verify
2024-01-19 18:38:35 +01:00
0baaf6f38b
better error message on 404
2024-01-19 18:38:35 +01:00
42182bb6c9
add commandline main() test units
2024-01-19 18:38:35 +01:00
8455c193eb
pass a io.Writer to loggers and outputs so we can test the cmdline
2024-01-19 18:38:35 +01:00
d1faa10a52
added more invalid tests
2024-01-19 18:38:35 +01:00
e28137bf9b
upd httpmock+deps
2024-01-19 18:38:35 +01:00
1ff5c240c8
put ad code into separate file, enhance error checking
2024-01-19 18:38:35 +01:00
T.v.Dein
f893f9c3d7
Test/add mock tests ( #24 )
...
* add scrape unit test using httpmock lib
2024-01-19 18:38:35 +01:00
T.v.Dein
c4e88d98f2
fix linter errors ( #23 )
2024-01-19 18:38:35 +01:00
T.v.Dein
0cca387982
add ci pipeline ( #22 )
...
Co-authored-by: Thomas von Dein <tom@izb.net >
2024-01-19 18:38:35 +01:00
T.v.Dein
9e619fb3c5
Doc/add prior art ( #21 )
...
* add mor prior art
2024-01-19 18:38:35 +01:00
T.v.Dein
0fdfed2929
added windows screenshots ( #20 )
...
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
T.v.Dein
73c09ec38b
Revert "Fix/newline windows ( #18 )" ( #19 )
...
This reverts commit eaf4db6cef
2024-01-19 18:38:35 +01:00
T.v.Dein
f901af4f0c
Fix/newline windows ( #18 )
...
* fix #17 : use fmt.Println() after stats
* bump version
2024-01-19 18:38:35 +01:00
T.v.Dein
2a8f53ca98
added screenshots and a section about prior work ( #16 )
2024-01-19 18:38:35 +01:00
T.v.Dein
4a95cb1f5e
add doc link ( #15 )
...
Co-authored-by: Thomas von Dein <tom@vondein.org >
2024-01-19 18:38:35 +01:00
